feat: add DataFrame writeback to remote ClickHouse server#561
Draft
wudidapaopao wants to merge 4 commits intochdb-io:mainfrom
Draft
feat: add DataFrame writeback to remote ClickHouse server#561wudidapaopao wants to merge 4 commits intochdb-io:mainfrom
wudidapaopao wants to merge 4 commits intochdb-io:mainfrom
Conversation
wudidapaopao
force-pushed
the
support_dataframe_writeback
branch
from
c57e6f5 to
a54fc6e
Compare
|
|
ClickHouse doesn't yet support atomic CREATE OR REPLACE MATERIALIZED VIEW (PR #100539 still open), and emulating it via DROP + CREATE leaves a window where source INSERTs are silently dropped. Remove if_mv_exists from create_materialized_view and reject if_exists='replace' in save(); existing MV now always raises. Re-enable once upstream lands.
…chema mismatch - _check_remote_table_exists: query system.tables via remote() instead of EXISTS TABLE through remote(query=...), which only returns DDL status rows - _build_engine_clause: emit SETTINGS allow_nullable_key = 1 for MergeTree so DESCRIBE-inferred Nullable columns can appear in ORDER BY - normalize df index once via _materialize_for_writeback so CREATE TABLE, schema evolution and INSERT all see the same column shape - tests: expect ExecutionError for fail-mode collisions and assert ClickHouse rejection for cross-server materialized views
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #560
Add writeback APIs to DataStore for writing data back to a remote ClickHouse server.
Changes
save()— unified entry pointto_clickhouse()— write data to a remote table (fail/replace/append)create_view()/create_materialized_view()— create views on remote server