Open
Conversation
kaapstorm
force-pushed
the
nh/write_table
branch
from
440ae37 to
faf2b84
Compare
Contributor
Author
|
Note It seems, based on testing, that this change is still unstable when exporting large amounts of data. I am going to upgrade SQLAlchemy from 1.4 to 2.0 to get the benefit of better transaction management. New commits are expected on this branch. |
Contributor
|
Suggestion: convert to DRAFT until you're done pushing to the branch |
Contributor
Author
|
Based on testing, upgrading SQLAlchemy to 2.0 has resolved memory issues. 😅 |
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
kaapstorm
force-pushed
the
nh/write_table
branch
from
59cb10f to
5c9734c
Compare
Remove MetaData(bind=...), autoload=True, declarative_base() import, and branched-connection pattern in env.py. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
IntegrityError handling no longer aborts the outer transaction. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Raw SQL strings must be wrapped in text() in SQLAlchemy 2.0. Also convert positional %s parameters to named :name bindings, and add conn.commit() calls after DDL operations now that autocommit has been removed.
engine.execute() was removed in SQLAlchemy 2.0. Callers must obtain a connection via engine.connect() and execute statements on it.
In SQLAlchemy 2.0 rows are tuple-like by default. Call .mappings() on the Result to iterate dict-like RowMapping objects.
Accessing the raw DB-API connection via Connection.connection.connection is deprecated in SQLAlchemy 2.0. Use Connection.connection.dbapi_connection instead.
Replace raw SQL COMMIT with SQLAlchemy 2.0 transaction API: - __enter__/__exit__ manage begin/commit/rollback - _flush() commits current transaction and starts a new one - _flush_batch() rolls back failed transaction before retry Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SQLAlchemy 2.0's .mappings() returns RowMapping dicts, not tuples. Update assertions to compare against dicts with column names as keys. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Annotate compatibility dict in writers.py to accept dialect-specific types like JSON and BIT - Use separate variable names for postgres and mysql insert statements to avoid type conflicts - Remove unused type: ignore comment from Checkpoint class - Cast pagination_mode to str for enum indexing - Assert dbapi_connection is not None before accessing autocommit Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
kaapstorm
force-pushed
the
nh/write_table
branch
from
5c9734c to
037e92c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Optimizes
SqlTableWriter:make_table_compatible()on every row, only runs on the first 10 (SCHEMA_CHECK_ROWSconst) rows.MetaDataobject growth.Transactions are committed explicitly for now. When we upgrade SQLAlchemy from 1.4 to 2.0 then we will use SQLAlchemy's
connection.begin()context managers for managing transactions. That is out of scope for this PR.