Skip to content

orm: add bulk insert/update support with CASE WHEN batch updates, MySQL time conversion, and checker/codegen validation#27132

Open
Jengro777 wants to merge 4 commits intovlang:masterfrom
Jengro777:orm-bulk2
Open

orm: add bulk insert/update support with CASE WHEN batch updates, MySQL time conversion, and checker/codegen validation#27132
Jengro777 wants to merge 4 commits intovlang:masterfrom
Jengro777:orm-bulk2

Conversation

@Jengro777
Copy link
Copy Markdown
Contributor

fix #26993

Summary

Add bulk (batch) INSERT and UPDATE support to the V ORM, enabling efficient multi-row operations through a single SQL statement instead of individual per-row round trips.

Both the compiler-level sql { } syntax and the function-call API (QueryBuilder) are supported.

Changes

vlib/orm/orm.v — Core batch SQL generation

  • QueryData: add batch_rows and batch_key fields for batch operations
  • orm_stmt_gen(.insert): generate multi-value VALUES (?, ?), (?, ?) tuples when batch_rows > 0
  • orm_stmt_gen(.update): generate CASE "key" WHEN ? THEN ? ... ELSE "col" END expressions for batch updates when batch_rows > 0
  • prepare_insert_query_data: handle batch data layout (interleaved row-by-row), filter types/kinds/auto_fields arrays correctly, and auto-skip serial fields only when all batch rows have zero values
  • clone_query_data: propagate parentheses, batch_rows, batch_key

vlib/orm/orm_func.v — Function-call API

  • insert_many (QueryBuilder): changed from N individual conn.insert() calls to a single batch INSERT with interleaved data and batch_rows set
  • update_many (new standalone function): batch UPDATE using CASE WHEN with a key field, accepting field_names to select which columns to update and key_field as the matching column; WHERE clause uses IN

vlib/orm/orm_fn_test.v — SQL generation unit tests

  • test_orm_stmt_gen_bulk_insert: verify multi-value tuple generation for default, pg, and mysql dialects
  • test_orm_stmt_gen_bulk_update: verify CASE WHEN batch UPDATE generation for all three dialects
  • test_orm_stmt_gen_insert_default_values_mysql: verify batch () VALUES (), () for MySQL with auto-fields

vlib/orm/orm_func_test.v — Integration tests

  • test_orm_func_update_many: end-to-end SQLite test for update_many with 2-row batch, single-row batch, and empty-values error

vlib/v/ast/ast.v — AST node fields

  • SqlStmtLine: add is_array_insert, is_array_update, array_update_var, array_update_key

vlib/v/checker/orm.v — Compile-time validation

  • Detect array-typed objects in sql { insert array into T } and sql { update T set ... = array.field where key == array.key }
  • Validate element types match the table, reject pointer element types, reject non-primitive/non-enum/non-time fields in bulk mode
  • Reject array upsert (not yet supported)

vlib/v/gen/c/orm.v — C codegen for bulk operations

  • write_orm_bulk_insert: for non-serial fields, build a single QueryData with batch_rows and call conn.insert() once; for tables with serial fields, fall back to per-row inserts so each row gets its own auto-generated ID
  • write_orm_bulk_update: build per-column, per-row key+value data for CASE WHEN, with OR-connected WHERE conditions

vlib/db/mysql/orm.c.v — MySQL backend fix

  • convert_query_data_to_primitives: use i % data.fields.len to correctly map batch data elements back to field names for time.Time conversion
  • Construct QueryData with all fields propagated (including batch_rows, batch_key, parentheses, auto_fields)

vlib/v/tests/orm_bulk_insert_update_test.v — End-to-end compiler tests

  • test_orm_bulk_insert_and_update: insert 2 rows, verify, update both with CASE WHEN, verify
  • test_orm_bulk_insert_preserves_all_default_rows: insert 3 default rows, verify all get distinct IDs
  • test_orm_bulk_insert_with_mixed_serial_values_keeps_defaults: mix zero and explicit ID values in batch
  • test_orm_bulk_update_with_renamed_column: batch update with @[sql: 'display_name_text'] renamed column

vlib/v/checker/tests/orm_bulk_pointer_array_error.vv — Checker error test

  • Verify proper error messages when using pointer arrays ([&T]) for bulk insert/update

Testing

  • All 37 ORM unit/integration tests pass
  • The new orm_bulk_insert_update_test.v (4 test functions) passes
  • vlib/v/compiler_errors_test.v: 1565 passed, 5 skipped
  • vlib/v/gen/c/coutput_test.v: 95/95 .out files match, 117/117 .c.must_have patterns match
  • SQLite (in-memory) integration test for update_many API

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d7ba71c209

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread vlib/orm/orm_func.v
Comment on lines +1029 to +1030
mut batch := fill_data_with_struct[T](values[0], qb.meta)
batch.batch_rows = values.len
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve per-row default handling in insert_many

insert_many now unconditionally batches by setting batch_rows, which changes behavior for tables with serial/default columns: rows that used to omit the auto field on a per-row basis are now forced through one batched shape, so mixed explicit/zero IDs no longer preserve defaults and can write literal zero/default-breaking values. This regresses semantics from the previous per-row loop and should fall back to row-by-row inserts when auto/default fields are present (as the compiler SQL bulk path already does).

Useful? React with 👍 / 👎.

Comment thread vlib/orm/orm.v Outdated
Comment on lines 670 to 671
if sql_dialect in [.sqlite, .pg, .h2] && are_values_empty {
str += 'DEFAULT VALUES'
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Generate all rows for empty-field batch inserts

In the batch insert path, when all insert fields are skipped and batch_rows > 1, sqlite/pg/h2 still emit a single DEFAULT VALUES statement, so only one row is inserted instead of the requested batch size. Any caller that batches records containing only default/serial columns will silently lose rows; the SQL generation needs to account for row_count in this branch (or execute per-row statements).

Useful? React with 👍 / 👎.

@Jengro777
Copy link
Copy Markdown
Contributor Author

Failing doesn't seem to have anything to do with PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(orm): Add bulk insert and bulk update support

1 participant