Skip to content

Rebase the columnar fields and clustered indexes work onto the post-off-tokio tree - #104

Closed
pathscale wants to merge 4 commits into
feat/off-tokiofrom
feat/columnar-rebased
Closed

pathscale wants to merge 4 commits into
feat/off-tokiofrom
feat/columnar-rebased

Conversation

@pathscale

Copy link
Copy Markdown
Owner

PR #58 rebased onto feat/off-tokio (#103) and made green. #58 itself has been
CONFLICTING and untouched since 2026-08-06, with master 268 commits ahead of it.

Its two commits replay recognisably as the first two here; everything I had to
change to land them on the current tree is in two separate commits after, so the
adaptation can be audited apart from the original work.

The conflict was not what the diff suggested

The expensive change was not tokio::yield_now becoming worktable::prelude::yield_now.
That had already happened cleanly in files this never had to touch. The real
structural change is that codegen/src/common/{model,parser} was extracted into
the worktable_dsl crate, so every one of #58's schema-language files had to
move crates rather than merge. Thirteen conflicts, two of which needed judgement
rather than merging:

ColumnSlotIdExhausted rollback arms in reinsert and reinsert_cdc called
primary_index.insert(pk, old_link). On this tree the primary index is no
longer swung before the secondary-index work, so that rollback would have
written a spurious entry. Dropped in both, matching the sibling AlreadyExists
arm. Three further ColumnSlotIdExhausted arms were needed in paths added after
#58 was written.

Two bugs found while rebasing

worktable_dsl::schema mirrors the macro's section dispatch for the editor and
checker path, and had no columnar_indexes arm. A valid declaration parsed for
code generation and was then rejected by gen_schema_const. Fixed, and both
"expected one of" messages now list the section.

cargo check --no-default-features passes on feat/off-tokio and failed once
#58 landed: src/columnar.rs imported std::{collections, fmt, hash, sync} and
the generated side-index type named std::collections::BTreeSet and
std::mem::{take, replace}. Getting std out of the macro's contract is the
same goal as getting tokio out, so the module goes through alloc/core and
the emitted paths through the prelude. That is its own commit because CI may not
run that check and it would have shipped silently.

What the columnar lock actually costs

Worth stating plainly, because the PR description understates it and because it
decides which benchmarks have to be re-run.

A table that declares no columnar fields pays nothing, structurally rather
than approximately: every emitter in codegen/src/generators/columnar.rs opens
with if columns.columnar_fields.is_empty() { return quote!{} }, so no field is
declared, no lock constructed, and no #columnar_dirty token interpolated into
any update generator. Generated code for a non-columnar table is byte-identical
to pre-#58 output. So existing benchmark numbers stand and only columnar
benchmarks need re-running.

On a table that does declare columnar fields, every insert, delete, update,
in-place update and reinsert takes the exclusive lock. "Serializes concurrent
side-index writers" understates it: side-index writers are the whole
row-mutation path. And ensure_columnar_current() takes the writer lock
unconditionally on entry, before it can observe that dirty is false
(codegen/src/generators/columnar.rs:441-442), so concurrent readers of a clean
replica serialize against each other too. If this path is benchmarked, the
number that matters is row-mutation throughput on a columnar table against the
same schema without columnar.

Verification

feat/off-tokio here
cargo test 936 944
cargo test --workspace 1094 1116

Zero failures. cargo clippy --workspace --all-targets clean.
cargo tree -e normal -i tokio prints nothing; tokio stays a dev-dependency and
no tokio:: path was resurrected.

The PDF was not a merge problem: feat/off-tokio has neither .gitattributes
nor output/, so both applied as clean adds. Verified by blob hash rather than
by the status line, 38ac26c71522f5712560b3523acc1673bfddc942, 27746 bytes,
byte-identical to #58's.

Supersedes #58. Base is feat/off-tokio, so #103 lands first.

meh added 4 commits September 10, 2026 02:11
The rebase target moved the schema language into the `worktable_dsl`
crate and made row mutation async, so the columnar work needs adapting
rather than replaying:

- `codegen/src/common/{model,parser}` are `worktable_dsl` now, so the
  columnar model and parser modules move with them and drop their
  `crate::common::` paths. `type_name` and `name` were `pub(crate)`
  helpers inside one crate and have to be `pub` across two.
- `validate_columnar_indexes` joins the other rules in
  `worktable_dsl::validate` instead of living in the macro crate.
- `worktable_dsl::schema` mirrors the macro's section dispatch, so it
  gets the `columnar_indexes` arm too; without it a valid declaration
  parsed for code generation and was rejected by the schema constant.
- `IndexError::ColumnSlotIdExhausted` needs arms in the three rollback
  paths off-tokio added. Two of them unwind less than the columnar
  patch assumed: the primary index is no longer swung before the
  secondary work, so there is nothing to roll it back to.
- `insert` is async on this tree, so the columnar tests await it.
`cargo check --no-default-features` passed on the rebase target and
failed once the columnar work landed: `src/columnar.rs` imported
`std::{collections, fmt, hash, sync}`, and the generated side-index
data type named `std::collections::BTreeSet`/`BTreeMap` and
`std::mem::{take, replace}`.

Nothing here needs std. The module goes through `alloc` and `core`,
and the generated paths go through `worktable::prelude`, which is
where the rest of the emitted code already resolves its collections.
`BTreeSet` joins `BTreeMap` in the prelude so a generated type can
name it without the consumer taking a dependency.
@pathscale

Copy link
Copy Markdown
Owner Author

Folded into #105, which now carries these four commits as the first four of one linear branch. The two touch the same three section-dispatch loops, so splitting them meant resolving the same conflict twice.

@pathscale pathscale closed this Sep 9, 2026
@pathscale
pathscale deleted the feat/columnar-rebased branch September 9, 2026 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant