Conversation
added 4 commits
September 10, 2026 02:11
The rebase target moved the schema language into the `worktable_dsl`
crate and made row mutation async, so the columnar work needs adapting
rather than replaying:
- `codegen/src/common/{model,parser}` are `worktable_dsl` now, so the
columnar model and parser modules move with them and drop their
`crate::common::` paths. `type_name` and `name` were `pub(crate)`
helpers inside one crate and have to be `pub` across two.
- `validate_columnar_indexes` joins the other rules in
`worktable_dsl::validate` instead of living in the macro crate.
- `worktable_dsl::schema` mirrors the macro's section dispatch, so it
gets the `columnar_indexes` arm too; without it a valid declaration
parsed for code generation and was rejected by the schema constant.
- `IndexError::ColumnSlotIdExhausted` needs arms in the three rollback
paths off-tokio added. Two of them unwind less than the columnar
patch assumed: the primary index is no longer swung before the
secondary work, so there is nothing to roll it back to.
- `insert` is async on this tree, so the columnar tests await it.
`cargo check --no-default-features` passed on the rebase target and
failed once the columnar work landed: `src/columnar.rs` imported
`std::{collections, fmt, hash, sync}`, and the generated side-index
data type named `std::collections::BTreeSet`/`BTreeMap` and
`std::mem::{take, replace}`.
Nothing here needs std. The module goes through `alloc` and `core`,
and the generated paths go through `worktable::prelude`, which is
where the rest of the emitted code already resolves its collections.
`BTreeSet` joins `BTreeMap` in the prelude so a generated type can
name it without the consumer taking a dependency.
Owner
Author
|
Folded into #105, which now carries these four commits as the first four of one linear branch. The two touch the same three section-dispatch loops, so splitting them meant resolving the same conflict twice. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR #58 rebased onto
feat/off-tokio(#103) and made green. #58 itself has beenCONFLICTING and untouched since 2026-08-06, with master 268 commits ahead of it.
Its two commits replay recognisably as the first two here; everything I had to
change to land them on the current tree is in two separate commits after, so the
adaptation can be audited apart from the original work.
The conflict was not what the diff suggested
The expensive change was not
tokio::yield_nowbecomingworktable::prelude::yield_now.That had already happened cleanly in files this never had to touch. The real
structural change is that
codegen/src/common/{model,parser}was extracted intothe
worktable_dslcrate, so every one of #58's schema-language files had tomove crates rather than merge. Thirteen conflicts, two of which needed judgement
rather than merging:
ColumnSlotIdExhaustedrollback arms inreinsertandreinsert_cdccalledprimary_index.insert(pk, old_link). On this tree the primary index is nolonger swung before the secondary-index work, so that rollback would have
written a spurious entry. Dropped in both, matching the sibling
AlreadyExistsarm. Three further
ColumnSlotIdExhaustedarms were needed in paths added after#58 was written.
Two bugs found while rebasing
worktable_dsl::schemamirrors the macro's section dispatch for the editor andchecker path, and had no
columnar_indexesarm. A valid declaration parsed forcode generation and was then rejected by
gen_schema_const. Fixed, and both"expected one of" messages now list the section.
cargo check --no-default-featurespasses onfeat/off-tokioand failed once#58 landed:
src/columnar.rsimportedstd::{collections, fmt, hash, sync}andthe generated side-index type named
std::collections::BTreeSetandstd::mem::{take, replace}. Gettingstdout of the macro's contract is thesame goal as getting tokio out, so the module goes through
alloc/coreandthe emitted paths through the prelude. That is its own commit because CI may not
run that check and it would have shipped silently.
What the columnar lock actually costs
Worth stating plainly, because the PR description understates it and because it
decides which benchmarks have to be re-run.
A table that declares no columnar fields pays nothing, structurally rather
than approximately: every emitter in
codegen/src/generators/columnar.rsopenswith
if columns.columnar_fields.is_empty() { return quote!{} }, so no field isdeclared, no lock constructed, and no
#columnar_dirtytoken interpolated intoany update generator. Generated code for a non-columnar table is byte-identical
to pre-#58 output. So existing benchmark numbers stand and only columnar
benchmarks need re-running.
On a table that does declare columnar fields, every insert, delete, update,
in-place update and reinsert takes the exclusive lock. "Serializes concurrent
side-index writers" understates it: side-index writers are the whole
row-mutation path. And
ensure_columnar_current()takes the writer lockunconditionally on entry, before it can observe that
dirtyis false(
codegen/src/generators/columnar.rs:441-442), so concurrent readers of a cleanreplica serialize against each other too. If this path is benchmarked, the
number that matters is row-mutation throughput on a columnar table against the
same schema without
columnar.Verification
feat/off-tokiocargo testcargo test --workspaceZero failures.
cargo clippy --workspace --all-targetsclean.cargo tree -e normal -i tokioprints nothing; tokio stays a dev-dependency andno
tokio::path was resurrected.The PDF was not a merge problem:
feat/off-tokiohas neither.gitattributesnor
output/, so both applied as clean adds. Verified by blob hash rather thanby the status line,
38ac26c71522f5712560b3523acc1673bfddc942, 27746 bytes,byte-identical to #58's.
Supersedes #58. Base is
feat/off-tokio, so #103 lands first.