Vector multi-dtype storage, SQL CREATE INDEX, and physical plan refactor#115
Merged
Conversation
Introduce `VectorStorageDtype` (F32, F16, BF16, I8) as a first-class parameter on `HnswParams`, allowing vectors to be stored and searched in reduced-precision formats without an intermediate conversion step. Key changes: - `nodedb-types/src/vector_dtype.rs`: new dtype enum with serde support and backward-compatible default (F32) - `nodedb-vector/src/dtype/`: cast_from_f32 for encoding on insert, cast_to_f32 for decoding on results - `nodedb-vector/src/distance/typed_scalar.rs`: pure-Rust F16/BF16 distance kernels (L2, cosine, inner product) using the `half` crate - `nodedb-vector/src/distance/dispatch.rs`: unified distance dispatcher that routes to the correct dtype kernel via SimdRuntime - `nodedb-vector/src/distance/simd/`: per-arch typed SIMD kernels (AVX2, AVX-512, NEON, WASM-SIMD128) for F16 and BF16 - `nodedb-vector/src/hnsw/graph/`: HNSW graph split into index.rs and types.rs; `NodeStorage` enum replaces bare `Vec<f32>` on Node, storing either F32 or packed bytes for half/int8 dtypes - `nodedb-vector/src/rerank/`: re-ranking pipeline (gating, pipeline, sidecar, codecs) for approximate-then-exact search flows - `nodedb-vector/src/collection/sidecar_build.rs`: sidecar index construction helpers - Executor handlers updated to propagate `dtype` through HnswParams during lifecycle ops and WAL replay
`PhysicalPlan`, `PhysicalTask`, and `PostSetOp` were previously re-exported through `nodedb::bridge`. They now live in the dedicated `nodedb-physical` crate. Update all test files and test-support helpers to import from `nodedb_physical` directly, and add the crate dependency to `nodedb-cluster-tests` and `nodedb-test-support` Cargo manifests.
Delete `nodedb-client/src/native/client.rs` (615 lines) and `nodedb-client/src/remote/client.rs` (767 lines). Their responsibilities have been split into focused modules. Minor fixup to `document.rs` and cleanup of an unused import in `response_parse.rs`.
Replace `&mut Vec<Vec<Value>>` with `&mut [Vec<Value>]` across all window evaluation and aggregation functions. Slices express the actual contract (no push/pop needed) and satisfy clippy's ptr_arg lint. Also apply let-chain syntax to `set_cell` and sort a use-statement import in the aggregate planner.
Add the `half` crate (F16/BF16 types, bytemuck feature) to the workspace dependency table for use by the vector dtype kernels. Wire `bytemuck` and `half` into `nodedb-vector`'s Cargo.toml.
Introduce `codec_envelope` — a shared serialization wrapper (5-byte magic + 1-byte version + MessagePack body) used by quantization codecs that need to persist their calibrated parameters. Wire `BbqCodec` and `RaBitQCodec` into this envelope via `to_bytes` / `from_bytes` helpers, derive `Serialize`/`Deserialize` and the zerompk traits on both structs, and add roundtrip + rejection tests for bad magic and bad version on each codec.
…upsert path Add `storage_dtype: VectorStorageDtype` to `VectorPrimaryConfig` and thread it through the full insertion pipeline: - DDL parser (`vector_primary.rs`) reads the optional `storage_dtype` WITH option (f32 / f16 / bf16); defaults to F32. - `SqlPlan::VectorPrimaryInsert` and `VectorOp::DirectUpsert` carry the dtype field so every layer has a single, authoritative source of truth. - `PlanVisitor::vector_primary_insert` signature gains the dtype param; `VectorPrimaryInsertCfg` bundles field, quantization, dtype, and payload indexes to reduce call-site argument lists. - Executor validates dtype consistency on subsequent inserts into an existing index and seeds `vector_params.dtype` for new collections so the HNSW graph is constructed with the correct `NodeStorage` variant. - Export `nodedb_types::VectorStorageDtype` from the crate root and expose the `rerank` module from `nodedb-vector`. - Update tests and the sync-compat roundtrip to include the new field.
Add a new `index_ddl` planner module that converts `CREATE [UNIQUE] INDEX` and `DROP INDEX [IF EXISTS]` SQL statements into typed `SqlPlan` variants. Supported syntax: - `CREATE [UNIQUE] INDEX [IF NOT EXISTS] [name] ON collection (field)` - `COLLATE NOCASE / CI / CASE_INSENSITIVE` on the indexed column maps to `case_insensitive = true`; other collations are accepted and ignored - `DROP INDEX [IF EXISTS] name [ON collection]` Rejected with typed errors: multi-column indexes, expression indexes, and partial (`WHERE`) indexes. The parser classifier (`StatementKind`) gains `CreateIndex` and `DropIndex` arms so `plan_statements` routes them without a fallthrough. The `PlanVisitor` trait and dispatch function are extended with matching `create_index` / `drop_index` methods. The unsupported-arms macro in the Origin control-plane visitor returns a structured `PlanError` until the physical executor is wired up.
…ry namespaces Add a `SEARCH(field, query)` built-in scalar that routes through the existing `TextMatch` WHERE-search path, providing a pgvector-compatible alias for full-text search in function position. Extend `extract_float_array` to accept JSON-array string literals of the form `'[1.0, 0.5, 0.0]'` in addition to SQL `ARRAY[...]` and `make_array(...)` expressions. This allows embedding vectors to be passed as SQL string literals, matching the pgvector wire convention. Add `StrictHistory` (id 13) and `GraphHistory` (id 14) to the `Namespace` enum, covering the bitemporal history tables for strict document and graph edge collections.
The new leader must commit a NoOp entry to advance its commit-index past any SurrogateAlloc entries the dead leader left mid-replication before lagging survivors can apply them through the metadata Raft path. Under suite load this window can exceed the previous 20-second budget; align the wait with the post-failover settle budget used in step 6 (60 s) and document the reasoning in-line.
leowmjw
added a commit
to leowmjw/nodedb
that referenced
this pull request
May 22, 2026
…r wasm32 compat merge - Rebase onto upstream/main 86a6023 (44 commits, PR NodeDB-Lab#114 + NodeDB-Lab#115/lite merge) - Fix O_DIRECT guards: add #[cfg(target_os = "linux")] inside wasm32-guarded blocks (upstream wasm32 compat work changed structure, dropping inner Linux-only guards) - Fix posix_fadvise guard: #[cfg(unix)] -> #[cfg(target_os = "linux")] in atomic_io.rs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
leowmjw
added a commit
to leowmjw/nodedb
that referenced
this pull request
May 24, 2026
…r wasm32 compat merge - Rebase onto upstream/main 86a6023 (44 commits, PR NodeDB-Lab#114 + NodeDB-Lab#115/lite merge) - Fix O_DIRECT guards: add #[cfg(target_os = "linux")] inside wasm32-guarded blocks (upstream wasm32 compat work changed structure, dropping inner Linux-only guards) - Fix posix_fadvise guard: #[cfg(unix)] -> #[cfg(target_os = "linux")] in atomic_io.rs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
leowmjw
added a commit
to leowmjw/nodedb
that referenced
this pull request
May 26, 2026
…r wasm32 compat merge - Rebase onto upstream/main 86a6023 (44 commits, PR NodeDB-Lab#114 + NodeDB-Lab#115/lite merge) - Fix O_DIRECT guards: add #[cfg(target_os = "linux")] inside wasm32-guarded blocks (upstream wasm32 compat work changed structure, dropping inner Linux-only guards) - Fix posix_fadvise guard: #[cfg(unix)] -> #[cfg(target_os = "linux")] in atomic_io.rs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
storage_dtypepropagation through vector-primary DDL and upsert paths, versioned envelope for quantization codecs.CREATE INDEX/DROP INDEXplanning,SEARCHfunction alias, JSON vector literals, history namespaces.PhysicalPlan/PhysicalTaskimports tonodedb-physical, split monolithic native and remote client files, tighten window function signatures.halfandbytemuckas workspace dependencies.11 commits, 472 files changed (+12363 / -4017).