bug: schema plan refuses to drop a property that carries @index #1014

Activity

1. ragnorc

   added

   bugSomething isn't workingSomething isn't working

   needs-triageAwaiting maintainer triageAwaiting maintainer triage

   on Oct 16, 2026
2. ragnorc commented on Oct 17, 2026

   

   ragnorc

   on Oct 17, 2026 – with Cursor

   ContributorAuthor

   More actions

   Triage: reproduced on latest main; a planner-only fix is recommended

   Summary

   schema plan refuses to drop a property when any constraint names that property. The cause is in the planner, not in storage. A property-level @index is also stored as a type-level @index(score) constraint. When the property is dropped, the planner sees that constraint disappear and refuses the change. It does not check that the property the constraint names is dropped in the same plan.

   The same refusal applies to @unique, @range and @check on a dropped property, and to edge properties. The storage path already handles the drop correctly: Lance removes the index on a dropped column. A local prototype of a planner-only fix applies cleanly end to end.

   Reproduction

   Verdict: reproduced. The output is identical to the report.

   • Commit: main at 4d7536d68b1db6da34bf9cff22a90d9fdd77bed4. I checked it again at the end of triage. It is newer than the 864495a named in the report.
   • Build: cargo build -p omnigraph-cli --locked, debug profile, toolchain 1.97.1 from rust-toolchain.toml. The binary reports omnigraph 0.14.0.
   • Environment: Linux x86_64 (kernel 6.12), local filesystem storage, direct CLI.
   • Inputs: the reporter's v1.pg, drop-note.pg, drop-score.pg and seed.jsonl, with no changes.

   $ omnigraph init --schema v1.pg g.omni
   initialized g.omni
   $ omnigraph load --data seed.jsonl --mode append g.omni
   omnigraph load → g.omni (direct, local)
   loaded g.omni on branch main with append: 1 entities across 1 node types and 0 edge types
   $ omnigraph schema plan --schema drop-note.pg g.omni
   schema plan for g.omni
   supported: yes
   - drop property 'Doc.note' of node 'Doc'
   $ omnigraph schema plan --schema drop-score.pg g.omni
   schema plan for g.omni
   supported: no
   - drop property 'Doc.score' of node 'Doc'
   - unsupported change on node:Doc: removing constraints from 'Doc' is not supported in schema migration v1
   $ omnigraph schema apply --schema drop-score.pg g.omni
   omnigraph schema apply → g.omni (direct, local)
   removing constraints from 'Doc' is not supported in schema migration v1
   command_outcome: {"execution":"unknown","effects":"unknown","action":"reconcile"}
   [exit 1]
   $ omnigraph schema show g.omni
   node Doc {
       slug: String @key
       score: I64 @index
       note: String?
   }
   

   The result is the same on an empty graph (init, then plan and apply with no load).

   Differences from the report: I used a 0.14.0 source build, not the 0.13.0 release binary, and Linux x86_64, not macOS arm64. Neither difference changes the result. The planner code does not depend on the platform.

   Scope. I planned these variants with the same build:

   The dropped property carries	Plan on main
   @index on a scalar, a free-text String or a Vector	refused
   @unique	refused
   @range(p, ..) or @check(p, ..)	refused
   @unique(a, b), with a and b both dropped	refused
   @index on an edge property	refused
   @unique(a, b), with only a dropped	refused (correct)
   @key	refused (correct)
   no constraint	supported

   Removing a bare @index from a property that stays is refused twice: once by the property-level check and once by the type-level check.

   Minimal failing test

   GQT cannot express this contract. Its step set (Step in crates/omnigraph-gqt-core/src/lib.rs) has query, mutate, load, branch and commit control, settings, show, branch read, restart and concurrent steps. It has no schema plan or apply step, and the schema of a case is fixed at setup.

   The cheapest owner is the in-source planner test module in crates/omnigraph-compiler/src/catalog/schema_plan.rs, beside plan_emits_drop_for_removed_nullable_property. This test needs no graph and no I/O.

   #[test]
   fn plan_drops_indexed_property_with_its_index_issue_1014() {
       let accepted = ir(r#"
   node Doc {
       slug: String @key
       score: I64 @index
       note: String?
   }
   "#);
       let desired = evolve(
           &accepted,
           r#"
   node Doc {
       slug: String @key
       note: String?
   }
   "#,
       );
   
       let plan = plan_schema_migration(&accepted, &desired).unwrap();
       assert!(
           plan.steps.iter().any(|step| matches!(
               step,
               SchemaMigrationStep::DropProperty { type_name, property_name, .. }
                   if type_name == "Doc" && property_name == "score"
           )),
           "expected DropProperty Doc.score: {plan:?}",
       );
       assert!(
           plan.supported,
           "dropping a property must take its own @index with it: {plan:?}"
       );
   }

   Clean baseline first: cargo test -p omnigraph-compiler --locked --lib catalog::schema_plan:: gave 21 passed. Then, with only this test added:

   $ cargo test -p omnigraph-compiler --locked --lib issue_1014
   running 1 test
   test catalog::schema_plan::tests::plan_drops_indexed_property_with_its_index_issue_1014 ... FAILED
   
   dropping a property must take its own @index with it: SchemaMigrationPlan { supported: false, steps: [DropProperty { type_kind: Node, type_name: "Doc", property_name: "score" }, UnsupportedChange { entity: "node:Doc", reason: "removing constraints from 'Doc' is not supported in schema migration v1", code: None }] }
   
   test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 483 filtered out
   

   One test ran. It failed on the supported assertion. The DropProperty assertion passed.

   Root cause

   The accepted SchemaIR keeps a property-level constraint in two places: in PropertyIR::property_constraints, and as a type-level ConstraintIR that refers to the property by stable id. The planner compares these two places independently.

   1. plan_properties puts the dropped property in its leftover loop and emits DropProperty (schema_plan.rs#L717-L730). It does not call plan_property_constraints for that property, so the property-level path stays quiet.
   2. plan_constraints compares the type-level constraints by key (schema_plan.rs#L817-L850). @index(score) is in the accepted map but not in the desired map. Any removal emits UnsupportedChange. The function has no input that tells it which properties the plan drops.
   3. Apply refuses a plan with supported: false before any effect (schema_apply.rs#L168-L175).

   The apply path does not need this refusal:

   • DropProperty already sends the table to a metadata-only Operation::Project (schema_apply.rs#L426-L444). The doc comment on stage_schema_evolution already says "an index on a dropped column leaves with it" (table_store.rs#L4640-L4651).
   • In the pinned Lance 11.0.0, Operation::Project calls retain_relevant_indices. That function removes each index whose field is no longer in the schema (manifest_build.rs#L912-L931, index_maintenance.rs#L221-L238). The surface guard metadata_only_schema_evolution_keeps_blob_files_ids_and_index_coverage pins this behavior (lance_surface_guards.rs#L811-L821). It passes on this build.
   • Apply takes the full-text declarations of an evolved table from the desired catalog (declared_full_text_columns), so it gives a dropped column no analyzer segment.
   • After the drop, @unique, @range and @check on that property have no value left to check.

   So the refusal is a gap in how the planner classifies the change. It is not a safety rule.

   Recommended fix (correct by design)

   Treat a constraint as dependent on the properties it names. When the plan ends the lifetime of every property a constraint names, the constraint ends with them.

   • In plan_nodes and plan_edges, compute the set of dropped StablePropertyIds once per type. Use the same stable ids that plan_properties uses. Do not compare names (invariant 6).
   • Pass that set to plan_constraints. Do not refuse a removed constraint when every field it names is a property in the dropped set.
   • Continue to refuse these changes:
     • every @key change, because entity ids come from the key;
     • a constraint that names a property that stays, or a system field (@src, @dst). An example is @unique(a, b) with only a dropped;
     • removal of a constraint from a property that stays (see the maintainer question below).
   • Apply needs no change.
   • Write the rule on SchemaMigrationStep::DropProperty and in the supported-changes text of docs/user/schema/index.md. Add a changelog.d fix note.

   I made a local prototype of this rule (about 40 lines in schema_plan.rs). I did not push it. Results on the same commit:

   • The failing test passes. cargo test -p omnigraph-compiler --locked --lib gives 484 passed.
   • A temporary engine test in crates/omnigraph/tests/schema_apply.rs loads rows and runs ensure_indices. Then it drops score: I64 @index and body: String? @index and reads the Lance indexes of the pinned table:

     indexed columns before: ["__id", "body", "score", "slug"]
     indexed columns after: ["__id", "slug"]
     

     ensure_indices and optimize after the apply do not change that set, and a query returns every row. The complete --test schema_apply binary gives 33 passed.
   • CLI journey: the reporter's apply now prints supported: yes and applied: yes. The same is true for @unique, @range, @check, a free-text String, a Vector, a fully dropped @unique(a, b) and an edge property. A partial composite, a @key drop and a bare @index removal are still refused.

   Tests to land with the fix:

   • the planner test above, plus refusal rows for a partial composite, a @key drop and a bare removal. Put issue_1014 in the test name for the Fix Regression Gate;
   • an engine test in schema_apply.rs that asserts the pinned table has no index on the dropped column, and that ensure_indices does not build one again.

   I expect no change to the DST storage-cost golden. The apply reads and writes the same storage as an unindexed drop.

   Alternatives

   Option	Cost	Liability
   A. Dependency rule in the planner (recommended)	one function and one argument	Low. One rule, keyed on stable ids, covers every constraint kind.
   B. Emit an explicit DropConstraint step, or list the removed constraints on DropProperty	new serde variant, OpenAPI change, CLI renderer change	The plan output is more transparent, but this is a wire-format change for information the drop already implies. It can follow later.
   C. Special-case only a single-property @index	smallest diff	High. @unique, @range, @check and composites stay broken, and each new constraint kind needs one more special case.
   D. Document the refusal as intended	docs only	Users must keep a deprecated column forever. This conflicts with invariant 7 and with the documented supported changes.

   The quality that option A deprioritizes is plan transparency. The plan line "drop property 'Doc.score'" does not name the constraints that leave with it. The property is gone, so no constraint can still apply to it. Option B can add that detail later without a change in behavior.

   The open maintainer question: removing a bare @index from a property that stays

   Recommendation: keep this change refused in the fix for this issue, document the exception, and decide it in its own issue or RFC. The reasons are different for each index kind.

   • Scalar and vector indexes. Removal changes only cost. But apply must then do something with the physical index.
     • If apply keeps it, the table holds an index that no declaration describes. Index maintenance builds from declarations (ensure_indices reads node_type.indices), so nothing refreshes it. Results stay correct, because Lance scans the fragments an index does not cover. But the index drifts, and I did not check whether the v2 planner still uses it on reads.
     • If apply drops it, schema apply gets a new table effect: Lance drop_index, which is a CreateIndex commit with removed_indices. It must be staged detached and pinned in the same publication, with crash-window rows in detached_commit_matrix.rs.
     • Draft RFC 0046 (docs/rfcs/0046-index-status.md) already expects declared: false indexes whose intent "was dropped by a schema change". So this direction fits the roadmap.
   • Free-text String indexes. Here the index is not only performance state. In this engine, the full-text segment carries the analyzer. With no segment, Lance's flat path tokenizes the text bare, with no lowercasing and no stemming (table_store.rs#L3961-L3975). The test full_text_analyzer_survives_schema_evolution_issue_904 shows that search(body, "deep") then misses "Deep Learning" (schema_apply.rs#L3146-L3155). So removing @index from a free-text String changes the results of search, match_text, fuzzy and bm25. If apply keeps the old segment instead, the physical index becomes the only record of the query semantics. That is a shadow source of truth.
     • The open RFC PR rfc: analyzed lexical search on the shared expression model #792 (analyzed lexical search) moves the analyzer into the schema with @analyzed. After that change, removing a full-text @index changes only cost. See also Unindexed full-text search is case-sensitive #747, fuzzy() bypasses the index analyzer at a nonzero edit budget #748 and Index state changes which rows a text query matches #749.

   Until that decision, the supported-changes list in docs/user/schema/index.md (line 155) should state that removing a constraint from a property that stays is refused. Line 172 says "adding or removing most constraints are rejected". That is close, but it does not name this case.

   Separate point: execution: unknown on a refused apply

   The engine marks this refusal as before-effect: capture(...).map_err(OmniError::before_effect) (prepared.rs#L283-L285). The CLI does not read that evidence. Failure::classify decides not_started only from HTTP dispatch evidence, so a direct write that opened a writable handle reports unknown (command_outcome.rs#L207-L209).

   This behavior is documented as conservative: "writable open can complete earlier work" (docs/user/operations/troubleshooting.md, line 99). Qualifying direct-mode outcomes is the scope of #466. #976 fixed only the served 401, 403 and contract-400 cases. I suggest tracking this point in #466, not here. It needs proof that the writable open itself did no work, and that proof is wider than this bug.

   Lance upstream assessment

   The pinned version is Lance 11.0.0 (docs/dev/lance.md). No upstream change fixes or blocks this issue. The defect is in the OmniGraph planner, and the Lance behavior that the fix needs already exists in 11.0.0.

   • retain_relevant_indices keeps the same "remove indices for fields no longer in schema" rule in v13.0.0 (the latest release) and in v14.0.0-beta.19. In the v14 beta, Operation::Project still calls it, and also calls a new retain_relevant_overlays for dropped fields.
   • lance#10107 (open) changes which empty segment retain_relevant_indices keeps when every segment of an index is empty. It does not touch the dropped-field rule.
   • lance#8208 (merged before 11.0.0) keeps empty vector index definitions. It matters for the untrained full-text segments that OmniGraph publishes, not for drops.
   • lance#7158 (merged before 11.0.0) makes alter_columns with a cast fail on an indexed column, instead of silently dropping the index. Upstream treats a silent index loss on a cast as a defect, but a column drop still removes its index by design. OmniGraph does not cast here.
   • lance#8646 (open, Transaction V2 actions) can change how Project-like operations are expressed later. The existing surface guard covers this at a future bump. No action is necessary now.

   The Lance data-evolution guide confirms that drop_columns is metadata-only and reversible while old versions are retained.

   Related work

   • docs(schema): OG-DS-102/103/104 are listed as refusals, but populated drops plan as supported #984: the drop documentation lists rejection codes OG-DS-102/103/104 that nothing emits. It is in the same docs section, but it is a different defect. It is not a duplicate.
   • schema: support adding required, unique properties to existing types #721: adding required, unique properties to existing types. Same planner, opposite direction (adding a constraint).
   • PR rfc: schema apply with live branches #963 (RFC: schema apply with live branches): merges bring a newer contract into a branch with the same metadata-only commits that schema apply uses. The fix here adds no table effect, so it composes with that RFC with no change.
   • PR rfc: analyzed lexical search on the shared expression model #792 (RFC: analyzed lexical search): relevant to the bare-@index question above.
   • RFC 0046 (draft, index status): relevant to undeclared physical indexes.

   Supersession

   No open issue supersedes this one. #984 is related, but it describes a different defect. I am not closing anything.

   Limitations

   • I did not test the 0.13.0 release binary or macOS arm64. I tested a 0.14.0 source build on Linux x86_64.
   • The prototype fix and the engine test are local and not reviewed. I did not run the full workspace gate, clippy, the failpoint or DST suites, or S3 and Azure storage.
   • I could not show a historical read of the dropped values through a query. By design, a historical query is type-checked against the current schema (docs/user/branching/index.md, "Historical reads"). An unindexed drop on main behaves the same way.
3. ragnorc

   closed this as completedin ff9519fon Oct 18, 2026
4. ragnorc

   added 2 commits that reference this issue on Oct 18, 2026

   fix(schema): drop a property together with the constraints that name …

   ...

   This issue will close once commit ff9519f is merged into the 'main' branch.

   ff9519f

   test(cleanup): qualify sweep retries across storage and DST scheduling (

   ...

   This issue will close once commit 366be88 is merged into the 'main' branch.

   366be88