Repository navigation
feat(doc-ir): carry per-cell confidence across the doc.v1 bridge - #229
Merged
Merged
Conversation
…ready) Reconciles the D-OGAR-DOC-LAYER charter (#191) with the as-built D-DOC-IR-SECOND-RETINA W1/W2/W3 (ogar-doc-ir #197, ogar-from-docv1 #202, spider_doc_ir). W4 is the persistence half: persist a DocIr as a GUID-keyed SoA subtree + raw KV reference, and reconstruct an updated PDF from it. - Input reconciled to ogar_doc_ir::DocIr (the neutral IR both retinas emit), not doc.v1 directly; the IR already pins the persistence facet mapping. - Two minimal new mints, canon-high, council-gated: typed_field 0x080A (v2-reserved) + document 0x080B. Page/region reuse PAGE_LAYOUT 0x0807, figures reuse PAGE_IMAGE 0x0808, cells are facets (no new class). - Three facts-only ActionDefs (External kausal): persist_document / read_document / reconstruct_document. - Renderer = DocRenderer trait (runtime-bound legs), template = ClassView x WideFieldMask (A1-A5 rulings, no new DSL). - OGAR-AS-IR section 3 + SURREAL-AST-TRAP answered; P-XRETINA idempotence is the pre-persist gate. Mints NOTHING: the canon mints land WITH the council-verified build on the operator's word (charter section D3). This doc is the blueprint that gate unlocks. Co-Authored-By: Claude <noreply@anthropic.com>
…ial verify pass Rewrites the doc-layer W4 spec against the as-built A2UI screen-addressing wave (#204-#207) and the a2ui-rs consumer (#3/#4/#9/#11), then applies a 17-finding adversarial verify pass (3 refuter lenses: citations, canon, layer boundaries). Substantive changes over v1: - W4-0 (new): wave-namespace rule — a2ui owns its own W0-W5 (its W4 = P-REHOST); this wave is written doc-W4 in all board/PR text. - Headline generalized per amendment A3: one ClassView x WideFieldMask template projection — re-issue as PDF (executor leg) or serve as a living addressed screen (a2ui session tier). - reconstruct_document stays produces=pdf_bytes, deliberately scoped to artifact legs; the living screen is session-tier project_node -> NodeDelta frames, never an ActionDef output. - C1.4 mask discipline inherited verbatim (RBAC by projection fail-closed on the re-issue path, retype-in-place, zero-extension, sentinel ban, permit-all identity consumed not re-decided). - Ordinal-stability wire invariant for the three ActionDefs (T2). - Four-leg renderer table with honest per-leg as-built status; DocRenderer trait = three impls (byte artifacts), a2ui = fourth family member at the session tier; deviation D-1 numbered (askama leg = compile-time callee, charter A5 runtime-bound annotation revised on stated factual basis). - A1 dep list restored to ratified A5 (ogar-vocab + ogar-doc-ir; codebook membrane returned to the consumer side of the BBB). - Page-register carving named (never-widen canon): number = one u8:u8 typed pair; native w/h out-of-line provenance. - Dependency ledger: value-slab render path, permit-all decision, HTML->PDF step — named so the council can't assume unbuilt legs whole. - P-TEMPLATE-MASK falsifier (new), split cross-repo over one shared fixture (dependency-direction honest). Mints NOTHING: 0x080A/0x080B stay operator-gated (D-OGAR-DOC-LAYER ledger; charter section D3) and land WITH the council-verified build. Co-Authored-By: Claude <noreply@anthropic.com>
TableCell gains `confidence: u8`, and `from_doc_v1` populates it. Why it was missing and why that mattered: `TypedField` already carried a confidence byte, but `TableCell` did not — and the values a consumer actually imports (a lab result, an invoice line amount) are CELLS, not typed fields. So per-cell OCR confidence died at this bridge and a downstream review gate had nothing to gate on. Worse, `V1Cell` is a plain `Deserialize` with no `deny_unknown_fields`, so a producer emitting `conf` was silently discarded: the omission was invisible from both sides. Conversion is the identical `.round().clamp(0.0, 255.0) as u8` the `TypedField` arm already uses, so the two confidence surfaces stay comparable. Read through `DocIr::source` exactly like `TypedField`'s — for `Provenance::Ocr` this is recognition certainty. `#[serde(default)]` on `V1Cell::conf` keeps legacy `doc.v1` payloads parsing; they read 0. That is deliberately FAIL-CLOSED: a payload that carries no cell confidence routes everything to human review rather than auto-committing at an assumed score. `0` must never be read as "measured zero-confidence" — it is indistinguishable from "not reported", and the field docs say so. The producer aggregates as the MINIMUM over a cell's words, not the mean: one misread digit invalidates a whole value, and a mean dilutes exactly the signal the gate exists to catch (a cell of 95 and 40 reports 40, not 67). Documented on the field so a future producer does not "improve" it to an average. The bridge test now asserts BOTH paths, which is why its fixture gives cell 0 a `"conf":97.0` and deliberately withholds one from cell 1 — the legacy default is asserted reachable, not merely assumed. Note for the in-binary case: this makes the JSON path lossless, but that path is itself avoidable when tesseract-rs and OGAR compile together — constructing DocIr directly from the producer's own types removes the V1* mirror structs entirely, and with them this whole class of silently-dropped-field bug. This fix is for the out-of-process wire. Co-Authored-By: Claude <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_6e979c79-b293-4046-99fd-0f6c48cafbfc) |
AdaWorldAPI
marked this pull request as ready for review
July 29, 2026 14:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Carries per-cell OCR confidence across the
doc.v1→DocIrbridge.The gap
TypedFieldalready carried aconfidencebyte.TableCelldid not — and the values a consumer actually imports are cells, not typed fields: a lab result, an invoice line amount. So per-cell OCR confidence died at this bridge, and a downstream review gate had nothing to gate on.Worse, it was invisible from both sides:
V1Cellis a plainDeserializewith nodeny_unknown_fields, so a producer emittingconfhad it silently discarded. Fixing only the producer would have changed nothing.The change
ogar-doc-ir::TableCellgainsconfidence: u8, read throughDocIr::sourceexactly likeTypedField::confidence— forProvenance::Ocrthis is recognition certainty.ogar-from-docv1::V1Cellgains#[serde(default)] conf: f32, converted with the identical.round().clamp(0.0, 255.0) as u8theTypedFieldarm already uses, so the two confidence surfaces stay comparable.Two decisions worth knowing
0is fail-closed, not "measured zero".#[serde(default)]keeps legacydoc.v1payloads parsing; they read0. That is deliberate — a payload carrying no cell confidence routes everything to human review rather than auto-committing at an assumed score.0is indistinguishable from "not reported" and the field docs say so explicitly.The producer aggregates as MIN over a cell's words, not mean. One misread digit invalidates a whole value; a mean dilutes exactly the signal the gate exists to catch (a cell of 95 and 40 reports 40, not 67). Documented on the field so a future producer doesn't "improve" it to an average.
Why this matters concretely
Measured upstream this session: an unevenly-lit page recognized at
mean_conf 99.47with an actual CER of 0.6154 — 99.5% confident, 62% wrong. Confidence that never reaches the consumer can't gate anything; confidence that arrives at cell granularity can.Tests
The bridge test asserts both paths, which is why its fixture gives cell 0 a
"conf":97.0and deliberately withholds one from cell 1 — the legacy default is asserted reachable, not merely assumed. 16 + 4 tests green,clippy -D warningsclean.Note on the in-binary case
This makes the JSON path lossless, but that path is itself avoidable when tesseract-rs and OGAR compile together. Constructing
DocIrdirectly from the producer's types removes theV1*mirror structs entirely — and with them this whole class of silently-dropped-field bug, rather than this one instance.DocIris a quantizing projection (BBoxRailis 4 bytes,rail_bboxmaps pixels onto0..=255), so it isn't zero-copy, but it can be one projection instead of three.This fix is for the out-of-process wire, where the copy is genuinely required.
Producer side: AdaWorldAPI/tesseract-rs#59.