Repository navigation
docs: the ogar-doc INGESTION SPINE — transferred production experience (council input) - #266
Conversation
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_138c7a75-0b4f-436f-9cc6-d84229296e64) |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_4ecac5ec-fd6e-4b87-bca7-63935ec76bd8) |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5ffc511da3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| We can do better than suggest-vs-apply as a binary, because our extraction | ||
| carries verification that theirs cannot: `HarvestedField.checks` records which | ||
| checks passed (`arithmetic_ok`, `iban_mod97_ok`; empty = harvested but | ||
| unverified), and confidence aggregates as a **MIN** over joined words — chosen |
There was a problem hiding this comment.
When a recognized document crosses the existing recognize_document → ogar-from-docv1 → DocIr path, this proposed auto-apply gate cannot be evaluated: V1Field in crates/ogar-from-docv1/src/lib.rs:132-138 does not deserialize checks, the conversion at lines 253-262 drops them, and TypedField in crates/ogar-doc-ir/src/lib.rs:163-177 has no field for them. Consequently, after persistence a booking consumer either cannot auto-apply anything or must fall back to confidence/match strength despite this section claiming that checks makes automatic posting safe; the spine needs to require carrying the verification evidence through the IR and W4 subtree.
Useful? React with 👍 / 👎.
| **Lands:** a preflight gate keyed on `content_sha256` **ahead of** the OCR | ||
| ActionDefs, not only inside `persist_document`. The convergence key already | ||
| exists (`DocIr.content_sha256`); the lesson is *where it is consulted*. |
There was a problem hiding this comment.
When a renderer-produced PDF or other derived export is re-ingested, a gate keyed only on DocIr.content_sha256 still misses the duplicate that this section specifically intends to catch. The field is documented in crates/ogar-doc-ir/src/lib.rs:348-356 as the hash of the original acquisition only, so the export's bytes have a different hash; moving that lookup ahead of OCR handles identical-file retries but not the derived-artifact case described at lines 51-54. The preflight design must also retain derived-artifact hashes and compare the incoming digest against both hash sets.
Useful? React with 👍 / 👎.
OGAR-DOC-W4-BUILD-SPEC is strong on the half it covers (persist / read / reconstruct, content_sha256 idempotency, the SoA subtree, the four-leg renderer) but it BEGINS ONE STAGE TOO LATE: it starts from a DocIr that already exists. Nothing in OGAR covers the stage where a file arrives and something must decide what it is, whether it needs recognition at all, and in what order the work happens. That stage is where a document system accumulates its scar tissue, and this stack has none on that axis. paperless-ngx has run exactly that stage in production for roughly a decade. This document extracts its spine - the ordering constraints, the centralisation rules, and the failure modes it fixed - as 13 invariants over four axes: ingestion ordering (S-1..S-5), escalation (S-6), rules (S-7..S-10), business identity (S-11..S-13). Every claim is cited to source and every one is graded [S]: transferred claim, NOT measured in this stack. It is borrowed operating time, not a finding, and the document says so in its own header. The two invariants that would be expensive to retrofit: S-2 - dedup is consulted BEFORE recognition spend, not only inside persist_document, and it matches the incoming hash against the derived-artifact checksum as well as the original (verified: consumer.py's Q(checksum=...) | Q(archive_checksum=...)). W4's content_sha256 idempotency prevents a duplicate SUBTREE; it does not prevent a duplicate SPEND, and for OCR that asymmetry is seconds-to-minutes of compute per re-ingest. S-5 - the write order between the KV blob and the subtree decides whether a failure leaves collectable garbage or an undetectable dangling reference. paperless creates the row inside a transaction and writes bytes after, under a FileLock, so a failure can never leave an addressable document with missing bytes. W4 does not currently answer this question; the document frames it as a question rather than asserting a defect. S-4 is the one entry better than [S]. "One decision, two consumers, ONE function" is independently corroborated in-stack: tesseract-rs's region_is_table had to become one shared primitive for exactly the reason is_born_digital_text did (their GH #13387, cited in its own docstring - two independent computations of one decision drifted apart silently). Two projects, same shape, same fix, so the rule grades [H]. Explicitly NOT transferred, with reasons (section NT): the sklearn estimator stack (duplicates deepnsm / lance-graph-arm-discovery, violates the no-serialization-in-hot-path rule ADR-022/023); the filing data model (correspondent/tags/document_type is a competing identity model against classid -> ClassView -> facet rails); storage-path templating (consumer's per W4-8); and the dual store, whose cost is measured in their own retry policy (autoretry_for=(SearchIndexLockError,), max_retries=5, retry_backoff=60 - lock contention that exists BECAUSE the index is a second authority). Operator-directed deferral recorded: a tantivy inverted index is NOT adopted as the search layer. Typed fields with provenance make the dominant queries structural - the SoA columns already answer "invoices from supplier X in period Y" - where paperless is forced into full-text search because its extraction stops at a flat content blob. If free-prose retrieval is wanted later for a specific use case, it indexes the out-of-line value-slab keyed by document_guid: a lens, never a parallel source of truth. Falsifiers named where cheap: S-2 (ingest the same page twice; the second pass must be ~0 elapsed or the gate is missing) and S-8 (express real SKR03 booking rules with CONTAINS/LIST_CONTAINS/EQ only - if REGEX/FUZZY prove load-bearing rather than convenience, the ogar-loco FnIndex additions are not optional and the council should know that before ratifying the mapping). Sections C and D are not W4's business - they belong to the rule layer (ogar-loco, where the rule-bearing object is the Konto exactly as it is the Correspondent there) and to the consumer. They are recorded so the council can see the full arc the document layer sits inside, and so typed_field 0x080A is minted knowing what it will eventually have to carry. Documentation only. No crate, no mint, no code. DISCOVERY-MAP entry D-OGAR-DOC-SPINE appended per CLAUDE.md.
5ffc511 to
63a1a9e
Compare
Summary
OGAR-DOC-W4-BUILD-SPECis strong on the half it covers — persist / read /reconstruct,
content_sha256idempotency, the SoA subtree, the four-legrenderer. It begins one stage too late: it starts from a
DocIrthatalready exists. Nothing in OGAR covers the stage where a file arrives and
something must decide what it is, whether it needs recognition at all, and in
what order the work happens.
That stage is where a document system accumulates its scar tissue, and this
stack has none on that axis.
paperless-ngxhas run exactly that stage inproduction for roughly a decade. This adds
docs/OGAR-DOC-INGESTION-SPINE.md— its spine extracted as 13 invariants, cited to source, so the W4 council
can check the build against experience we have not had to pay for.
Documentation only. No crate, no mint, no code.
Grading discipline
Every invariant is
[S]— transferred claim, NOT measured in this stack.The document says so in its own header. It is borrowed operating time, not a
finding. Falsifiers are named where cheap.
One exception, and it is better than
[S]:S-4("one decision, twoconsumers, ONE function") is independently corroborated here —
tesseract-rs'sregion_is_tablehad to become one shared primitive forexactly the reason
is_born_digital_textdid (their GH #13387, cited in its owndocstring: the two used to be computed independently and drifted apart
silently). Two projects, same shape, same fix → the rule grades
[H].The two that would be expensive to retrofit
S-2 — dedup is consulted BEFORE recognition spend. Their preflight checks
the incoming SHA-256 against both the original and the derived-artifact
checksum (verified:
Q(checksum=checksum) | Q(archive_checksum=checksum)),before any parsing. W4's
content_sha256idempotency sits insidepersist_document— after recognition has run. It prevents a duplicatesubtree; it does not prevent a duplicate spend, and for OCR that asymmetry
is seconds-to-minutes of compute per re-ingest.
S-5 — write order between the KV blob and the subtree. They create the row
inside a transaction and write bytes after, under a
FileLock, so a failure cannever leave an addressable document with missing bytes. W4's raw-ref points at a
consumer-owned blob store; if the subtree lands and the KV put fails, the graph
holds a resolvable
document_guidpointing at nothing. Framed as a questionW4 does not currently answer, not as a defect in it — the defensible answer is
bytes first, address second (a failed put leaves collectable garbage rather than
an undetectable dangling reference).
Explicitly NOT transferred (§NT), with reasons
MLPClassifier/CountVectorizer/pickle.Duplicates
deepnsm/lance-graph-arm-discovery, violatesno-serialization-in-hot-path (ADR-022/023). Take the meta-architecture
(versioned artifact, retrain-skip via label-assignment hash), never the
estimator. Our replacement is better on the axis that matters: an association
rule is its own explanation, so a discovered rule materialises into the
same
ogar-locorepresentation as a hand-authored one — somethingMATCH_AUTOvsMATCH_LITERALstructurally cannot do.correspondent/tags/document_typeis acompeting identity model against
classid → ClassView → facet rails.W4-8.autoretry_for=(SearchIndexLockError,), max_retries=5, retry_backoff=60.Lock contention that exists because the index is a second authority.
Deferral recorded (operator-directed)
A
tantivyinverted index is not adopted as the search layer. Typed fieldswith provenance make the dominant queries structural — the SoA columns
already answer "invoices from supplier X in period Y", and
palette256answerssimilarity deterministically at ρ=0.9973 vs cosine where BM25 is merely lexical.
Paperless leans on full-text search because its extraction stops at a flat
contentblob; ours does not. If free-prose retrieval is wanted later for aspecific use case, it indexes the out-of-line value-slab keyed by
document_guid— a lens, never a parallel source of truth.Scope note
§C and §D (rules, business identity) are not W4's business — they belong to
the rule layer (
ogar-loco, where the rule-bearing object is the Kontoexactly as it is the Correspondent there; the chart of accounts, booking
validation and DATEV export already exist in
woa-rs) and to the consumer.Recorded here so the council sees the full arc the document layer sits inside,
and so
typed_field 0x080Ais minted knowing what it will eventually carry.Verification
2a8579f: the dual-checksum dedupQ(), the#13387docstring, theSearchIndexLockErrorretry policy, the--deskew is not compatible with --redo-ocrcomments,StopConsumeTaskError, the versioning fields + partial unique constraintpaperless-ngxfork confirmed clean upstream (no local divergence) — read as upstreamDISCOVERY-MAPentryD-OGAR-DOC-SPINEappended per CLAUDE.md §D9git diffvs the remote tip shows additions only, no work lost🤖 Generated with Claude Code