Skip to content

Commit 5ffc511

Browse files
committed
docs: the ogar-doc INGESTION SPINE - transferred production experience
OGAR-DOC-W4-BUILD-SPEC is strong on the half it covers (persist / read / reconstruct, content_sha256 idempotency, the SoA subtree, the four-leg renderer) but it BEGINS ONE STAGE TOO LATE: it starts from a DocIr that already exists. Nothing in OGAR covers the stage where a file arrives and something must decide what it is, whether it needs recognition at all, and in what order the work happens. That stage is where a document system accumulates its scar tissue, and this stack has none on that axis. paperless-ngx has run exactly that stage in production for roughly a decade. This document extracts its spine - the ordering constraints, the centralisation rules, and the failure modes it fixed - as 13 invariants over four axes: ingestion ordering (S-1..S-5), escalation (S-6), rules (S-7..S-10), business identity (S-11..S-13). Every claim is cited to source and every one is graded [S]: transferred claim, NOT measured in this stack. It is borrowed operating time, not a finding, and the document says so in its own header. The two invariants that would be expensive to retrofit: S-2 - dedup is consulted BEFORE recognition spend, not only inside persist_document, and it matches the incoming hash against the derived-artifact checksum as well as the original (verified: consumer.py's Q(checksum=...) | Q(archive_checksum=...)). W4's content_sha256 idempotency prevents a duplicate SUBTREE; it does not prevent a duplicate SPEND, and for OCR that asymmetry is seconds-to-minutes of compute per re-ingest. S-5 - the write order between the KV blob and the subtree decides whether a failure leaves collectable garbage or an undetectable dangling reference. paperless creates the row inside a transaction and writes bytes after, under a FileLock, so a failure can never leave an addressable document with missing bytes. W4 does not currently answer this question; the document frames it as a question rather than asserting a defect. S-4 is the one entry better than [S]. "One decision, two consumers, ONE function" is independently corroborated in-stack: tesseract-rs's region_is_table had to become one shared primitive for exactly the reason is_born_digital_text did (their GH #13387, cited in its own docstring - two independent computations of one decision drifted apart silently). Two projects, same shape, same fix, so the rule grades [H]. Explicitly NOT transferred, with reasons (section NT): the sklearn estimator stack (duplicates deepnsm / lance-graph-arm-discovery, violates the no-serialization-in-hot-path rule ADR-022/023); the filing data model (correspondent/tags/document_type is a competing identity model against classid -> ClassView -> facet rails); storage-path templating (consumer's per W4-8); and the dual store, whose cost is measured in their own retry policy (autoretry_for=(SearchIndexLockError,), max_retries=5, retry_backoff=60 - lock contention that exists BECAUSE the index is a second authority). Operator-directed deferral recorded: a tantivy inverted index is NOT adopted as the search layer. Typed fields with provenance make the dominant queries structural - the SoA columns already answer "invoices from supplier X in period Y" - where paperless is forced into full-text search because its extraction stops at a flat content blob. If free-prose retrieval is wanted later for a specific use case, it indexes the out-of-line value-slab keyed by document_guid: a lens, never a parallel source of truth. Falsifiers named where cheap: S-2 (ingest the same page twice; the second pass must be ~0 elapsed or the gate is missing) and S-8 (express real SKR03 booking rules with CONTAINS/LIST_CONTAINS/EQ only - if REGEX/FUZZY prove load-bearing rather than convenience, the ogar-loco FnIndex additions are not optional and the council should know that before ratifying the mapping). Sections C and D are not W4's business - they belong to the rule layer (ogar-loco, where the rule-bearing object is the Konto exactly as it is the Correspondent there) and to the consumer. They are recorded so the council can see the full arc the document layer sits inside, and so typed_field 0x080A is minted knowing what it will eventually have to carry. Documentation only. No crate, no mint, no code. DISCOVERY-MAP entry D-OGAR-DOC-SPINE appended per CLAUDE.md.
1 parent 8d31552 commit 5ffc511

2 files changed

Lines changed: 409 additions & 0 deletions

File tree

‎docs/DISCOVERY-MAP.md‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1768,3 +1768,71 @@ isolation. The map's job is to keep them visible.
17681768
EXACTLY ONE), Klickwege wiring second (W2), PowerAutomate-shaped skin third
17691769
(W3) — both skins Mario-editor ergonomics over `ClassView : WideFieldMask`,
17701770
which is T1 applied at editor scale.
1771+
1772+
- **D-ELK-FACTFINDER (`ogar-elk` — the EL subsumption closure as the third
1773+
factfinder; 2026-08-07; [G], CODED, operator-directed):** `ogar-obo` and
1774+
`ogar-ro` say what is **asserted**; nothing said what **follows**.
1775+
`ogar-elk` closes that gap with the smallest calculus that does the job —
1776+
three rules (R1 reflexivity, R2 transitivity, R3 merge-soundness) over
1777+
ABI-shaped `(classid, identity)` addresses. It answers exactly two
1778+
questions: does `A ⊑ B` follow, and is adding a set of axioms to an
1779+
existing closure sound. **Ungraded by construction** — an EL entailment is
1780+
a fact, so nothing here is scored, ranked or weighted; the thinking that
1781+
consumes these facts lives one layer out. **Addresses, never a file:** the
1782+
crate never parses an ontology, resolves a CURIE, or reads a label —
1783+
reasoning over the addressed form is the point of having addressed it.
1784+
**R3 is why this is a crate and not a transitive-closure helper:** two
1785+
independently authored sources can each be internally consistent and still
1786+
disagree about a relation's DIRECTION; merging then derives `A ⊑ B` and
1787+
`B ⊑ A` for classes neither calls equivalent, and that cycle — found at any
1788+
distance, including through chains no pairwise check would look at — is the
1789+
disagreement made mechanical. **Deliberate boundary, named in the crate
1790+
doc:** no existential restrictions, no role composition, no bottom
1791+
propagation, no conjunction/disjunction/complement. Each becomes necessary
1792+
the moment a typed cross-angle `ogar-ro` predicate enters the closure, and
1793+
at that point the correct move is to wrap a full reasoner (`whelk-rs`) —
1794+
not to grow the file. The hazard that boundary guards: without role
1795+
composition, walking subsumption and part-of together derives FALSE
1796+
ancestors (`A part_of B`, `B ⊑ C` does **not** give `A ⊑ C`), which is why
1797+
`Closure::from_asserted` takes a `Subsumption` type rather than raw pairs.
1798+
Zero-dependency, `forbid(unsafe_code)`, 8 tests each carrying the input
1799+
that would falsify it — depends: D-CLASSID-CANON-HIGH-FLIP (the address
1800+
form it consumes).
1801+
1802+
- **D-OGAR-DOC-SPINE (`docs/OGAR-DOC-INGESTION-SPINE.md` — borrowed operating
1803+
time for the stage W4 does not have; 2026-08-07; [S] — transferred claim,
1804+
council input):** `OGAR-DOC-W4-BUILD-SPEC` is strong on persist / read /
1805+
reconstruct but **begins one stage too late** — it starts from a `DocIr` that
1806+
already exists, and nothing in OGAR covers the stage where a file arrives and
1807+
something must decide what it is, whether it needs recognition at all, and in
1808+
what order the work happens. That stage is where a document system accrues its
1809+
scar tissue and this stack has none; `paperless-ngx` has run it in production
1810+
for ~a decade. Thirteen invariants extracted, cited to source, over four axes:
1811+
**ingestion ordering** (S-1..S-5), **escalation** (S-6), **rules** (S-7..S-10),
1812+
**business identity** (S-11..S-13). The two that would be expensive to
1813+
retrofit: **S-2** — dedup is consulted BEFORE recognition spend, not only
1814+
inside `persist_document`, and it matches against the derived-artifact hash as
1815+
well as the original (W4's `content_sha256` idempotency prevents a duplicate
1816+
*subtree*, not a duplicate *spend*); and **S-5** — the write order between the
1817+
KV blob and the subtree decides whether a failure leaves collectable garbage
1818+
or an undetectable dangling reference, a question W4 does not currently answer.
1819+
**S-4 is the one entry better than [S]** — "one decision, two consumers, ONE
1820+
function" is independently corroborated in-stack (`tesseract-rs`'s
1821+
`region_is_table` had to become one shared primitive for exactly the reason
1822+
`is_born_digital_text` did, their issue #13387), so the rule grades **[H]**.
1823+
**Explicitly NOT transferred** (§NT): the sklearn estimator (duplicates
1824+
`deepnsm` / `lance-graph-arm-discovery`, violates ADR-022/023), the filing
1825+
data model (`correspondent`/`tags` — a competing identity model against
1826+
`classid → ClassView → facet rails`), storage-path templating (consumer's per
1827+
W4-8), and **the dual store** — their full-text index is a second authority
1828+
whose cost is measured in their own retry policy
1829+
(`autoretry_for=(SearchIndexLockError,), max_retries=5`). **Operator-directed
1830+
deferral:** a `tantivy` index is NOT adopted as the search layer — our typed
1831+
fields make the dominant queries *structural* (the SoA columns already answer
1832+
them) where paperless is forced into full-text search by a flat `content`
1833+
blob; if free-prose retrieval is later wanted it indexes the out-of-line
1834+
value-slab keyed by `document_guid`, a lens and never a parallel authority.
1835+
Falsifiers named for S-2 (ingest the same page twice; second pass must be ~0)
1836+
and S-8 (express real SKR03 rules with `CONTAINS`/`LIST_CONTAINS`/`EQ` only —
1837+
if `REGEX`/`FUZZY` prove load-bearing the `FnIndex` additions are not
1838+
optional). Depends: D-OGAR-DOC-LAYER, D-DOC-IR-SECOND-RETINA.

0 commit comments

Comments
 (0)