Repository navigation
Commit 5ffc511
committed
docs: the ogar-doc INGESTION SPINE - transferred production experience
OGAR-DOC-W4-BUILD-SPEC is strong on the half it covers (persist / read /
reconstruct, content_sha256 idempotency, the SoA subtree, the four-leg
renderer) but it BEGINS ONE STAGE TOO LATE: it starts from a DocIr that
already exists. Nothing in OGAR covers the stage where a file arrives and
something must decide what it is, whether it needs recognition at all, and in
what order the work happens. That stage is where a document system accumulates
its scar tissue, and this stack has none on that axis.
paperless-ngx has run exactly that stage in production for roughly a decade.
This document extracts its spine - the ordering constraints, the
centralisation rules, and the failure modes it fixed - as 13 invariants over
four axes: ingestion ordering (S-1..S-5), escalation (S-6), rules (S-7..S-10),
business identity (S-11..S-13). Every claim is cited to source and every one
is graded [S]: transferred claim, NOT measured in this stack. It is borrowed
operating time, not a finding, and the document says so in its own header.
The two invariants that would be expensive to retrofit:
S-2 - dedup is consulted BEFORE recognition spend, not only inside
persist_document, and it matches the incoming hash against the derived-artifact
checksum as well as the original (verified: consumer.py's
Q(checksum=...) | Q(archive_checksum=...)). W4's content_sha256 idempotency
prevents a duplicate SUBTREE; it does not prevent a duplicate SPEND, and for
OCR that asymmetry is seconds-to-minutes of compute per re-ingest.
S-5 - the write order between the KV blob and the subtree decides whether a
failure leaves collectable garbage or an undetectable dangling reference.
paperless creates the row inside a transaction and writes bytes after, under a
FileLock, so a failure can never leave an addressable document with missing
bytes. W4 does not currently answer this question; the document frames it as a
question rather than asserting a defect.
S-4 is the one entry better than [S]. "One decision, two consumers, ONE
function" is independently corroborated in-stack: tesseract-rs's
region_is_table had to become one shared primitive for exactly the reason
is_born_digital_text did (their GH #13387, cited in its own docstring - two
independent computations of one decision drifted apart silently). Two
projects, same shape, same fix, so the rule grades [H].
Explicitly NOT transferred, with reasons (section NT): the sklearn estimator
stack (duplicates deepnsm / lance-graph-arm-discovery, violates the
no-serialization-in-hot-path rule ADR-022/023); the filing data model
(correspondent/tags/document_type is a competing identity model against
classid -> ClassView -> facet rails); storage-path templating (consumer's per
W4-8); and the dual store, whose cost is measured in their own retry policy
(autoretry_for=(SearchIndexLockError,), max_retries=5, retry_backoff=60 -
lock contention that exists BECAUSE the index is a second authority).
Operator-directed deferral recorded: a tantivy inverted index is NOT adopted
as the search layer. Typed fields with provenance make the dominant queries
structural - the SoA columns already answer "invoices from supplier X in
period Y" - where paperless is forced into full-text search because its
extraction stops at a flat content blob. If free-prose retrieval is wanted
later for a specific use case, it indexes the out-of-line value-slab keyed by
document_guid: a lens, never a parallel source of truth.
Falsifiers named where cheap: S-2 (ingest the same page twice; the second pass
must be ~0 elapsed or the gate is missing) and S-8 (express real SKR03 booking
rules with CONTAINS/LIST_CONTAINS/EQ only - if REGEX/FUZZY prove load-bearing
rather than convenience, the ogar-loco FnIndex additions are not optional and
the council should know that before ratifying the mapping).
Sections C and D are not W4's business - they belong to the rule layer
(ogar-loco, where the rule-bearing object is the Konto exactly as it is the
Correspondent there) and to the consumer. They are recorded so the council can
see the full arc the document layer sits inside, and so typed_field 0x080A is
minted knowing what it will eventually have to carry.
Documentation only. No crate, no mint, no code. DISCOVERY-MAP entry
D-OGAR-DOC-SPINE appended per CLAUDE.md.1 parent 8d31552 commit 5ffc511
2 files changed
Lines changed: 409 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1768 | 1768 | | |
1769 | 1769 | | |
1770 | 1770 | | |
| 1771 | + | |
| 1772 | + | |
| 1773 | + | |
| 1774 | + | |
| 1775 | + | |
| 1776 | + | |
| 1777 | + | |
| 1778 | + | |
| 1779 | + | |
| 1780 | + | |
| 1781 | + | |
| 1782 | + | |
| 1783 | + | |
| 1784 | + | |
| 1785 | + | |
| 1786 | + | |
| 1787 | + | |
| 1788 | + | |
| 1789 | + | |
| 1790 | + | |
| 1791 | + | |
| 1792 | + | |
| 1793 | + | |
| 1794 | + | |
| 1795 | + | |
| 1796 | + | |
| 1797 | + | |
| 1798 | + | |
| 1799 | + | |
| 1800 | + | |
| 1801 | + | |
| 1802 | + | |
| 1803 | + | |
| 1804 | + | |
| 1805 | + | |
| 1806 | + | |
| 1807 | + | |
| 1808 | + | |
| 1809 | + | |
| 1810 | + | |
| 1811 | + | |
| 1812 | + | |
| 1813 | + | |
| 1814 | + | |
| 1815 | + | |
| 1816 | + | |
| 1817 | + | |
| 1818 | + | |
| 1819 | + | |
| 1820 | + | |
| 1821 | + | |
| 1822 | + | |
| 1823 | + | |
| 1824 | + | |
| 1825 | + | |
| 1826 | + | |
| 1827 | + | |
| 1828 | + | |
| 1829 | + | |
| 1830 | + | |
| 1831 | + | |
| 1832 | + | |
| 1833 | + | |
| 1834 | + | |
| 1835 | + | |
| 1836 | + | |
| 1837 | + | |
| 1838 | + | |
0 commit comments