From d78d49e1e121d90b6d716c5c0d0d2f2bc514f9eb Mon Sep 17 00:00:00 2001 From: Frey <370805886@qq.com> Date: Mon, 19 Oct 2026 15:57:18 +0800 Subject: [PATCH 1/4] docs(rfc): propose text material import in English and Chinese --- docs/en/rfcs/0000-text-material-import.md | 1453 +++++++++++++++++++++ docs/zh/rfcs/0000-text-material-import.md | 933 +++++++++++++ 2 files changed, 2386 insertions(+) create mode 100644 docs/en/rfcs/0000-text-material-import.md create mode 100644 docs/zh/rfcs/0000-text-material-import.md diff --git a/docs/en/rfcs/0000-text-material-import.md b/docs/en/rfcs/0000-text-material-import.md new file mode 100644 index 0000000000..abf405bec6 --- /dev/null +++ b/docs/en/rfcs/0000-text-material-import.md @@ -0,0 +1,1453 @@ +--- +title: Text Material Import +--- + +- Proposal Name: `text_material_import` +- Start Date: 2026-09-18 +- Status: Proposed +- RFC PR: [oceanbase/powercontext#0000](https://github.com/oceanbase/powercontext/pull/0000) +- Design Baseline: [PowerContext master / cab054db (2026-10-19)](https://github.com/oceanbase/powercontext/tree/cab054dbf6d86e2311b079388e5158178257def2) +- Related RFCs: [Source Definition and Observation](1400_source_definition_and_observation_model.md), [Source and Artifact REST API](1437_source_artifact_rest_api.md), [Source access and generation eligibility](1458_artifact_generation_source_access.md), [Scope and Source discovery](1502_scope_source_discovery.md) + +# Summary + +Provide text material import for text, office documents, saved webpages, spreadsheets, email, ebooks, source code, +images, and archives. Users must select an existing Scope with `scope.read` and `scope.contribute` before pasting text, +uploading files, or using an independent OpenDAL worker to collect files from directories or object storage. Format +detection, decoding, parsing, optional OCR, and bounded segmentation produce immutable `content` Sources and an +`ImportedContent` Artifact (`family=importedcontent`), with an exact Revision receipt. + +The product contract combines broad format coverage with explicit capability levels. Lightweight text can be saved +directly; complex files use isolated parser workers; scans and images require explicit OCR enablement. The standard +document package targets PDF, modern and legacy Office, OpenDocument, HTML, email, and ebooks. Deployments expose actual +formats, parsers, and limits. Detecting a format or extracting only metadata is not successful body import. + +One logical document has one ImportedContent Artifact. `source_kind` is always `content`; each newly accepted snapshot +becomes an immutable Revision whose exact ContentSource references hold the body. Success means the imported Artifact +and its evidence were saved, not that downstream Memory generation, binary archival, or full-text indexing completed. +Head is the latest version accepted by PowerContext, not necessarily the current external file. Continuous collection +does not propagate deletion. This RFC proposes contracts; it does not claim released functionality. + +Saved materials reuse Artifact tables, standard reads, Revisions, and lineage. ImportedContent-specific operations own +writes. Progress and private previews belong to a reclaimable internal operation ledger, not Artifact Revisions or a +separate FileImportJob business resource. + +# Motivation + +Project materials extend beyond TXT and Markdown to Word, PDF, Excel, PowerPoint, webpage exports, email, and scans. An +extension allowlist alone cannot decode binary formats, recognize scans, preserve table relationships, segment long +documents, or explain partial extraction and recovery. + +Existing ContentSource, Scope authorization, Source journals, text projections, Source collection/exact reads, and the +OpenDAL evaluation Connector's durable receipts and checkpoint CAS provide persistence foundations, not general file +parsing. + +The existing OpenDAL evaluation uses `text-file-snapshot` and registers a Definition manifest. Its UTF-8 collection +behavior, permissions, and public reads do not define this proposal's support contract. This design reuses the built-in +`content` Source and adds an import application service and optional parser deployments without changing existing Source +Create/Capture APIs. + +Users need to know which formats their deployment can read, what was extracted or omitted, the destination Scope, and +how to recover from retries and failures. + +# Guide-level explanation + +## Supported formats and capability levels + +The following matrix is the target. A is the base capability, B the standard document package, and C optional OCR or +extension adapters. Each published entry requires real fixtures. Missing dependencies mean unavailable; unfinished +adapters mean planned, not supported. + +| Category | File types and common extensions | Level and extracted content | +| --- | --- | --- | +| Plain text and logs | `.txt`, `.text`, `.log`; explicitly selected extensionless text | A; body and line numbers | +| Markdown and lightweight markup | `.md`, `.markdown`, `.mdown`, `.rst`, `.adoc`, `.asciidoc`, `.org`, `.tex` | A; markup source, paragraphs, and lines; no typesetting execution | +| Data and configuration | `.csv`, `.tsv`, `.json`, `.jsonl`, `.ndjson`, `.yaml`, `.yml`, `.toml`, `.ini`, `.cfg`, `.conf`, `.xml` | A; original text and verifiable record/row/column locations; no object construction or external entities | +| Code and build files | `.py`, `.js`, `.ts`, `.jsx`, `.tsx`, `.java`, `.c`, `.h`, `.cpp`, `.hpp`, `.go`, `.rs`, `.sh`, `.bash`, `.zsh`, `.sql`, `.css`, `.scss`, `.vue`, `.svelte`, `.rb`, `.php`, `.swift`, `.kt`, `.kts`, `.scala`, `.r`, `.R`, `.lua`, `.pl`, `.ps1`, `.bat`, `.cmd`, `.proto`, `.graphql`, `.gql`, `.diff`, `.patch`; Dockerfile, Makefile, CMakeLists.txt | A; preserve text without running, compiling, or installing dependencies | +| Subtitles and records | `.srt`, `.vtt`, `.ass`, `.ssa`, `.ics`, `.vcf` | A; timestamps or original fields; no calendar/contact API calls | +| Saved webpages | `.html`, `.htm`, `.xhtml`, `.mhtml`, `.mht` | B; static DOM body, headings, lists, tables, and link text; no scripts, external fetches, or browser-computed visibility guarantee | +| PDF | `.pdf` | B; text layer, page numbers, extractable tables; scanned/mixed pages require C-level OCR | +| Word and templates | `.docx`, `.doc`, `.dot`, `.dotx`, `.docm`, `.dotm` | B; paragraphs, headings, tables, footnotes; macros never execute | +| Excel and templates | `.xlsx`, `.xls`, `.xlsb`, `.xlt`, `.xltx`, `.xlsm`, `.xltm` | B target; sheets, cell values, formulas, existing cached results, and ranges; full and values-only XLSB profiles have separate admission rules in Section 1.4 | +| PowerPoint and templates | `.pptx`, `.ppt`, `.pps`, `.pot`, `.ppsx`, `.potx`, `.pptm`, `.ppsm`, `.potm` | B; slide text, tables, speaker notes, and slide order | +| Open office formats and rich text | `.odt`, `.ott`, `.ods`, `.ots`, `.odp`, `.otp`, `.fodt`, `.fods`, `.fodp`, `.rtf` | B; document, sheet, or presentation content with corresponding location rules | +| Ebooks | `.epub`, `.fb2` | B; navigation, chapters, and body; no DRM decryption | +| Email and mailbox exports | `.eml`, `.msg`, `.mbox` | B; subject, sender information, body; one document per message, explicitly selected attachments | +| Notebooks | `.ipynb` | B; Markdown/code cells and saved text outputs; no execution or recalculation | +| XML documents and feeds | `.rss`, `.atom`, `.opml`, `.xlf`, `.xliff`, `.tmx`, `.svg` | A source preservation; B semantic text extraction; SVG text nodes only, no execution or graphical understanding | +| Images and scans | `.png`, `.jpg`, `.jpeg`, `.tif`, `.tiff`, `.bmp`, `.webp`; static `.gif` | C; OCR text, image/page order, and available locations; animated GIFs cannot silently use only the first frame | +| Batch archives | `.zip`, `.tar`, `.tar.gz`, `.tgz`, `.gz`, `.bz2`, `.xz` | B; safe expansion and per-member identification/import; an archive is not a body format | +| Extended archives | `.7z`, `.rar`, `.tar.bz2`, `.tbz2`, `.tar.xz`, `.txz` | C; installed and validated unpacking adapters required | +| Proprietary/long-tail formats | `.pages`, `.numbers`, `.key`, `.wps`, `.et`, `.dps`, `.wpd`, `.hwp`, `.chm`, `.one`, `.pst`, `.mobi`, `.azw`, `.azw3`, `.heic`, `.heif` | C extension targets; admission per format generation, actual body extraction, decoder, and license; a generic parser installation is insufficient | + +A and B are the formal delivery scope; C uses the same extension mechanism with deployment-specific enablement. A +lightweight installation may enable only A. Standard installations must bundle and pin B dependencies. Reading +WPS-produced DOCX/XLSX/PPTX does not imply native WPS/ET/DPS support; detecting newer iWork containers does not prove +body extraction. + +The explicit “read as plain text” fallback is not limited to listed suffixes, but must pass encoding, control-character, +and size checks. Directory collection scans allowed types, not every decodable file. Defaults exclude keys, credentials, +`.env`, version-control directories, dependencies, and build outputs; users can explicitly select other rules. + +Audio/video transcription, arbitrary database dumps, executable decompilation, graphic/chart reasoning, and breaking +encryption/DRM are out of scope. Yuque, webpage, and Git URL acquisition belong to independent Connectors. Importing +local HTML or exported Markdown does not implement URL fetching. + +## User flow + +Users select an existing contributable Scope first. Global entry points require explicit selection; an entry inside a +Scope may preselect it but must display and authorize it. Without a valid destination, creating an ImportSession, +pasting, and uploading are disabled. When none is available, direct users to the existing Scope management flow. The +import UI neither offers Scope creation nor creates one automatically. + +After selection, show actual formats, limits, and caller permissions. Creating an ImportSession fixes a nonempty +`scope_id` and its creator. All inputs in that session belong to that Scope. Changing destinations requires a new +session; parsing or unconfirmed inputs cannot migrate. + +Validated plain text can be confirmed immediately. Files are uploaded and parsed first, then previewed with +page/sheet/slide counts, segment counts, actual OCR use, and omissions. Confirmation rechecks Scope and write +permission, and saves one document's Sources, ImportedContent Revision, Head, lineage, and receipt in one transaction. +Failure, cancellation, or preview alone publishes nothing. + +```mermaid +flowchart TD + A["Select an existing Scope"] --> P["Check scope.read + scope.contribute; create ImportSession with fixed scope_id"] + P --> B{"Input"} + B -->|Paste text| C["Validate and normalize"] + B -->|Upload file| D["Family prepare: durable staging and isolated parsing"] + D --> E["Preview body, parts, OCR, and omissions"] + E -->|No usable body| X["Fail; publish nothing"] + E -->|Confirm complete or explicitly accept partial| F["Confirm frozen bundle_digest"] + C --> G["Reauthorize; atomically save one document in the session Scope"] + F --> G + G --> I["ContentSource parts + ImportedContent Revision + lineage + receipt"] + I --> J["Subsequent files retain the Scope; configured downstream processing"] +``` + +Ordinary imports require `scope.read` and `scope.contribute`, not `server.admin`. A session is not an authorization +grant. Revocation, a missing destination, or an unwritable target prevents confirmation; it cannot silently choose +another target. Administrative operations follow Section 10. + +A first manual upload or paste defaults to creating a new document and allocating `artifact_id`. Updating requires the +existing document's update action, explicit `artifact_id`, and expected Head Revision. Matching filenames, similar +bodies, or identical digests never merge identities automatically; duplicate suggestions require user choice. Renaming a +file does not alter an explicitly selected identity. + +Show each file as awaiting upload, parsing, awaiting confirmation, saved, already present, needs attention, version +conflict, or outcome uncertain. Keep filename, selected Scope, and create/update intent visible. Concurrency still obeys +Scope-level quotas, idempotency, and Head checks. + +For example, Word may save successfully while a scanned PDF without OCR needs attention and Excel contains formulas +without cached values. Identify affected PDF pages and offer a new OCR attempt or explicit partial acceptance. Display +Excel formulas and “no saved calculation result,” without invented values. Do not report all three as fully extracted. + +Use “view extracted text” for parsed formats and “view original text” for plain text. Show supported handling and +omissions for images, layout, charts, and embedded objects. Neither parsing completion nor preview visibility means +Source or Artifact persistence has occurred. + +## Format-specific behavior + +- **Encoding:** UTF-8, BOM-marked UTF-16/UTF-32, and explicitly specified GB18030, GBK, Big5, Windows-1252, ISO-8859-1. Detection suggests an encoding; except for BOM and strictly valid UTF-8, guesses require confirmation. Never hide decode errors with replacement characters. +- **Sheets:** preserve sheet names and cell coordinates across all sheets. CSV/TSV retain source text and may add validated record/column locations. Never recalculate or refresh links. Hidden sheets/rows/columns are included and marked by default; explicit pre-preview exclusion enters snapshot identity. +- **Slides:** preserve slide order and separate notes. Include and mark hidden slides by default. Image text requires OCR and must not be counted as already read. +- **Word/open documents:** extract headings, body, tables, and footnotes under declared rules. Use the accepted-changes view. Comments are excluded by default with counts; optional inclusion records authors. A parser unable to produce these views reliably reports insufficient capability. +- **HTML/notebooks:** do not execute JavaScript/cells or fetch CSS, images, or links. Include saved notebook text outputs by default; hidden metadata, rich media, and widgets are not body text. +- **PDF/OCR:** choose native text or OCR per page/region without double counting. Distinguish blank, image-only, and failed pages. OCR is machine recognition, not an exact transcription guarantee. Unavailable layout/page counts are unknown, not proof of completeness. +- **Email/archives:** retain quoted history; select one plain-text or HTML alternative and record the rule. List attachments by default but import only selections. Expand the first archive layer by default; nesting requires explicit enablement and depth limits. Keep failed/unselected members in the manifest. + +## Saving and downstream processing + +One logical document has one ImportedContent Artifact; accepted snapshots are Revisions. Long bodies become multiple +immutable ContentSources. All parts, the Revision, Head, lineage, pending processing, and receipt commit together in the +session's existing Scope, without creating a Scope or initializing permissions. Different files, archive members, and +messages may succeed independently. + +Success requires neither an LLM nor Candidate approval. Existing processor configuration and domain contracts determine +whether Memory and other processors consume these Sources, how much they consume, and their outputs. ImportedContent +registers no default generation processor or recursive generation behavior. + +Historical Revisions and Sources remain immutable when Head changes. Details may show history and downstream references +based on actual exact SourceRefs or ArtifactRefs; import cannot prepopulate future related Artifacts. A consumer window +need not consume every part, so do not claim full-document understanding. + +Original binaries are parser staging, not a permanent archive. Persist extracted text, digests, parser versions, +locations, and omissions; binary downloads and archival are outside this release. + +## Continuous collection and Dashboard + +An OpenDAL worker near the storage holds read credentials and submits bytes. The Server does not read arbitrary +user-supplied filesystem paths or URLs. Text and complex files use the same capability matrix; an external scheduler +owns periodic execution. + +Dashboard retains Tabler, Jinja2, HTMX, both languages, and light/dark themes. +`POWERCONTEXT_SERVER_DASHBOARD_TEXT_IMPORT_ENABLED` explicitly enables its write entry point and defaults to false, +preserving static Bearer and enforced-access prerequisites. File parsing also requires its service; disabling it does +not block saved-material reads. + +The page flow is existing Scope selection, material list, text/file selection, parsing preview and issues, confirmation, +then exact reading. Scope navigation cannot retarget running operations or submit pending inputs. Accepted parsing +operations resume after browser closure; incomplete uploads require reselecting the local file. + +# Reference-level explanation + +## 1. Architecture + +The user-flow diagram separates input, parsing, and persistence. ImportedContent prepare creates a frozen +ImportedContentBundle through an internal ledger and isolated workers. Only user confirmation or a worker's explicit +acceptance policy invokes ContentImportApplication's per-document atomic commit. Existing Artifact processing retains +its configuration and progress semantics. + +Acquisition, parsing, text persistence, and Artifact generation are four boundaries. Core does not import arbitrary +user-selected parsers, execute macros, or run code. Production provides independent parser workers; lightweight text +does not require document-parser dependencies. + +### 1.1 Parser responsibilities and selected components + +Prefer native structure, route PDF by layout, and enable OCR explicitly. PowerContext owns detection, structural +auditing, bundle normalization, evidence segmentation, operations, and persistence. Libraries decode formats and perform +layout/OCR; this proposal does not build a general OCR model. Profile names below are proposed product contracts, not +existing upstream APIs or completed adapters. + +| Component | Selected responsibility | Excluded responsibility | +| --- | --- | --- | +| Native text/container adapters | Strict decoding; CSV/JSON/XML locations; OOXML/ODF audits; email/archive discovery | No execution or LLM rewriting | +| LiteParse | Simple PDF text, geometry, and complexity signals; built-in OCR off on the standard path | No universal Office-to-PDF conversion; cleaned Markdown cannot replace complete evidence | +| Docling | Complex PDF layout and modern Word/open-document structure; selected local OCR engine when enabled | No material identity, commits, indexing, or direct reuse of retrieval chunks as Sources | +| openpyxl / Apache POI | Native cells, formulas, stored values; POI also adapts legacy Office | No formula calculation, refreshed connections, or workbook printing to PDF | +| python-pptx + OOXML audit | Modern slide text, tables, notes, hidden state | No macros, animation interpretation, or chart reasoning | +| Apache Tika | Controlled legacy, RTF, MSG, and admitted long-tail parsing | No unrestricted autodetection, recursion, cloud OCR fallback, or metadata-only success | +| Local RapidOCR | Scanned PDFs/images through Docling; initial target languages Simplified Chinese and English | No automatic external transfer; Traditional Chinese, other languages, and handwriting require separate validation | +| MinerU / DeepDoc / LlamaParse | Independently enabled C-level alternatives; LlamaParse is external or enterprise self-hosted | No implicit fallback from standard profiles | +| LlamaIndex | Optional future Reader/retrieval-transform reference and indexing use | Does not own transactions, versions, or Sources; no default silent Reader error skipping | + +Selections draw on the [RAGFlow study](https://yuque.antfin.com/obopensrc/knowledge_sharing/lmziuic5tzhgny8c), +[Docling/MinerU study](https://yuque.antfin.com/obopensrc/knowledge_sharing/rcga2z7gr8r3vwbo), and [LiteParse/LlamaIndex +study](https://yuque.antfin.com/obopensrc/knowledge_sharing/yl6nfo4yugep4n6a). These are implementation choices, not a +quality ranking. Initial adapter baselines are LiteParse Python 2.14.7 and Docling 2.129.0; MinerU 4.0.5 is an optional +validation baseline. Release images must pin transitive dependencies, models, and conversion tools, never `latest`. Pin +other components and image digests after fixture and security validation. + +### 1.2 Level A: native text, markup, and locations + +Each group covers all corresponding extensions in the matrix. A preserves decoded original text; structural parsing adds +verifiable locations without rewriting the body by default. + +| Input | Default profile and implementation | Extraction, locations, and failures | +| --- | --- | --- | +| TXT/TEXT/LOG, explicit extensionless text | `text-native-v1`: strict Python codecs | Preserve newlines, indentation, characters, lines; uncertain encoding requires confirmation | +| Markdown/RST/AsciiDoc/Org/TeX | `markup-source-v1`: same decoder; optional markdown-it-py block identification | Preserve markup source; no TeX rendering, includes, plugins, or remote resources | +| All code/build/diff/patch files | `code-source-v1`: original text and line index | No compiler/formatter; failed language recognition does not discard text | +| CSV/TSV | `delimited-text-v1`: source plus Python csv reader | Freeze delimiter, quotes, encoding; logical records, fields, source ranges; multiline fields allowed; ambiguous delimiter requires confirmation | +| JSON/JSONL/NDJSON | `json-source-v1`: bounded syntax scanning and record parsing | Preserve source and duplicate-key order; record/field occurrence locations; never replace source with dict serialization | +| YAML/YML/TOML/INI/CFG/CONF | `config-source-v1`: source; optional PyYAML SafeLoader, tomllib, interpolation-disabled ConfigParser validation | Bound YAML aliases/depth, no arbitrary construction; ambiguous key locations fall back to source lines | +| XML | `xml-source-v1`: DTD/entities/network disabled | Source, node paths, namespaces, available lines; no XInclude/XSLT | +| SRT/VTT/ASS/SSA | `subtitle-source-v1`: text and bounded timestamp scan | Original fields and time ranges; no sorting or translation | +| ICS/VCF | `record-source-v1`: source, folded lines, record boundaries | Distinguish physical lines/logical records; no calendar/contact calls | +| RSS/Atom/OPML/XLF/XLIFF/TMX/SVG source mode | `xml-source-v1` | Source preservation; semantic extraction uses a separate B profile, never silent switching | + +Failed auxiliary validation reports usable source text but unavailable structural locations. If the selected profile +requires complete structure, fail or require review rather than substituting source mode. A's source fidelity does not +depend on structural parsing, but encoding, control-character, and size checks remain mandatory. + +### 1.3 PDF: native text, layout, and OCR routing + +`pdf-standard-v1` is a pinned composite profile. LiteParse first performs bounded probing. Use its structured +`text_items` when the entire native text layer is usable and no complex-layout signal exists. Multiple columns, complex +tables, unusual order, or uncertainty route to Docling's standard layout/table path. Both disable OCR and generative +enhancement by default; fixture-validated routing thresholds are versioned profile configuration. + +| PDF condition | Parsing | Completeness rule | +| --- | --- | --- | +| Simple native text, normal reading order | LiteParse JSON/structured output → ordered PC blocks | Physical page order, display labels, dimensions, available bbox; not Markdown that drops headers/footers by default | +| Multiple columns, complex tables, contents/footnote layout | Docling pipeline, DoclingDocument and provenance | Structure/reading order; table rows, columns, merges, original page locations; not rendered Markdown alone | +| Scanned/mixed/garbled text, `ocr_mode=off` | Usable native text only; identify OCR-required pages/regions | Body exists: needs_review; metadata-only/empty: ocr_required or no_extractable_text failure | +| Scanned/mixed, `ocr_mode=auto` | Docling + pinned local RapidOCR where needed | Region deduplication; actual engine, language, OCR execution and omissions; needs_ocr is not proof of execution | +| `ocr_mode=force` | Same engine, forced within declared range | Use configured OCR output without duplicate native text; preview policy and possible quality changes | +| Unreliable formulas, handwriting, complex charts | Keep usable text/locations and report objects | Never invent text; user may start another attempt with a C-level MinerU/DeepDoc/LlamaParse profile | + +`needs_ocr` and layout complexity are separate signals. No OCR requirement does not imply simple layout. Count source +pages, requested ranges, returned pages, blank pages, and failures independently. Engine `max_pages` truncation is not +completeness; an unrequested partial range caused by a hard limit rejects the document. + +Choose one body backend per PDF in this release; it may use native text or OCR internally by page. Do not splice +different engines page by page because tables, reading order, and coordinates may disagree. Probe results may be cached, +but are not an interchangeable cross-engine representation. If cross-page table merging is unreliable, retain separate +tables/page ranges and report the unresolved relationship. + +Include extractable headers/footers by default and mark their block types. Profiles declare watermark, annotation, form, +and image-text inclusion. An unenabled category is not a known zero; detected but unprocessed content enters omissions. +Locations describe provenance, not a promise of permanent binary PDF preview. + +### 1.4 Level B: Office and open office formats + +Native structure takes precedence over print layout. Modern Office is not converted to PDF by default. Identify +templates and macro variants by MIME, container content types, and relationships; read the original container rather +than renaming suffixes to bypass library limitations. + +| Input | Default profile/backend | Preservation and admission | +| --- | --- | --- | +| DOCX/DOTX/DOCM/DOTM | `word-ooxml-v1`: Docling DOCX + PC OOXML audit/supplement | Headings, body, tables, footnotes/endnotes, revision view, selected comments; no macros; unsupported variants need a controlled OOXML adapter or remain planned | +| DOC/DOT | `word-legacy-v1`: controlled Tika OfficeParser / POI HWPF plus audit | Flat text does not prove footnote/revision/comment completeness; unreliable selected views fail capability_insufficient, without silent PDF conversion | +| XLSX/XLTX/XLSM/XLTM | `sheet-ooxml-v1`: openpyxl + raw OOXML cells/relationships | Sheets, A1 coordinates, types, merged ranges, hidden state, formulas/cached values; count macros as excluded objects | +| XLS/XLT | `sheet-biff-v1`: POI HSSF | Available formula-token expression and stored cache; no FormulaEvaluator; validate/reject individual BIFF generations | +| XLSB | `sheet-xlsb-v1`: POI XSSFB events plus explicit structure adapter | Full formula-and-value contract must pass before available; values-only is a separate profile, below | +| PPTX/PPSX/POTX/PPTM/PPSM/POTM | `slides-ooxml-v1`: python-pptx + raw OOXML audit | Slide order, text boxes/group text, tables, notes, hidden slides; template handling belongs to adapter, not file mutation | +| PPT/PPS/POT | `slides-legacy-v1`: POI HSLF in Java worker | Native slides/notes; do not infer page boundaries from Tika's concatenated text; unknown OLE/graphics are omissions | +| ODT/OTT/FODT | `odf-text-v1`: Docling + safe PC ODF XML | Validate package/flat forms, headings, tables, comments, revision views; no fallback to all XML text | +| ODS/OTS/FODS | `odf-sheet-v1`: safe native PC ODF XML | Tables/rows/columns, repeats, merges/covered cells, hidden state, formulas/stored values; budget repeats before expansion | +| ODP/OTP/FODP | `odf-slides-v1`: safe native PC ODF XML | draw:page order, text boxes, tables, notes, display state; explicitly downgrade location capability if object identity is unstable | +| RTF | `rtf-native-v1`: structured Tika RTFParser + PC normalization | Body/paragraphs, available tables/footnotes; list embedded objects without execution; report unsupported control groups | + +**Word supplemental reads are adapter work, not a claim that Docling/python-docx implements everything.** Audit OOXML +relationships for paragraphs, tables, footnotes/endnotes, comments, and revision nodes. The accepted view includes +insertions/moveTo and excludes deletions/moveFrom; unsupported property or complex revisions cannot be guessed. Reject +profiles unable to produce that view. Another view requires supported capabilities and a new attempt. Insert +supplemental text at original anchors without duplicating or reordering it at the end. + +**Never recalculate workbooks.** Align XLSX formula and `data_only` views by identical input digest, sheet, and +coordinates; read `` and `` directly when needed to distinguish empty cells, cached empty strings, and missing +caches. Formula, cached value, type, and formatted display are separate fields. Preserve external references as strings +without fetching. Report affected ranges for broken shared/array formulas or dynamic structures. + +**XLSB is a known admission risk.** POI's public XSSFB documentation limits formula-expression extraction; readable XLSB +does not imply formula fidelity. `sheet-xlsb-v1` remains planned until its full contract works. An explicitly selected +`sheet-xlsb-values-v1` may import cached values only when sheets/cells/hidden ranges are verifiable. It always requires +review, declares omitted formula expressions, and requires `accept_partial`. Unverifiable coverage or no body still +fails. This does not complete B-level XLSB delivery and cannot satisfy automated complete-only collection. See [POI XLSB +limitations](https://poi.apache.org/apidocs/5.0/org/apache/poi/xssf/extractor/XSSFBEventBasedExcelExtractor.html). + +**LibreOffice is outside the standard workbook path.** Conversion can change cached formulas, fonts, pagination, and +object semantics. Do not convert XLS/XLSB/ODS to XLSX/PDF to bypass the no-recalculation contract. Document conversion, +if needed, requires a separately published extension profile validated for no external links, macros, or computational +side effects; initial sheet guarantees cannot depend on it. + +### 1.5 Level B: webpages, ebooks, email, notebooks, and archives + +| Input | Default profile/implementation | Body/member rules | +| --- | --- | --- | +| HTML/HTM/XHTML | `html-dom-v1`: html5lib + PC DOM traversal; safe XML for XHTML | Headings, paragraphs, lists, tables, link text; exclude script/style/template and explicitly hidden nodes; no CSS/JS execution or browser visibility guarantee | +| MHTML/MHT | `mhtml-native-v1`: Python MIME → selected root HTML → html-dom | Identify root using root-part data/content-location; ambiguous roots fail; embedded resources listed, never fetched by URL | +| EPUB | `epub-spine-v1`: safe ZIP → container/OPF → spine → html-dom | Reading order, chapters, navigation, text; not ZIP filename order; no scripts, remote resources, or DRM | +| FB2 | `fb2-native-v1`: safe XML → body/section/notes | Body, chapters, footnote references; binary images are not text | +| EML | `mail-rfc822-v1`: Python email BytesParser | Decode headers/MIME; prefer usable text/plain in multipart/alternative, otherwise html-dom; list CID images/attachments, no double body | +| MBOX | `mailbox-mbox-v1`: bounded iteration + mail-rfc822 | Discover before selection/parsing; retain ordinal, available Message-ID, container chain; duplicate Message-ID is not unique identity | +| MSG | `mail-msg-v1`: controlled Tika/POI HSMF | Subject/sender/body; HTML via html-dom, RTF via rtf-native; recursive concatenation cannot replace complete attachment discovery | +| IPYNB | `notebook-native-v1`: bounded JSON + nbformat | Original Markdown/code/raw cell order; saved text/plain, stdout/stderr, textual errors by default; explicit sanitized HTML inclusion; rich media/widgets excluded | +| RSS/Atom/OPML | `feed-xml-v1`: format-specific safe XML mapping | Feed/channel/entry/item/outline titles, body, link text and node paths; no linked-body fetch | +| XLF/XLIFF/TMX | `translation-xml-v1`: schema-specific source/target/segment | Unit ID, language, source and translation marked separately; no duplicated combined body; validate XLIFF generations separately | +| SVG semantic mode | `svg-text-v1`: safe XML text/tspan/title/desc | No rasterization/script execution, path-text/graphic understanding, or image-resource fetch | +| ZIP/TAR/TAR.GZ/TGZ | `archive-native-v1`: bounded Python zipfile/tarfile streaming | Complete discovery before selection and dispatch; no extractall to arbitrary paths; reject links/devices/duplicate paths | +| GZ/BZ2/XZ | `compressed-stream-v1`: standard-library decoders | Reidentify decompressed stream: TAR enters archive discovery, otherwise one child document; not every stream is a multi-file archive | + +Python MIME/mailbox provide basic reading; PC still owns budgets, malformed structures, selection, and identity. A +returned object does not prove a complete manifest. See [email +parser](https://docs.python.org/3/library/email.parser.html) and +[mailbox](https://docs.python.org/3/library/mailbox.html). + +Tika invokes only registered format parsers with automatic OCR, network, and uncontrolled embedded recursion disabled. +Controlled callbacks expose attachments/resources for discovery rather than silently adding them to body text. If +detected attachments cannot be fully enumerated, discovery fails rather than publishing a selectable “complete” list. +Unselected embedded objects in ordinary documents follow declared exclusions. + +### 1.6 Level C: OCR, alternative engines, and long-tail formats + +| Input/need | Selected extension | Enablement/limits | +| --- | --- | --- | +| PNG/JPEG/BMP/WebP/static GIF | `image-ocr-v1`: Pillow decode/orientation → Docling + local RapidOCR | Explicit OCR; original-to-rotated coordinate mapping; no chart reasoning | +| Multipage TIFF/TIF | `image-multipage-ocr-v1`: bounded all-page enumeration → same OCR | Per-page counts, pixels, OCR budget; failed pages reported; not first frame only | +| Animated GIF/multiframe variants | Reject by default | Separate profile requires frame semantics and repeated-text validation | +| HEIC/HEIF | `heif-ocr-v1`: pillow-heif/libheif → image OCR | Distinguish primary, thumbnail, auxiliary, sequence images; fixed selection; metadata alone is insufficient | +| Alternative complex PDF/scans | `pdf-mineru-v1` / `pdf-deepdoc-v1` | Direct MinerU structure protocol or isolated DeepDoc wrapper; pin tier/models; no indirect whole-RAGFlow integration | +| Optional parsing service | `document-llamaparse-v1` | Explicit deployment enablement/user choice and visible data destination; pinned API/version/tier; download then freeze results; never automatic external fallback | +| 7Z/RAR | `archive-libarchive-v1`: validated libarchive-c decoders | Safe enumeration and bounded reads; per-method/generation admission; encrypted inputs rejected | +| TAR.BZ2/TBZ2/TAR.XZ/TXZ | `archive-native-extra-v1`: standard decompressor + tarfile | Same archive budgets; C switch remains required despite available dependencies | +| Pages | `iwork-pages-v1`: Docling format-iwork | Validate body/structure per generation; otherwise planned; metadata-only newer Tika support is insufficient | +| Numbers/Keynote | `iwork-legacy-v1`: only Tika-readable, structurally validated older generations | New generations planned; sheet/slide contracts mandatory; otherwise request XLSX/PPTX export | +| Native WPS/ET/DPS | `wps-native-v1`: reserved controlled adapter, initially planned | Export DOCX/XLSX/PPTX; renaming, recognition, or PDF printing is not native support | +| WPD | `wordperfect-tika-v1`: Tika WordPerfectParser | Validated generations and scope only | +| HWP | `hwp-tika-v1`: Tika HwpV5Parser | HWP v5 validation does not cover all HWP/HWPX | +| CHM | `chm-tika-v1`: controlled ChmParser discovery → html-dom | No help-program execution; require verifiable chapter order | +| ONE | `onenote-tika-v1`: Tika OneNoteParser | Validate pages/sections/embedded objects; not full notebook synchronization | +| PST | `pst-tika-v1`: controlled OutlookPSTParser discovery → mail adapter | Require complete folders/messages/attachments and stable identity rules; no mailbox account linkage | +| MOBI/AZW/AZW3 | `ebook-calibre-v1`: isolated ebook-convert → EPUB → epub-spine | DRM-free validated variants only; pinned conversion/options and loss reporting; disabled by default | + +References: [Tika formats](https://tika.apache.org/docs/4.0.x/formats.html), [Docling +formats](https://docling-project.github.io/docling/usage/supported_formats/), +[pillow-heif](https://pillow-heif.readthedocs.io/en/stable/pillow-plugin.html), and [calibre +conversion](https://manual.calibre-ebook.com/generated/en/ebook-convert.html). Profiles are PC implementation choices; +upstream support does not prove compliance with every required semantic. + +### 1.7 Routing, fallback, and freezing + +```mermaid +flowchart TD + A["Durable input and content identification"] --> B{"Format family"} + B -->|Text and code| C["Strict decoding + source locations"] + B -->|Office / ODF| D["Native structure + semantic audit"] + B -->|PDF| E["Text quality and layout probe"] + E -->|Simple native text| F["LiteParse; OCR off"] + E -->|Complex or uncertain layout| G["Docling layout / tables"] + E -->|OCR required and enabled| H["Docling + pinned local OCR"] + B -->|Images with OCR enabled| H + B -->|Email / archives| I["Complete discovery → selection → child routing"] + B -->|Webpages / ebooks / notebooks| J["Specialized structural reading"] + C --> K["PC normalization and coverage audit"] + D --> K + F --> K + G --> K + H --> K + I --> K + J --> K + K --> L{"Body and contract"} + L -->|Complete| M["ready; frozen preview"] + L -->|Body exists; explicit partial allowed| N["needs_review + omissions"] + L -->|Empty / hard limit / required semantics missing| O["failed"] +``` + +Freeze router/profile versions and candidate chain at input acceptance; record the selected branch after probing. +`pdf-standard-v1` only selects LiteParse or promotes to Docling under declared rules. No voting or default all-engine +execution. Fallback cannot enable OCR, bypass insufficient capability, encryption, budgets, or prohibited external +transfer. + +Allow at most one alternate parse for technical failure or structural validation failure, only when the profile +predeclares another local candidate with the same extraction contract. Share the original deadline and cumulative +resources. No fallback after a bundle is frozen. Engine, OCR, view, or acceptance-scope changes require a new attempt. +Missing required declared backends return parser_unavailable; incidental installed libraries do not alter routing. + +Record selected_profile, router_version, actual_backend, engine/model versions, conversion chain, routing reason, actual +OCR execution, output schema, and normalization version. Profile digest covers selection policy; bundle/snapshot digests +cover the actual semantic chain. Equal text from different engines cannot hide different locations, coverage, or +processing provenance. + +### 1.8 Deployment units and delivery scope + +The base installation contains lightweight A dependencies. Standard B workers pin LiteParse, Docling, openpyxl, +python-pptx, HTML/XML/notebook adapters, with Tika/POI in a separate Java worker. PC's ODF/OOXML supplements share safe +XML and container budgets, not duplicated unbounded XML entry points. OCR models, MinerU/DeepDoc, commercial services, +proprietary decoders, calibre, and libarchive are separately packaged/enabled. + +Standard workers may include layout models. OCR engines/language packs run only with C enabled. Prepare +dependencies/models at deployment, never download/install/update while parsing user files. Bound process pools, +concurrency, PDFium global locks, Java heaps, and OCR workers independently to prevent backend CPU oversubscription. + +B is a delivery target, not unconditional availability. Profiles missing required semantics remain planned. An explicit +partial profile may serve manual import but does not complete B delivery. Display body/formula/comment/location +capabilities rather than one misleading supported checkmark. + +## 2. Capability discovery and identification + +### Protocol compatibility and runtime availability + +Keep the existing separation between Server discovery surfaces. Package versions or static contracts do not establish +parser health. + +| Endpoint | Responsibility and proposed extension | +| --- | --- | +| `GET /v1/server-info` | Deployment identity and compatibility. Add `content.import: {major: 1, minor: 0}` to OpenAPI `x-powercontext-feature-contracts`; protocol support is not enablement | +| `GET /v1/capabilities` | Add optional `content_import` with `enabled`, `readiness` (ready/degraded/unavailable), and `capabilities_endpoint`. Absence cannot imply support. Runtime summary does not replace format health or Scope authorization | +| `GET /v1/import-capabilities?scope_id=...` | Actual formats, profiles, limits, and caller permissions for an existing Scope; drives UI availability | + +`content.import` and `content_import` are new contracts requiring OpenAPI, generated code, Server assembly, and client +compatibility tests. The existing global endpoints retain `server.observe`. Ordinary callers with only Scope permissions +can query the Scope-level endpoint directly; global observation is not an import prerequisite. That response also +includes import protocol version and enablement, avoiding write probes. + +Import-service configuration owns API enablement; the Dashboard switch controls only its UI write entry point. Ready +means all enabled, declared-available profiles passed health checks; partial failure is degraded while healthy profiles +remain usable; disabled/no usable profiles is unavailable. Planned profiles do not degrade enabled capabilities. +Disabling imports or losing parser availability does not block published Artifact/Source reads. Discovery is a current +snapshot: prepare, retry, and confirm recheck applicable permission, runtime state, and frozen-result validity. An +unrelated parser outage cannot cause confirmation to reparse or discard frozen text. + +### Format and profile identification + +The new `GET /v1/import-capabilities` endpoint requires an existing `scope_id` and `scope.read`; missing/malformed IDs return 422. +It reports `scope.contribute`, but starting import requires both permissions. Return fixed `source_kind=content`, +extensions, canonical MIME, capability level, available/unavailable/planned status, parser profiles, extraction scope, +OCR/conversion dependencies, and actual limits. Registration plus startup self-checks establish status; an API-mounted +parser does not prove a healthy backend. + +Combine magic, container structure, declared MIME, and extension. Suffix matching is case-insensitive; source-path +identity preserves case. A PDF disguised as TXT routes as PDF or fails type mismatch, never blind decoding. Recognize +DOCX/XLSX/PPTX/EPUB as specialized containers before generic ZIP discovery. + +Reject unknown binary inputs. Extensionless/unknown-suffix text uses fallback only with explicit `as_text`, recorded in +the response. Server and workers use the same pinned profile. + +Per format generation and profile, return profile_id, profile_digest, router_version, default flag, backend candidates, +installed versions, extraction_scope, required_features, optional_features, locator_kinds, known_limitations, +dependencies, data destinations, and limits. Declare formulas, caches, comments, revisions, notes, hidden content, +attachment discovery, and OCR independently. Unhealthy dependencies mean unavailable; incomplete adaptation/semantic +validation means planned; available requires every prerequisite for that exact scope, not complete support for every +same-suffix profile. + +A suffix may offer full and explicitly selected partial profiles; routing cannot silently downgrade. Unknown/conflicting +options return 422. Required semantics found unsupported after identification fail capability_insufficient. +`accept_partial` cannot bypass identification, safety, size, encryption, or user-required semantics. Only profiles +declaring permitted partial output may produce needs_review when usable body and explainable omissions exist. + +## 3. Data model + +Add the `ImportedContent` type and `importedcontent` Family (display name “Imported content”), written by +ContentImportApplication. One logical document is one Artifact; accepted snapshots are immutable Revisions. Do not +create an independent ImportRecord as a second identity/content authority. This Artifact represents accepted content, +not upload work, parsing intermediates, or generated Memory. + +```mermaid +flowchart TD + I["ImportSession: creator and fixed, nonempty existing scope_id"] --> J["Internal operation / item_id: transient parsing and confirmation"] + J --> B["ImportedContentBundle: frozen content"] + B -->|Atomic commit| R["ImportedContent Artifact Revision"] + S["Scope"] --> H["Artifact Head: latest accepted version"] + H --> R + R -->|Ordered lineage.sources| C["ContentSource 1..N: body and locations"] + D["Downstream Artifact Revision"] -->|Exact SourceRefs actually consumed| C + D -.->|ArtifactRef only when the document Revision was an actual input| R +``` + +| Object | Responsibility and persistence | +| --- | --- | +| ImportSession | Creator Principal, immutable nonempty existing scope_id, expiry; recovery/isolation/concurrency for that destination; no new authorization | +| Internal operation / item_id | prepare returns operation_id; staging, discovery, leases, quotas, parsing, confirmation. item_id is stable only within its operation, not permanent identity or an Artifact | +| ImportedContentBundle | Frozen body, semantic manifest, coverage, digest; confirmation uses it without recalculating preview | +| ImportedContent Artifact | Stable `(scope_id, family=importedcontent, artifact_id)`, allocated for manual creation or a first stable source | +| ImportedContent Revision | Exact `(scope_id, family, artifact_id, revision)` with structured metadata and ordered Source mapping | +| Artifact Head | Latest accepted Revision, changed with expected_head_revision optimistic concurrency; not a live external version | +| Source bindings/request mappings | Stable source key → Artifact, parsing cache, idempotency key → exact receipt; indexes/ledgers, not duplicate ImportRecord | + +### Naming and source semantics + +The domain/Python type is ImportedContent; persistence and HTTP wire name are `importedcontent`. `source_kind=content` +says its body uses ContentSource, not whether it was pasted, uploaded, or acquired from a webpage. + +| Field/concept | Meaning | +| --- | --- | +| `family=importedcontent` | Whole-material identity, versions, segment relationships | +| `source_kind=content` | Constant in ImportedContent metadata; every body part is a content Source | +| `input_media_type` | Original format, e.g. text/markdown, application/pdf, text/html | +| `origin` | namespace, key, optional display_name; provenance, not authorization | +| `parser` | Actual parser, configuration, version, OCR, conversion chain | + +Revisions, frozen bundles, and read responses require `source_kind: "content"`. texts/prepare may omit it; the Server +supplies it before fingerprints, frozen digests, and cache keys. Explicit other values return 422. Thus omitted/explicit +content are semantically identical given equal other fields. OpenAPI uses a single-value constraint and Python +`Literal["content"]`, not caller-selectable types. + +This does not replace `SourceRef.source_type`. Every parts/lineage Source must be content and match source_kind; no +other types or reserved roles in the body. No constant-value source_kind filter/index is needed; namespace filtering +remains useful. + +Input endpoints/authorized Connectors determine acquisition; format/profile determine extraction. HTML/PDF/TXT, pasted +text, and future webpage snapshots all become content evidence, but binary parsing and Connector-based URL acquisition +remain necessary. No arbitrary URL acquisition endpoint is included. + +Authorized Connectors generate/validate connection identity, source_authority_id, and stable bindings. Caller-provided +origin is not proof of authority. Future webpage Connectors supply page identity, acquisition time, and sanitized +display URL; URLs are neither arbitrary server-fetch instructions nor inherently stable identities. Do not persist +credentials, signed URLs, or sensitive query parameters. container_chain records archive/email ancestry; bodyless +containers are not ImportedContent. + +source_kind enters bundle_digest and indirectly snapshot_digest, but does not choose routing, artifact_id, bindings, or +cache partitions. Updates still check expected_head_revision and complete snapshot semantics for reuse/new Revision. A +document can change from file to pasted text while every version retains content plus its own origin/format/parser. + +### Revision content + +| Field | Constraint | +| --- | --- | +| `schema_version`, `title` | Trimmed title: 1–256 Unicode characters | +| `source_kind`, `origin` | content; required namespace/key, optional display_name (1–256 Unicode characters, no controls); provenance only | +| `input_media_type`, `input_bytes`, `input_digest` | Original type, byte length, SHA-256; pasted input is received UTF-8 | +| `text_media_type`, `text_bytes`, `text_digest` | text/plain or text/markdown; complete extracted length/digest | +| `parser` | selected_profile/digest, router_version, actual backend/version, route reason, conversion chain, output schema, normalization/segmentation version, encoding, OCR model/language/execution; semantic fields enter frozen digest | +| `bundle_digest`, `snapshot_digest` | Frozen extraction and acceptance semantics; not logical identity; exclude execution/persistence-assigned IDs | +| `content_manifest` | Ordered ordinal, bytes, part digest, full-text character interval, location; no SourceRef/journal position/staging blob ID | +| `parts` | SourceRef and journal position per ordinal, one-to-one with manifest and exactly matching lineage.sources order | +| `coverage`, `warnings`, `omissions`, `accepted_partial` | Completeness within declared scope and explicit acceptance; preserve known omissions | +| `container_chain` | Ancestor digests, member paths/ordinals, provenance; separate from cross-snapshot stable member identity | + +Sources alone retain body text; do not duplicate it in Artifact content. Source metadata includes ImportedContent +artifact_id/revision, part ordinal, source_kind, controlled origin, and location. Persistence-assigned IDs/SourceRefs do +not enter content digests; source_kind/origin/locations already entered the frozen bundle. Create ordinary captured +ContentSources with `working_note=False`, `internal=None`, and `handoff_receipt=False`. Reject lineage_only, handoff +receipts, working-note metadata, and reserved roles. content constrains type, not generation eligibility; apply existing +type/role validation and reject forged internal metadata rather than silently changing its meaning. + +namespace is at most 128 characters; key at most 1024; both nonempty, without outer whitespace or controls. File keys +are relative POSIX paths; reject absolute paths, backslashes, `.`, `..`, empty components; do not case-fold. Do not +store credentials, signed URLs, or local absolute paths. origin is required and namespace/key are caller supplied. +CLI/UI may generate visible labels, but cannot match existing Artifacts by filename. A pasted-text key is a provenance +label, not a stable binding. + +### Logical document identity + +| Input | Stable identity | +| --- | --- | +| First manual upload/paste | Allocate artifact_id even for equal name/content; replay of the same creation key returns that document | +| Explicit update | artifact_id + expected_head_revision; name, format, body may change | +| OpenDAL directory | Scope + stable source_authority_id + relative path; namespace/display name cannot replace authority | +| OpenDAL object storage | Scope + stable source_authority_id + object key; not ETag, mtime, or digest | +| Future knowledge-platform Connector | Scope + trusted platform/connection + native document ID; not title/mutable URL | +| Container member | Parent stable source key + nested stable member keys; ZIP canonical paths without duplicates; email prefers verified unique/stable native IDs | + +source_authority_id is durable configuration identity for a fixed root/bucket and connection, not credentials or run ID. +Changing roots/buckets creates a new source; rotating credentials or changing filters/profiles does not automatically +change identity. Binding creation/use follows Scope connection authorization; arbitrary origin cannot seize another +binding. + +Ordinals are stable within an operation, not necessarily across versions. Messages/duplicate-named attachments without +unique stable IDs use parent input digest plus nested ordinals as snapshot-local identity. Changed containers default to +new documents unless users explicitly select update targets. container_chain retains ancestor digests for provenance, +but stable member keys exclude those digests from cross-version identity. Renames without native stable IDs create new +documents; old documents remain. + +Unique stable-binding constraints prevent concurrent first collection from creating two Artifacts in one Scope. Explicit +manual creation has no filename/digest uniqueness. Equal content may belong to distinct documents; cache deduplication +cannot change identity, authorization, or origin. + +### Existing tables, registration, and reads + +Use `(scope_id, family=importedcontent, artifact_id)` as the only material identity. Reuse pc_artifacts for +content/immutable Revisions, pc_artifact_heads for accepted Head, pc_artifact_lineage_sources for ordered SourceRefs, +and pc_sources for body. Use exact pc_artifact_lineage_artifacts references only for actual upstream Artifact inputs. No +new ImportedContent master/version table or independent ImportRecord. + +Register type, serialization, reads, authorization, and allowed operations using existing registries, profiles, and +route adapters. Adding an enum alone is insufficient; a new global plugin framework is unnecessary. Update OpenAPI +family schemas, Server capabilities, and Dashboard mappings: + +| Capability | This release | +| --- | --- | +| list/get, exact Revision, history, comparison | Supported under Scope `scope.read` | +| texts/prepare/confirm | Supported with scope.read + scope.contribute, creator isolation, transactional validation | +| Generic Artifact POST/PUT | Unsupported; cannot bypass confirmation with forged parts/lineage/success | +| Generic restore/rollback, merge/undo_merge | Unsupported; no adapter/allowlist entry; HTTP and Runtime reject, UI hides actions | +| Model generation, Dream/Candidate publication | Unsupported; no ImportedContent generation/candidate processor | +| Independent Artifact sharing/owner authorization | Unsupported; published content is Scope-visible; creator is audit information | +| Full-text/semantic Artifact search | Unsupported initially; no searcher; namespace filtering is not body search | + +ImportedContent is shared within its Scope. Generic reads currently use artifact.read for most Families and writes have +additional policies. Adapt HTTP access declarations/resolution, runtime authorization profiles, standard list/exact +reads, Source reads, and UI actions consistently. Do not expose list entries whose detail still needs Artifact ownership +or copy Prompt's administrator-only write rules. Creator isolation applies to unpublished inputs/recovery, not permanent +ownership. Revoked Scope readers cannot use old receipts to read bodies. Governance retains separate authorization; +scope.contribute does not implicitly grant it. + +Generic restoration builds Artifact lineage from historical content without guaranteeing this Family's ordered +lineage.sources. Do not enable it. Future dedicated restoration must validate complete parts/exact readable Sources and +define evidence reuse/creation, provenance, Head CAS, and downstream pending/dirty. Generic PUT or history buttons +cannot act as restoration in this release. + +Published list/current/history/exact reads reuse Artifact APIs; compose the body from exact SourceRefs in parts using +existing Source reading. Origin filtering may use rebuildable projections/indexes. Reverse lineage reports actually +referenced parts/versions, not assumed full-document consumption. + +### Minimal internal operation storage + +Durable asynchronous state is not a second material model. The following are proposed storage contracts, not claims that +a reusable generic job table already exists: + +| Storage | Responsibility/constraints | +| --- | --- | +| `pc_import_sessions` | Creator, fixed nonempty existing scope_id, expiry; validate contributable destination at creation; immutable target | +| `pc_import_operations` | operation_id, session_id, family, operation kind, staged input ref/digest, frozen configuration, execution state, lease/generation, phase retries, deadline; execution recovery only | +| `pc_import_operation_items` | Unique `(operation_id,item_id)`; manifest, frozen target, selection/parse/commit state, staged bundle/digest, final exact ArtifactRef; per-member atomic commit | +| Bindings, receipts, input indexes | Unique stable source → Artifact, idempotent request → acceptance/commit receipt, reusable parsing lookup; no duplicated body/version authority | +| Quota ledger/rebuildable query projections | Cross-process admission/release, origin filtering/reverse lookup; not unlocked in-memory counts | + +Support bounded updates, lease scans, and cleanup. Original files/unconfirmed bodies live in a controlled staging store, +not Artifact content or giant ledger JSON. After confirmation, Source + Revision are authoritative for +body/provenance/relationships. Operations retain recovery receipts and expire under TTL. + +No FileImportJob CRUD, permanent job detail resource, or duplicated material catalog. The UI can recover unfinished +operations; operation IDs are not ArtifactRefs, and status handles do not permit arbitrary state mutation. Internal +tables remain necessary, but do not create a new task domain per format. + +### Writes and visibility + +ContentImportApplication owns one database transaction and reuses ArtifactRepository create/revise. Command Sources from +generic Artifact creation cannot substitute for actual body parts. Enforce one-to-one parts/lineage.sources. Every input +uses the same commit service; generic PUT cannot bypass confirmation. + +Before confirmation, new materials have no Artifact/Head/body Source. An artifact_id may be preallocated in the ledger +but is invisible to standard reads. Updating preserves the accepted Head during parsing, failure, retries, cancellation, +and progress changes. Only successful confirmation appends a Revision and switches Head; heartbeats/stages update +operations only. + +pc_artifact_heads.lifecycle_state active/deprecated/retired remains governance state, never uploading/parsing/failed. +Preview is creator-authorized; other Scope readers see published content only. In the commit transaction, lock/check +current permission, Head/governance, and operation validity. Reject retired or merged-away targets under existing +immutability rules; do not revive, follow merge destinations, or expose retired restoration. + +Sessions fix their existing scope_id at creation. Mutable preview/recovery/lease/TTL state never becomes Artifact +Revisions. Section 7 defines target, permission, and idempotency rules. + +## 4. Normalization, locations, and segmentation + +Strictly decode with the selected encoding and remove one leading BOM. Preserve line endings, indentation, and Unicode; +no replacement U+FFFD, global NFKC, or code formatting. Reject whitespace-only bodies, NUL, surrogates, and C0 controls +other than TAB/CR/LF. Do not force control-heavy data through ordinary text import. + +Converted output is UTF-8 with LF under a fixed profile and Section 1 routing. Native Office semantics do not default +through PDF. CSV/code/config markup can remain text/plain; specialized structured output requires explicit validated +selection. Failed JSON/XML parsing cannot silently switch semantics. + +Build an ordered bundle containing complete text, structure, page/sheet/slide/cell/message locations, coverage, and +omissions. Distinguish PDF pages, slides, sheets/A1 ranges, paragraph ordinals, code lines, and notebook cell indexes. +Do not invent Word page numbers. OCR bbox includes coordinate system and image dimensions. Identify whether locations +refer to original input or converted output; converted PDF coordinates are not original cells/slide objects. Represent +disabled, enabled-empty, failed, and unknown coverage separately. Persist text/locations, not permanent links dependent +on temporary screenshots, converted PDFs, or signed URLs. Exact Source reading survives staging cleanup. + +Segment at chapters/pages/slides/table-row groups/paragraphs with at most 256 KiB UTF-8 per part. Split oversized +paragraphs by lines then Unicode boundaries. Pin algorithm version; no overlap/truncation. Concatenating parts by +ordinal must exactly reproduce frozen text and text_digest. Display headings/repeated headers belong in metadata, not +inserted into every part body. + +Per document: at most 8 MiB extracted text, 128 parts, and 2 MiB locations/manifest. Exceeding limits rejects the +document, not its latter half. Each serialized Source must also satisfy the existing 4 MiB observation ceiling. +Processor windows keep their own progress; atomic persistence is not atomic generation. + +## 5. Size, time, and parsing limits + +| Object | Initial default | +| --- | --- | +| Paste | 256 KiB body, 2 MiB JSON request | +| Uploaded file | 50 MiB, measured from actual stream | +| Dashboard selection | 100 files, 200 MiB total, 4 concurrent uploads | +| PDF/presentation/multipage image | 500 pages/slides/frames; at most 200 OCR pages per task | +| Workbook | 100 sheets, 100,000 actual nonempty cells, also bounded by extracted bytes | +| Archive/mailbox | 1,000 members, 200 MiB expanded total, 50 MiB/member, explicitly enabled nesting depth ≤2 | +| Compression ratio | Actual expanded cumulative bytes / consumed compressed bytes ≤100; expanded byte ceiling independently enforced | +| All child documents of one operation | 32 MiB extracted text, 512 parts total, plus each document's own limits | +| Image decode | 40 million pixels; compressed size is insufficient | +| Ordinary/OCR task | 120/600 seconds, separate process CPU/memory quotas | +| Staging | 24 hours; actual expires_at returned | +| Directory scan | 10,000 selected inputs per round; bounded listing/reads | + +Validate these proposed budgets with fixtures; deployments may lower them within validated hard ceilings. Return +format-specific limits, not ambiguous max_file_size alone. Expanded-byte limits also apply inside Office/EPUB. Internal +document nodes use dedicated XML/page/output budgets rather than the ordinary archive 1,000-member limit. + +Errors identify input bytes, expanded bytes, pages, pixels, cells, or output overflow. Enforce while streaming, not +after unbounded loading; declared lengths cannot bypass actual metering. + +### Aggregate quotas and backpressure + +Browser concurrency is only UX. All entry points share server quotas across tabs, CLI, and API replicas. Capabilities +return actual configured defaults and caller-visible remaining budget, never other Principals' usage details. + +| Budget | Principal / Scope / deployment defaults | +| --- | --- | +| Actual staged bytes: originals, expansion, output caches | 1 GiB / 5 GiB / 20 GiB | +| Active task slots, based on work state below | 20 / 100 / 500 | +| Concurrent ordinary parses | 2 / 4 / 8 | +| Concurrent OCR | 1 / 2 / 4; also consumes ordinary parser slots | +| Actually attempted OCR pages per UTC calendar day | 1,000 / 5,000 / 20,000 | + +At acceptance, tasks belong to creator Principal and fixed session scope_id, even for administrative actions. All three +quota levels must permit admission; multiple sessions share Scope budgets. Reservation/settlement/release/retry always +use that fixed Scope: no temporary Scope quota or quota migration. Atomically reserve task slots and declared bytes in +the shared durable ledger before upload; unknown lengths reserve the file maximum. Enlarge reservations before consuming +more, or stop. Before parsing, reserve bounded expansion/output space, running slots, and OCR pages. Settle actual +streaming usage without writing beyond budget first. Rejected upload admission creates no task. Accepted work blocked on +parser reservations stays durably queued with blocked_reason/retry_after and no running slot. + +Task slots and parser/OCR slots are distinct. An operation holds at most one task slot, updated transactionally with +state: + +| Work state | Task slot | +| --- | --- | +| uploading/queued/running, including resource waits | Held | +| idle after discovery awaiting selection, or selected uncommitted ready/needs_review documents | Held | +| partially_committed with queued/parsing/selection/confirmation work | Held | +| failed, or partially_committed with only failures and no scheduled automatic retry | Released; explicit retry needs readmission | +| completed/cancelled/expired | Released | + +Scheduled automatic retries remain queued and hold a slot; exhausted retries enter failed and release it. Slot release +neither releases extant staged bytes nor removes retry eligibility. Any transition back to pending work from a slotless +operation must atomically reacquire all three levels; multi-document retry still needs only one slot, and held slots are +not charged twice. Insufficient quota returns 429/503 with Retry-After without changing task/item/generation/retry +counters, enqueueing, or successful acceptance receipts. The same key may retry admission after backoff. Replaying an +accepted key returns its receipt without another reservation. + +Bind running slots to fenced leases. Replicas share transactions or equivalent atomic ledgers. Cancellation, inactive +failure, and expiry release unused reservations, but one member failure cannot release resources needed by others. Disk +budget releases only after actual deletion; task completion does not imply blob removal. Attempted OCR pages are not +refunded on failure/cancellation; retries consume more, only unused reservations release. Recovery reconciles +tasks/blobs/leases; orphan files consume quota until deleted. Long-term Source storage follows existing governance +separately. + +Principal/Scope exhaustion returns 429; deployment/dependency exhaustion returns 503, both with Retry-After; per-input +hard limits remain 413. Queries expose waits for already accepted tasks. Workers durably schedule jittered backoff +instead of hot-loop reuploads. + +## 6. Internal execution, private previews, and durable staging + +Family prepare accepts complex files and returns recoverable operation_id, backed by Section 3's ledger and isolated +workers. Revisions represent accepted snapshots only. “Task” below means internal operation execution, not FileImportJob +or another Artifact type. + +Reuse validated transactions, leases, fencing, backpressure, and worker primitives. Do not repurpose Artifact Processing +Supervisor generation jobs directly: they lack upload staging, private preview, and human confirmation semantics. Shared +infrastructure must preserve these constraints. Parse success freezes a bundle without writing Source/Artifact. + +Store execution, parse, and commit states separately. ready is not publication or governance state. + +| Level | Field/values | +| --- | --- | +| Task execution | execution_state: uploading / queued / running / idle / failed / cancelled / expired; idle means no current parse work | +| Phase | phase: upload / discovery / selection / parsing / confirmation / done; discovery_state: not_required / pending / running / complete / failed | +| Selection | selected; container membership frozen through selection API after discovery | +| Document parsing | parse_state: pending / parsing / ready / needs_review / failed; ready/needs_review refer to immutable bundles | +| Document persistence | commit_state: uncommitted / committed; no exposed half-commit state | + +```mermaid +flowchart TD + U["Task: uploading"] --> Q["Task: queued"] + Q --> R["Task: running; discover documents"] + R --> I["Task: idle; awaiting selection or confirmation"] + I -->|Frozen selection with pending documents| Q + R -->|Transient failure; at most three automatic retries| Q + R -->|Task-level failure or retries exhausted| F["Task: failed"] + F -->|Retryable discovery; valid input and quota readmission| Q + Q -->|Cancel or expire| X["Task: cancelled / expired"] + R -->|Cancel or expire| X + I -->|Cancel or expire| X +``` + +```mermaid +flowchart LR + P["Document: pending"] --> R["Document: parsing"] + R --> A["Document: ready"] + R --> B["Document: needs_review"] + R --> F["Document: failed"] + F -->|Retryable; valid input and quota admission| P + A -->|Confirm frozen digest| C["Persistence: committed"] + B -->|Confirm digest and accept partial| C +``` + +Derive aggregate_status and counts in priority order: cancelled/expired; uploading while receiving; queued/parsing with +pending/running work; partially_committed when some documents committed, not all complete, and no active work; completed +when nonempty selection is all committed; awaiting_confirmation for uncommitted ready/needs_review or unselected +discovered members; failed for task-level failure or only unparseable members. Unselected members count as excluded, not +required commits. Cancellation/expiry still return committed counts and exact Revisions, preserving partial success +visibility. + +Single-document input automatically fixes selected=true. Archives/mailboxes/attachments require bounded complete +discovery, stable item_ids, container_chain, types, and default suggestions. Attachments default unselected; ordinary +archive members default selected. Pause after discovery; selection API freezes one explicit set before body parsing. +Overflow/incomplete discovery is a discovery error, not a complete list. Structural container nodes are not selectable; +omitted attachments are excluded. Changed selection/options require a new attempt. Persistent item mapping preserves IDs +through automatic retries. Publish complete manifest and discovery_state=complete atomically; intermediate lists cannot +be selected or advance checkpoints. Transient failure records discovery_state=failed, phase error, discovery_retryable. +Explicit discovery retry remains possible after automatic retries exhaust. Before publication it may rediscover staged +input while reusing internal identities; published manifests/IDs cannot be rediscovered or rewritten. +Corruption/encryption/overflow are permanent, nonretryable errors. + +Return 202 and queue only after bytes are durable in controlled staging and digest-verified. Interrupted uploads never +enqueue partial files; GC reconciles placeholders/orphans. Single-node staging may use a controlled state directory; +replicas need shared durable storage, not API-process /tmp. Persist pinned profile, digest, lease owner/generation, +attempt, deadline, selection version, frozen manifest. Claims, publication, retries, cancellation, and expiry +transactionally check state/generation; stale workers cannot overwrite results. + +Automatically retry transient failures at most three times; permanent format/encryption/corruption/overflow failures are +not retried. Within an operation, retry only unfrozen documents, never overwrite ready/needs_review/committed. Explicit +retry separates discovery and document phases, only recovering unfrozen retryable failures. Track automatic counters per +phase; manual retries do not reset them and retain deadline/quota/attempt accounting. Confirm the returned +bundle_digest; do not preview A and save recalculated B. ready means complete within declared scope with required +semantics satisfied. Known omissions or unprovable completeness require review; missing required semantics outside +allowed partial behavior fails. Persist accept_partial=true permanently. + +Operation TTL is at most 24 hours from upload-placeholder creation and never beyond session.expires_at; retry/selection +do not extend it. Expiry/cancellation first fences uncommitted writes and leases, then cleans staging. Serialize with +confirmation: a winning commit retains its receipt; winning cancellation/expiry rejects uncommitted confirmation. +Cancellation never deletes published content. Keep status/idempotency mappings until 30 days after expires_at; status +returns a 200 tombstone with state/times/durable receipt references. Uncommitted preview/confirm after expiry returns +410; after cancellation, 409. Same upload key returns original operation/tombstone, never implicit creation. After +idempotency_expires_at only permanent Revision reads are guaranteed; use new attempt keys. + +Clean originals/parsing caches within one hour after all selected members commit, or after cancellation/expiry; bytes +remain charged until deletion. Incomplete containers retain staging until expires_at. Committed preview reads Sources; +receipts, locations, omissions remain durable. Staging has no arbitrary public read URL or long-term binary download +promise. + +## 7. Identity, idempotency, and atomic persistence + +### Sessions and Scope + +Require an existing scope_id and current scope.read + scope.contribute at session creation. Persist creator and +immutable nonempty target; no deferred resolution or Scope-creation parameters. Creating a session creates no +Scope/Artifact/Source or permission grant. + +Session, operation, item, quota, and eventual Artifact/Sources share that Scope. Child routes derive it from session_id; +bodies cannot override scope_id. Cross-Scope artifact_id, SourceRef, or bindings cannot target the session. Changing +Scope requires a new session and inputs; existing operations stay put. + +Reauthorize session creation, text, upload, selection, retry, and confirmation. Queries/previews require creator plus +current Scope access, with explicitly authorized/audited administrative exceptions. Confirmation also checks destination +existence/writability, session/operation expiry/cancellation, Head, and governance. Invalid targets or revoked +permissions cannot trigger repair/creation or alternate destinations. Cancellation/cleanup follow Sections 6 and 10. + +Sessions accept uploads for 24 hours from creation; new operations/children do not extend this and operation expiry +cannot exceed it. Session/operation receipts remain at least 30 days after the last operation expiry. Session expiry +prevents uploads/uncommitted saves but preserves committed receipts and the Scope/Artifacts/Sources. + +### Separate content, identity, and requests + +Require Idempotency-Key, 1–128 ASCII letters/digits or `-_.`. Session creation uses stable Principal + operation + key, +with normalized scope_id in its fingerprint; same key/different Scope returns 409. Other writes use Principal + +session_id + operation + key. Equal request/key replays the original result; different requests conflict, and all +replays reauthorize. Retain initially frozen defaults/profile. File fingerprints cover actual digest, normalized +metadata, and target intent. Every fingerprint includes action and applicable path/body operation_id/item_id; +confirmation additionally includes bundle_digest, accept_partial, frozen target/expected Head. + +Four identities differ: artifact_id is logical document; operation_id/item_id/attempt key is an attempt; bundle_digest +is frozen extraction; ArtifactRef is an accepted version. snapshot_digest covers content/acceptance and may recur across +documents or historical versions. Never impose global `(scope_id,snapshot_digest)` uniqueness. + +Compute bundle_digest as SHA-256 of RFC 8785 canonical JSON over schema_version, source_kind, controlled origin, +container_chain, title, input type/bytes/digest, parser/profile/encoding/OCR and normalization/segmentation versions, +text type/bytes/digest, ordered content_manifest, coverage, structured warnings/omissions. Manifest includes only +ordinal, bytes, part digest, full-text interval, locations; no staging refs. Exclude diagnostic messages, elapsed time, +timestamps, Principal, session/operation/run/item IDs, artifact_id/revision, SourceRef, journal position. Compute +snapshot_digest over canonical `{schema_version, bundle_digest, accepted_partial}`; accepted_partial=true only for +needs_review. Persistence adds parts receipts without changing frozen digests. + +### Revision, Head, and repeated inputs + +| Situation | Behavior | +| --- | --- | +| First manual creation | Allocate artifact_id, save Revision 1; another create intent with equal name/content is another document | +| Update with matching expected Head and current snapshot_digest | reused + current exact ArtifactRef; no new Source/Revision | +| Update with matching Head but different snapshot | New Revision/parts and atomic Head advance; preserve old evidence | +| Expected Head mismatch | 409 revision_conflict; display current version and require reconfirmation, even if content matches | +| A → B → A | R1(A), R2(B), R3(A); no global-digest shortcut back to R1. R3 creates its own Sources in this release | +| Replay successful key | First exact receipt even if Head advanced; no new version or Head rollback | +| Reparse same input | reparse=true creates attempt; equal current frozen semantics reuses, changed output or semantically relevant parser/profile creates Revision | + +Manual updates freeze target/expected Head at item creation. Collection resolves binding/expected Head at acceptance; +new sources freeze expected absence. Binding uniqueness races return recoverable conflicts, never automatically +overwrite a newer version with old parsing. A retried create intent retains its allocated identity; independent creates +never merge by filename/digest. + +### Parsing reuse and partial extraction + +Cross-key operation reuse requires same creator/session, explicit target/expected Head, original input/origin/title, +pinned profile/options, and frozen selection. Independent manual create intents never merge. Eligible work is unexpired +with valid staging, queued/running/idle, unfinished, and no committed partial. Exclude +failed/cancelled/expired/completed. Serialize reuse and request mapping. Same upload key always restores its original +operation/tombstone; reparse=true disables merging. Never return another Principal's private task/preview. + +Durable input mappings key explicit logical target, input digest, controlled origin/title, profile, and complete +options; containers include stable member mapping/selection. They index frozen extraction and exact historical +ArtifactRefs, not overwrite authority. Cache may skip OCR but current target/expected Head still determines reuse/new +Revision. A historical version is not current automatically; manual create remains new, and cross-Scope hits stay +hidden. + +Complete cache output still needs this interactive attempt's confirmation; only explicitly configured accept-complete +workers auto-confirm. Partial cache yields needs_review/uncommitted with candidate_ref, coverage, omissions and no +completed count. Require this attempt's explicit accept_partial=true, never inherit another Principal's or earlier +attempt's decision. Same successful commit-key replay preserves its original decision. + +reparse=true requires resubmitted bytes and recorded actual execution. Temporary operation history owns runtime +diagnostics; Revisions retain content-affecting provenance. Cache pointers may refer to history but never change Head. +The latest complete parse may update the cache index while historical versions remain exact-readable. + +### Atomic commit boundary + +One document transaction includes session/operation/item and necessary binding/Head locks, current authorization/state +checks, idempotency, fixed-Scope quota settlement, all ContentSources/journal positions, durable pending/dirty, +Revision, Head CAS, ordered lineage, input mappings, item result, and receipt. reused adds no parts but atomically +records acceptance/receipt. Any failure rolls back everything. + +Determine Source metadata's target Revision inside the transaction, outside digests; verify manifest/parts/lineage +order. Independent per-part Source HTTP calls cannot provide document atomicity. SQLite, embedded seekdb, and OceanBase +require transactions and uniqueness/CAS, not process locks. Persistent document versions/evidence are not automatically +collected here. + +Existing record capture opens its own transaction, so do not loop it and separately save Artifact. Extract/reuse an +internal connection-taking save unit with ContentImportApplication owning the outer transaction: + +1. Validate authorization, session/item, receipts, governance, binding, expected Head; acquire locks in a consistent global order and determine target Revision. +2. Validate frozen bundle/manifest and ordinary Sources; call `SourceRepository.add_with_status(connection, ...)` per part for exact refs/positions. This Repository already calls processing-intent `mark_dirty` for inserted Sources in the transaction; retain that single responsibility rather than another dirty implementation. +3. For created=True, preserve capture's configured Source-processing-binding rules and durable pending registration. Existing identical Sources do not trigger again. The shared unit must retain existing capture behavior, not omit pending after Repository calls or add an ImportedContent generation consumer. +4. Build parts and lineage.sources from actual results; call connection-taking `ArtifactRepository.create/revise`. Commit Head CAS, quotas, bindings, item result, idempotency receipt together. Any part/pending/Artifact/receipt failure rolls back all. +5. Notify and return success only after commit. Durable pending/dirty recover lost notifications. reused/replays add no Sources, Head advances, or processing registrations. + +Lock/CAS failures follow conflict/replay rules, not a second transaction that patches missing evidence or pending work. +Background processors can see only committed document parts, but their windows need not consume the whole document in +one generation. + +## 8. HTTP, client, and page contracts + +Published materials reuse standard Artifact APIs with family=importedcontent. Extend registration/schemas for these +existing operations instead of a separate material resource: + +| Existing operationId | Method/path | Behavior | +| --- | --- | --- | +| `list_artifacts` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent` | Visible current Heads; exact namespace filter | +| `get_artifact` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent/{artifact_id}` | Current accepted snapshot and ordered parts | +| `list_artifact_revisions` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent/{artifact_id}/revisions` | Accepted history | +| `get_artifact_revision` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent/{artifact_id}/revisions/{revision}` | Exact immutable snapshot; receipt Location points here | + +Imports are Family-specific writes. Select an existing Scope before creating a session; prepare/confirm derive its fixed +target from session_id. Session routes recover a batch, not a new permanent file-import business resource. All +operations below are proposed, not existing recovery APIs: + +| operationId | Method/path | Meaning | +| --- | --- | --- | +| `get_import_capabilities` | `GET /v1/import-capabilities?scope_id=...` | Required existing Scope; content Source type, formats, permissions, limits | +| `create_import_session` | `POST /v1/import-sessions` | Existing scope_id required; validate contribution and fix destination/creator | +| `list_import_sessions` | `GET /v1/import-sessions` | Creator's recoverable sessions, paginated | +| `get_import_session` | `GET /v1/import-sessions/{session_id}` | Fixed scope_id, expiry, receipt summaries | +| `import_content_text` | `POST /v1/import-sessions/{session_id}/importedcontent/texts` | content; synchronously validate/confirm bounded body, 201/200 | +| `prepare_imported_content` | `POST /v1/import-sessions/{session_id}/importedcontent/prepare` | content; metadata + one file; 202 and operation_id after durable acceptance | +| `list_import_operations` | `GET /v1/import-sessions/{session_id}/operations` | Creator's recovery list, not material catalog | +| `get_import_operation` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}` | Execution/errors/deadlines/counts/available actions | +| `list_import_operation_items` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}/items` | Bounded manifest; single files also have one item | +| `get_import_operation_item` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}/items/{item_id}` | Manifest, coverage, warnings, omissions, frozen target | +| `get_import_operation_part` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}/items/{item_id}/parts/{ordinal}` | Creator's bounded preview; committed body from Source | +| `confirm_imported_content` | `POST /v1/import-sessions/{session_id}/importedcontent/confirm` | operation_id + item_id + bundle_digest + accept_partial; atomic save, 201/200 | +| `select_imported_content` | `POST /v1/import-sessions/{session_id}/importedcontent/selection` | operation_id, members and individual targets; 202 after frozen selection | +| `retry_import_operation` | `POST /v1/import-sessions/{session_id}/operations/{operation_id}/retry` | Controlled Family retry; 202 after readmission | +| `cancel_import_operation` | `POST /v1/import-sessions/{session_id}/operations/{operation_id}/cancel` | Fence uncommitted writes; atomic cancellation/completed result, 200 | + +prepare's 202 returns `{operation_id, session_id, family: "importedcontent", expires_at, status_url}`. Location points +to status, not a nonexistent Artifact. texts is itself confirmation and returns the same saved receipt as confirm +without a separate parsing operation. Operations/items have no generic PUT/PATCH state mutation. Confirm/selection +verify operation, session, creator, and Family; cross-session/invisible handles follow existing 404/403 policy. Generic +Artifact POST/PUT for importedcontent returns 422 from Family capability validation, without falling back to command +Sources. + +Session creation accepts `{"scope_id":"existing Scope ID"}`. Require nonempty valid ID; missing, empty, unknown fields, +or Scope-creation parameters return 422. Missing/invisible/unauthorized targets follow existing 404/403 visibility +policy without accepting files or creating sessions. Return session_id, fixed nonempty scope_id, expires_at. Every write +requires Idempotency-Key; no target-mutation API. + +Targets are `{"mode":"create"}` or `{"mode":"update","artifact_id":"...","expected_head_revision":2}`. Manual defaults +to create, never infers identity from origin. Updates must target this Scope and Family; cross-Scope targets follow +visibility rules. Authorized registered Connector Principals may instead use the mutually exclusive +`{"mode":"source","source_authority_id":"...","source_key":"..."}`. Resolve bindings and freeze target/expected Head or +absence at acceptance; return the resolved target. origin cannot impersonate authority. Each container member has its +own target; multiple members cannot update the same artifact_id. + +A paste request stays simple: + +```json +{ + "title": "API conventions", + "artifact_target": {"mode": "create"}, + "source_kind": "content", + "media_type": "text/markdown", + "origin": {"namespace": "project-notes", "key": "note-001"}, + "text": "# API conventions\n\nPagination uses opaque cursors.\n" +} +``` + +Upload metadata contains artifact_target, title, origin, parser_profile, optional fixed-content source_kind, and +optional encoding, as_text, reparse, ocr_mode, language, include_notes, include_comments, include_hidden, archive_depth. +Defaults: as_text=false, reparse=false, ocr_mode=off, include_notes=true, include_comments=false, include_hidden=true, +archive_depth=1. Omitted language uses the profile's fixed language set; omitted encoding follows identification or +returns encoding_required. Without parser_profile, freeze the deployment router/profile registry version, identify +format, and freeze its default profile; PDF chooses its backend under that profile. Containers also freeze child routing +registry so discovered/selected children cannot adopt changed defaults mid-operation. Across-key retries should send the +resolved profile digest. include_notes applies only where supported; irrelevant defaults are ignored, explicitly +inapplicable nondefaults return 422. archive_depth is 1 or 2. Publish all defaults through capabilities. + +as_text=true still requires strict text validation, binary identification, and resource limits. ocr_mode is +off/auto/force; auto handles required regions only, force is disclosed in preview. reparse=true needs new upload and +Idempotency-Key with original bytes; do not assume old staging exists or alter old bundles. Selection controls +attachments. + +selection accepts `{operation_id, expected_selection_version: 0, items: [{item_id, artifact_target}, ...]}` only after +complete discovery, before selection freezes, and while valid. IDs must be selectable, unique, nonempty, from the same +complete manifest, at most 1,000. Validate targets/no duplicate Artifact updates, atomically freeze version 1, and +enqueue. Changed selection returns 409; unknown/foreign IDs return 422 without disclosure. Single documents need no +selection call. + +retry requires one of `{stage: "discovery"}` or `{stage: "documents", item_ids: [...]}`. Discovery forbids item_ids and +requires discovery_state=failed, discovery_retryable=true, unpublished manifest, valid staged container. Acceptance sets +discovery_state=pending, phase=discovery, execution_state=queued. Document retry requires complete/not-required +discovery and 1–1,000 unique selected failed retryable item IDs; set these pending and enqueue. Invalid structures +return 422; phase/state mismatches return 409 for the entire request; neither alters frozen output. Both branches return +202 only after atomic task-slot admission, enqueue, and idempotent receipt. + +cancel accepts an empty object. First cancellation is atomic; repeats return the same receipt. Completed returns its +completion result; expired returns tombstone. No published Revision is deleted. Selection/retry/cancel require creator, +fixed-Scope scope.read + scope.contribute, and Idempotency-Key, with Section 10 administrative exceptions. Replays +retain acceptance receipts; GET shows later progress. Selection/retry on cancelled return 409, expired 410. + +Operation responses independently expose operation_id, session_id, family, phase, discovery_state, discovery_retryable, +phase errors, execution_state, aggregate_status, task_slot_held, selection_version, counts, expires_at, +idempotency_expires_at, blocked_reason. Item responses include selected, parse_state, commit_state, retryable, coverage, +warnings, omissions, bundle_digest, optional candidate_ref and artifact_ref (exact, Scope-bearing ArtifactRefs). +artifact_ref means this attempt committed; candidate_ref does not. Clients must not infer a combined status. + +Each upload accepts exactly one file and one metadata object. metadata ≤64 KiB; reject unknown fields and extra +multipart fields/files. Queries/manifests do not return raw blob addresses. + +confirm accepts `{operation_id, item_id, bundle_digest, accept_partial}` with Idempotency-Key; target and expected Head +are already frozen and echoed in the response. Changed bundle returns 409; expired uncommitted item 410; +cancelled/unparsed item 409. Committed items replay durable receipts throughout mapping retention even if the operation +later expires; thereafter use standard exact Revision reads. ready requires accept_partial=false; needs_review requires +true, otherwise 409. Return `{outcome: created|reused, scope_id, artifact_ref, snapshot_digest, parts}`. created +includes first and subsequent new Revisions. artifact_ref contains family, artifact_id, revision, with Location at the +standard exact Revision resource. New import responses uniformly use parts[]; pasted text has one part. Existing Source +responses are unchanged. + +Task/item/material lists default to limit 50, range 1–100, SQL keyset pagination with a first-request high-water mark. +Bind cursor to operation, caller, stable session_id or scope_id, filters, limit, high-water mark, and TTL; reauthorize +each page. Mutable task states mean lists are not state snapshots. Namespace filtering extends the Artifact-list +projection, not full-text search. + +Python client exposes these operations. CLI provides text, files, resume, cancel, dry-run. New imports require --scope +before uploading; --session resumes its fixed destination and displays it. If both are supplied they must match. +--artifact-id and --expected-head-revision explicitly update; expose --as-text, --reparse, member selection. Containers +list members before selection; unattended use needs explicit rules. dry-run validates only locally knowable facts: +without running a parser offline, report server parsing required rather than full importability. Default CLI behavior +confirms interactively after deployment parsing. Explicit --accept-complete can auto-save ready; partial needs +additional explicit permission, including cached partial snapshots. UI/CLI offer discovery retry or failed-document +retry based on phase/discovery_state, without requiring undiscovered item IDs. Quota rejection backs off per Retry-After +and cannot appear queued. + +Reopening/previewing is read-only; logging in again does not submit. Confirm carries frozen digest/original key. Verify +uncertain outcomes through status/receipts or same-key retry. Separate upload, parsing, and Source persistence progress; +show indeterminate processing if the parser cannot report pages. + +Conflicts do not mutate frozen targets. Read current Head, then create a new file attempt with new expected Head, +original bytes, and new key; parsing cache may help, but old items/receipts stay immutable. Show current-version +differences before confirmation. Generic writes cannot rewrite provenance/parts/lineage. + +### Published bodies, history, and comparison + +Reuse Dashboard history pagination, Revision selection, and comparison pages with a Family-specific read projection, not +another version system. Exact historical parts read their Sources in batches, preserve order, and detect +missing/digest-invalid evidence. Do not depend on expired operations/staging or substitute current Head text for missing +historical parts. + +Compare body text plus meaningful origin, parser/profile, OCR, coverage, omissions, accepted_partial changes. +SourceRefs, positions, digests, timestamps are provenance details, not a substitute JSON diff. Lazily load parts with +bounded comparison characters/output/compute. Over-budget output says comparison is incomplete and offers part-level +reading without silently truncating exact originals. Hide restore/rollback/merge buttons and reject direct API attempts +too. + +No ImportedContent body search in this release. The existing `/v1/scopes/{scope_id}/artifacts/{family}/search` returns +its standard unsupported result for this Family; namespace filtering is metadata only. Future keyword/semantic support +registers an ArtifactSearcher with ArtifactSearchService, defining Scope authorization, version selection, and index +updates, rather than a duplicate search endpoint or implicit vector-ready promise. + +Existing stable Artifact handles, if used, bind one exact immutable Revision. They are not live external-document +identity or a replacement for source bindings. + +### Examples + +1. Paste Markdown: select an existing Scope, create a session with its scope_id, then call importedcontent/texts with source_kind=content. Receive exact artifact_ref; read via standard Artifact get and Source parts. Failure leaves no partial Artifact. +2. Upload PDF: importedcontent/prepare returns operation_id for status/paged preview. Confirm operation_id, item_id, bundle_digest, accept_partial; read the resulting importedcontent Revision. PDF counts/OCR are extraction metadata, not another Family. +3. Update: explicitly supply artifact_id/expected_head_revision in prepare/texts artifact_target. Old Head stays readable during parsing; only confirmation creates a Revision. Failure/cancellation creates no failed version or governance change. +4. Upload HTML: source_kind=content, input_media_type=text/html. A webpage description in origin triggers no network access and changes no Source type. +5. Future webpage Connector: after separate validation, an authorized adapter supplies a frozen page snapshot and reuses content evidence, commit service, and Artifact reads. No link-import endpoint/UI is available now; source_kind=content never enables fetching. + +## 9. Continuous collection with OpenDAL + +Add `opendal-text-imports` while retaining the existing opendal-text-files/text-file-snapshot evaluation Definition, +naming, permissions, and checkpoint behavior. Use public import APIs, not the old RemoteConnectorWorker path that +registers manifests. Require configured existing scope_id plus scope.read + scope.contribute, not server.admin. Missing +destination/permission fails without Scope creation. Use the same ImportSession protocol. + +Use the same matrix/profiles. Each scan obtains a complete bounded selected-input list, reads actual bytes, and hashes +input_digest. mtime/ETag aid diagnostics but alone cannot prove unchanged content. Identity is Scope + +source_authority_id + relative path/object key. Profile/include-exclude/encoding/OCR changes explicitly create a new +collection configuration binding/checkpoint while retaining document identity under the authority; changed snapshots +create Revisions. New root/bucket means new authority. Display namespace and checkpoint configuration fingerprints do +not define document identity. + +Checkpoint stores completed input identity, raw digest, profile, and exact Scope-bearing ArtifactRef. Stable input_key +derives from binding, input identity/digest, profile, and complete selection/parsing policy; it is not directly the +upload key. + +A durable recovery table separate from successful checkpoint stores `(binding, input_key, session_id, attempt_uuid, +operation_id, expected_head_revision, idempotency_expires_at, retry_after)`. Derive upload key from input_key + +attempt_uuid, persist attempt_uuid with transaction/CAS before sending. Lost responses/crashes resume the same +attempt/key. Replicas racing one input share the durable attempt; different service Principals never share private +recovery tables. + +On expired tombstone, retain committed receipts, then CAS-allocate new attempt_uuid/key and resubmit bytes. Expired +sessions require a persisted new session for the same existing Scope. Also start new attempts when mappings expired and +cannot be recovered. Reuse frozen results through stable mappings but check current/expected Head. Recover committed +receipts without resubmitting those commits; continue unsaved members instead of being permanently stuck at 410. After +discovery, select a stable member set under explicit rules. Input completes only when all expected members have durable +receipts. Recovery-table progress is not checkpoint progress. User/admin-cancelled input is paused; later scans cannot +recreate it until explicit resume with a new attempt key. + +Automatic confirmation must be configured and accept only coverage=complete. Apply this to fresh parses, cached +candidates, and recovered receipts: an existing partial Revision cannot satisfy complete-only or advance checkpoint. +Workers cannot accept historical partial on the user's behalf. Retry discovery with stage=discovery and document +failures with stage=documents; quota rejection preserves state and durable backoff. Missing OCR, damaged attachments, +unsupported selected members, or partial extraction need attention, never silent acceptance. CAS checkpoint only after a +complete round listing and saved/reused receipts for every selected member. operation creation, upload success, or ready +parsing alone is insufficient. + +Other valid files may commit; failures do not roll them back, but retain the old checkpoint. Recover with original keys +while valid, new attempt keys after expiry, and existing receipts. CAS conflicts reload/rescan, never force overwrite. +External deletion affects comparison only, not Artifact/history/Source deletion. Incomplete listings, permission denial, +or connection errors are not deletion evidence. + +Prefer immutable external revisions or conditional reads. Otherwise promise only the bytes actually read, never a +timestamp disguised as exact revision. Credentials stay in the worker, not Sources, checkpoints, logs, or parsers. + +## 10. Security, authorization, and isolation + +Text/file writes require selected Scope scope.read + scope.contribute; lists, history, and Source reads require +scope.read. Ordinary imports do not require server.admin, create Scopes, or initialize access. Sessions/tasks bind +stable creator Principal. Other ordinary callers cannot read uncommitted previews or confirm them. Administrative access +requires explicit server.admin plus corresponding Scope permissions, with audit. Reauthorize each confirmation; old +sessions cannot survive revoked write authority. + +Parser workers receive only assigned operation, staging blob, profile, not arbitrary Scope write permission. +ContentImportApplication commits as the confirming caller. Background parsing may finish accepted input but cannot +create Sources itself. Revocation blocks confirmation; staging expires normally. + +Dashboard write routes validate same-origin Origin and forward public APIs using existing session identity. Preserve +HttpOnly/Secure/SameSite/cookie-path rules, no direct database writes, expanded `/v1` cookie authority, or untrusted +proxy headers. + +Run parsers in restricted processes/containers with CPU, memory, time, file-count, pixel limits. Disable network by +default; prohibit XML external entities, scripts, macros, formula recalculation, link refresh, notebook execution, +dynamic plugin installation. Filenames cannot choose arbitrary disk paths. Raw and extracted content are data, not +execution instructions. + +Archives reject traversal, absolute paths, symlinks/hardlinks, device files, duplicate overwrites. Meter actual +expansion, not just index-declared sizes. Do not accept decryption passwords; encrypted_document for encrypted input, no +DRM bypass. + +Default OCR is local with installed language packs. Cloud OCR/vision needs explicit deployment enablement and visible +external-transfer disclosure, not unknown-provider fallback. Ordinary extraction does not implicitly invoke generative +models. Meter OCR, vision, and downstream Artifact usage separately. + +## 11. Errors and observability + +Retain the error envelope and distinguish unsupported_format, parser_unavailable, capability_insufficient, +encoding_required, invalid_encoding, encrypted_document, damaged_document, ocr_required, no_extractable_text, +parse_timeout, input_limit_exceeded, output_limit_exceeded, archive_limit_exceeded, partial_extraction, +idempotency_conflict, revision_conflict, source_binding_conflict, session_expired, bundle_conflict, selection_conflict, +operation_cancelled, operation_expired, quota_exceeded. + +401/403/404 follow existing visibility. Invalid fields 422; hard request limits 413; unsupported media 415; +idempotency/state/cancel conflicts 409; expired uncommitted operations 410; throttling 429; dependencies unavailable +503. Failures after 202 belong to operation state returned by a 200 query, not a retroactive upload HTTP failure. +Network timeouts may mean outcome unknown; verify with the same key. + +Logs exclude body, tokens, full local paths, Prompts, OCR images. Metrics distinguish uploaded bytes, detected formats, +parse outcomes, OCR pages, confirmation-needed items, committed documents/parts, reuse hits, parse/commit latency, +checkpoint conflicts, last successful completion. High-cardinality IDs are not metric labels. + +“Saved” counts durable document receipts only. Container summaries separate saved, failed, excluded, and pending. +Metadata-only/empty bodies are not success. Downstream cost/progress must come from processor evidence, not parsing +success. + +## 12. Compatibility, implementation, and acceptance + +### Implementation boundaries + +Implement Section 1's fixed parser families/routes, not ad hoc runtime library selection. Stage work by responsibility: +text persistence/unified manifest, Family prepare/confirm plus durable ledger/standard parsing package, then +OCR/long-tail/OpenDAL. Both A and B remain formal targets; A alone does not complete this RFC. + +Register importedcontent, operation storage, indexes without changing existing Source/Artifact references. Historical +Sources are not automatically wrapped as ImportedContent. Update allowlists, exact reads, history, lineage, Dashboard +readers without generation/Candidate inclusion. Do not switch old OpenDAL bindings in place; explicit reimport may +create new content Sources, without claiming old type identity preservation. Core Python support remains unchanged; +heavy dependencies belong to separate workers. + +### Database initialization, upgrades, and recovery + +All persistent structures must support SQLite, embedded seekdb, and OceanBase. Version-manage Section 3's three core +tables plus chosen bindings, receipts, quota ledger, and indexes; do not lazily create auxiliary tables on first +request. Published material remains in Artifact/Source tables, without new ImportedContent master/version tables. + +`server db-migrate status/plan/apply/verify` supplies infrastructure, but current registered bundles manage limited +tables, not all business schemas. Register import schema bundles/revisions, frozen table/index manifests, postcondition +verification, dependencies, compatible versions. Do not modify already released revisions. + +| Area | Required constraint | +| --- | --- | +| Empty/existing databases | Initialization and upgrade yield the same schema; inspect managed tables/indexes/history/conflicting residues, never automatically adopt arbitrary existing tables as migrated | +| Uniqueness/indexes | Session/operation keys, `(operation_id,item_id)`, stable source identity, partitioned idempotency uniqueness; bounded recovery/GC indexes by creator, Scope, state/lease expiry, TTL. Long keys use collision-checked digest indexes where backend limits require, never silent truncation | +| Maintenance gate | API writes, parser dispatch, Connector commits, GC obey migration maintenance; no import before schema ready or mixed-version replicas writing incompatible structures | +| Interrupted DDL | SQLite uses transactions as supported. seekdb/OceanBase MySQL-compatible DDL may implicitly commit, so whole-upgrade rollback is not guaranteed. Record steps/results; inspect/plan before resume/manual recovery, no blind repeat of nonidempotent DDL | +| TTL/runtime data | Session/operation/receipt/staging cleanup follows Sections 6–7 and preserves tombstones/receipts. Stable bindings, published Artifacts/Sources, and referenced mappings do not expire with operations | +| Downgrade/release | Disable new imports without deleting materials; no unverified schema downgrade. Bundle verification proves only managed postconditions, not whole-database/authorization/end-to-end readiness | + +Deliver empty-database, existing-data upgrade, repeated execution, interrupted recovery validation on all three +backends; existing Sources/Artifacts/exact refs must remain intact. Ship scripts/manifests/indexes/initialization with +the feature; SQLite development auto-creation alone is not production support. + +### Contract and acceptance + +Use openapi/powercontext.yaml as source of truth, run make api-generate and make contract-test, never hand-edit +generated code. Implementation validation includes make check, relevant Source/Server/Access/Dashboard/Connector tests, +make docs-test, and tox when Python support changes. Cross-component acceptance belongs in tests/e2e/. RFC acceptance +does not mean these feature tests have passed. + +| Scenario | Acceptance requirement | +| --- | --- | +| Complete Scope-authorized read/write path | scope.read + scope.contribute suffice for import/list/detail/history/body without Artifact owner/Prompt administration; corresponding paths stop after revocation | +| Generic restore/merge, POST/PUT, retired targets | HTTP/Runtime both reject without orphan Revision/lost lineage.sources; no UI actions or retired revival | +| Ordinary content roles/eligibility | working_note=False, internal=None, handoff_receipt=False; forged roles rejected; eligible parts reach configured processors | +| Atomicity, replay, lost notification | Parts, dirty/pending, Artifact, Head, receipt commit/rollback together; replay does not retrigger; lost notify remains recoverable | +| Three-backend schema lifecycle | Equivalent constraints, existing data readable, no false ready after DDL interruption; replicas obey maintenance | +| Static/runtime/Scope discovery | Contract is not enablement; ordinary users need no server.observe; profile failures isolated; disabling imports preserves history | +| Body comparison/search boundary | Exact Source body/semantic comparison; explicit budget warning; history independent of staging; absent searcher unsupported, namespace not full text | +| Required existing Scope/missing-invalid target | UI blocks; malformed/missing 422, invisible/missing/unauthorized 404/403; no accepted input/operation | +| Fixed target/navigation/cross-Scope | Immutable nonempty scope_id; child operations cannot override; targets/bindings check ownership; changing destination needs new session | +| Ordinary permissions/revocation/invalid destination | No server.admin needed; confirmation rechecks and never repairs/creates target | +| Multiple sessions/quotas/concurrent commits | Charge fixed Scope from acceptance; atomic reservation/release, no quota migration/bypass | +| CLI/OpenDAL target setup | Missing existing Scope rejected before upload; resumed target fixed; conflicting --scope fails | +| Equal names, renames, equal independent content | Names/digests are not identity; explicit updates retain ID, independent creates stay distinct | +| A→B→A, old replay, Head races | R3 for returning content; old receipt does not roll back Head; 409 conflicts never overwrite | +| Parser/filter changes, renames, member reordering | Configuration does not misidentify documents; no guessed correspondence without stable IDs | +| Metadata/parts/lineage/reverse lookup | Exact versions/parts agree; no duplicated body, invented full consumption, or future relationships | +| Type/wire name | ImportedContent/importedcontent consistent across persistence, registration, API, clients, Dashboard; four standard reads work | +| Unified content across inputs | Paste, HTML/PDF, members, OpenDAL persist content; SourceRef types agree; webpage acquisition remains a Connector | +| Constant source_kind/default normalization | Omitted/explicit content share fingerprints/digests/cache semantics; others 422; no constant-field filter index | +| Cross-input updates/history | File→paste still checks Head/snapshot; each content version keeps origin/format/parser | +| New preview/existing update | No new public Artifact/Source before confirmation; old Head survives parse/failure/retry/cancel | +| Generic writes/internal-state forgery | No direct forged Artifact or arbitrary operation PUT/PATCH; reject cross-session handles | +| Private operations/ordinary Scope readers | Other readers cannot preview/confirm private input; standard reads expose accepted content only | +| Operation cleanup/governance | Exact history survives cleanup; progress creates no Revision/governance parsing state | +| Default routing/backend loss/one fallback | Pinned profile/router; no implicit OCR/external transfer; shared budgets; frozen output immutable | +| LiteParse limits/page_errors/complexity/OCR | Probes are not execution; missing pages not complete; fixed Docling complex-layout branch | +| Full/values-only XLSB | Missing formula semantics keeps planned; values-only explicit needs_review; complete-only rejects | +| OOXML template/macros, ODF/flat XML/repeats | Correct original-container relationships; no suffix hacks/macros; enforce node/expansion budgets before loading | +| Office audit/converted locations | Selected revisions/footnotes/hidden/notes retained; no invented native coordinates from converted pages | +| Service tasks/signed resources/temporary images | Versions/transfers traceable; saved text independent of expiring URLs; no permanent-image promise | +| Every claimed format/generation | Real fixtures with verifiable body/locations/omissions; detection alone fails acceptance | +| TXT/code/config and UTF-8/UTF-16/GB18030/Big5 | Strict decoding, Chinese and lines correct, no silent replacement | +| Native/scanned/mixed/multicolumn/table PDF | Page order, OCR deduplication, omissions correct; missing OCR never full success | +| Word tables/footnotes/revisions/comments | Declared view/selection respected; no silent selected-content loss | +| XLS/XLSX/XLSB, multisheet/hidden/missing caches | All sheets, no recalculation/invented values; verifiable locations | +| PPT/PPTX/templates/notes/hidden slides | Correct order/inclusion; no macro execution | +| HTML/notebook/EPUB/EML/MSG/MBOX | No external fetch/execution; correct alternative body/child boundaries | +| OCR images/multipage TIFF/pixel limits/blank pages | Enforced limits, no metadata-as-body, traceable outputs | +| Office/EPUB versus ordinary ZIP/TAR | Correct distinction; traversal/nesting/bombs bounded | +| Missing library/metadata-only result | No available declaration or empty Source | +| Long text/many parts/overflow/huge paragraph | Concatenated digest matches complete text; overflow fails whole document | +| Disconnect/crash/stale lease | Recover same operation; old generation cannot overwrite frozen output | +| Retry after OCR preview/concurrent confirms/profile upgrade | Confirm frozen bundle; identical semantics reuse, changed output preserves old evidence | +| Pre/post-save digest and differing refs/times | Receipts do not affect content digest; same target/expected Head has at most one successful advance | +| Two Principals/equal content in one Scope | No shared private operation; independent creates differ; updates check permission/Head | +| Expiry/idempotency cleanup/worker recovery | New attempt keys progress without duplicate committed materials/early checkpoint | +| Discovery/selection/partial commit/cancel/GC races | Consistent states/aggregation; committed work remains, uncommitted cannot pass cancellation/expiry | +| as_text/reparse/selection/retry/cancel | API/CLI/UI agree; invalid combinations reject; same current semantics do not duplicate Sources | +| Replicas/clients/quota/crash | Atomic cumulative reservation; release disk only on deletion; no OCR refund; durable backoff | +| A accepted partial; B/new attempt hits snapshot | needs_review/uncommitted with omissions and explicit acceptance; reuse only same current document snapshot; independent create remains new; successful same-key replay needs no new confirmation | +| OpenDAL cached/recovered partial | complete-only rejects; no checkpoint advance merely because Revision exists | +| New key with old expired/cancelled/failed/completed task | New operation, coverage-aware persistent reuse; old key retains old result | +| Transient discovery failure/retries exhausted | stage=discovery works without item_ids; publish complete manifest atomically, retain frozen IDs | +| Retry slotless failed/partially_committed | 429/503 leaves state/counters unchanged; later same-key admission holds one slot and enqueues once | +| Awaiting partial confirmation/only failed members/one failure | Waiting confirmation holds slot; inactive failure may release; do not release other active resources or extant bytes | +| Any part/record/pending write failure | Whole document rollback; recovery does not duplicate Sources | +| Failed container members/explicit partial | Accurate per-document outcomes and durable omissions, no false completeness | +| Revocation/cross-Scope/cross-site/malicious file | No unauthorized confirmation/reads/execution | +| Checkpoint/profile/filter changes | Advance only complete committed results; no silent configuration-based skips | +| SQLite/seekdb/OceanBase, shared staging/expiry GC | Consistent recovery/commit; GC cannot destroy confirming bundles | +| Chinese/English, themes, small screens, reopen | Capability lists, previews, omissions, per-item retry, full reading usable | + +# Drawbacks + +Broad support increases dependency, fixture, and security-maintenance costs. Legacy Office, complex spreadsheets, and +proprietary generations differ; maintain capability matrices and failures instead of promising arbitrary files. + +Parser workers, durable staging, leases, and segmented transactions are more complex than text capture. OCR needs +independent speed/accuracy/language budgets. Text/digests cannot reconstruct all binary layout, imagery, or interaction. + +Per-document atomic commits append many Sources, increasing transaction and downstream cost. Validate 8 MiB/128-part +ceilings across all three backends rather than raising them without bounds. Containers permit partial success, which +users must distinguish from document completion. + +History retains old and new text; Head is latest accepted content, not external real-time truth or deletion propagation. +Successful parsing does not establish complete model understanding. + +# Rationale and alternatives + +**ImportedContent Artifact + Revision.** Existing immutable versions, Head, lineage, and reads simplify +history/relationship queries versus ImportRecord, at the cost of Family registration, dedicated transactional +writes/readers, bindings, indexes. Constant content reuses evidence; origin/format/parser remain separate. Reclaimable +ledgers hold execution without FileImportJob or progress Revisions. + +**Explicit identity and fixed existing Scope.** Distinguish creation/update, use authorized stable keys for collection, +avoid filename/content merging. Sessions own target/private preview/recovery, not Scope creation, permissions, quota +migration. Upload, confirmation, and accounting share one authorization boundary. + +**Native structure separately from PDF layout.** Sheets/slides/email/archives have native object/member identities and +are not universally printed to PDF. LiteParse handles simple PDF, Docling layout/OCR, independent profiles complex +alternatives. Multiple dependencies buy explainable structure/locations/omissions; RAGFlow/LlamaIndex jobs, +deduplication, indexes, and default Readers do not own persistence here. + +**Pluggable parsers with fixed routing.** UTF-8-only APIs cannot handle Office/PDF, and one generic engine cannot +guarantee all semantics. Validated adapters are more diagnosable than implicit fallback and allow future extensions. + +**Isolated parsing, reused ContentSource.** Keep dependencies and file risk outside Core requests; normalized evidence +reuses Source/Scope/lineage/processing without a Source family per suffix. + +**Asynchronous parsing, synchronous per-document commit.** OCR/large inputs exceed normal HTTP timeouts. prepare returns +durable recovery handles; confirmation atomically commits frozen output instead of exposing partially parsed Sources. + +**Segmentation instead of truncation/manual splitting.** Natural boundaries/locations remain usable, exact +concatenation/digest preserve completeness. Atomic saving avoids half-published documents but does not force every +consumer to process the entire body in one window. + +**Extraction is not universal understanding.** Body/table/OCR/metadata coverage differs. Report omissions, never infer +missing text or count metadata as body; retain explicit partial acceptance. + +**Library support is not product support.** Generations, dependencies, options, quality, budgets, and fixtures all +determine availability; discovery advertises validated deployed adapters only. + +# Prior art + +The proposal builds on verifiable repository designs and implementations: + +- [RFC template](0000_example.md) and [process](README.md): structure and numbering. +- [ContentSource](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/builtin/sources/content.py): captured text, metadata, projections, internal roles. +- [Relational capture](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/builtin/persistence/records.py): transactional Source/pending persistence. +- [OpenDAL evaluation](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/integrations/opendal/README.md) and [ConnectorLifecycle](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/sources/connectors.py): snapshots, per-item results, checkpoint CAS. +- [Dashboard principles](../development/dashboard.md): Scope boundaries, full text, recovery, static identity. +- [Adjacent Dashboard RFC](https://yuque.antfin.com/obopensrc/knowledge_sharing/frkh86luw67rkcpt): Artifact pages/exact evidence reading; its profile/handoff-export contracts are unchanged. +- [SourceRepository](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/builtin/persistence/sources.py) and [eligibility](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/builtin/source_eligibility.py): connection-taking persistence, dirty intents, body roles. +- [Authorization profiles](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/server/authz/profiles.py) and [Server routes](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/server/app.py): Family authorization and API integration. +- [Revision pages](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/server/dashboard/revisions.py) and [merge restoration](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/builtin/artifacts/merge_restoration.py): reuse history/compare without assuming generic evidence restoration. +- [Artifact Search](1858-artifact-search.md) and [Artifact Merge](1859_artifact_merge.md): search registration and merge/restoration boundaries. +- [Database migrations](../docs/operate/database-migrations.md) and [migration RFC](1771-unified-database-migrations.md): managed bundles, three backends, recovery limitations. +- [OpenAPI](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/openapi/powercontext.yaml) and [ServerInfo](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/server/info.py): protocol/feature contracts versus runtime capabilities. + +Upstream references inform adapters, not proof of completed PowerContext support: + +- [Tika formats](https://tika.apache.org/docs/4.0.x/formats.html): detection exceeds body extraction; image metadata is not OCR. +- [Docling formats](https://docling-project.github.io/docling/usage/supported_formats/): unified document representation and layout/structure conversion. +- [MarkItDown](https://github.com/microsoft/markitdown): optional-dependency Markdown conversion, an alternative for selected formats. +- [RAGFlow study](https://yuque.antfin.com/obopensrc/knowledge_sharing/lmziuic5tzhgny8c), [Docling/MinerU study](https://yuque.antfin.com/obopensrc/knowledge_sharing/rcga2z7gr8r3vwbo), [LiteParse/LlamaIndex study](https://yuque.antfin.com/obopensrc/knowledge_sharing/yl6nfo4yugep4n6a): platform boundaries, routing, versions, evidence. +- [Pinned LiteParse](https://github.com/run-llama/liteparse/tree/python-v2.14.7): structured PDF, complexity, page ranges, failures; Office-to-PDF is not the default workbook path. +- [Docling OCR](https://docling-project.github.io/docling/concepts/OCR/), [openpyxl options](https://openpyxl.readthedocs.io/en/stable/api/openpyxl.reader.excel.html), [POI spreadsheets](https://poi.apache.org/components/spreadsheet/), [python-pptx notes](https://python-pptx.readthedocs.io/en/latest/user/notes.html): component evidence; PC still needs semantic audits and per-format acceptance. + +# Unresolved questions + +1. Default parser families/candidate boundaries are fixed in Section 1. Freeze release dependency versions, PDF thresholds, and performance budgets after fixture validation. XLSB formulas, legacy Word revision views, and long-tail gaps remain planned until implemented; partial profiles cannot imply full delivery. +2. Validate 50 MiB input, 8 MiB output, 500 pages, 128 parts on SQLite/embedded seekdb/OceanBase and actual deployment resources, including transaction/downstream costs. +3. Validate initial OCR language packs and proprietary generations with real materials. Apple/WPS availability requires body fixtures, not detection alone. +4. Permanent binaries/downloads need separate quotas, retention, exact-evidence, and deletion authorization; this proposal retains parser staging only. +5. If downstream consumers must use only Sources from document Head, separately define version selection, old pending work, and historical references. Head does not change current Source consumption rules automatically. + +# Future possibilities + +Extend registered parsers/capabilities to more languages, domain formats, and proprietary documents. Define +transcription, chart reasoning, formula semantics, and handwriting separately; their cost/accuracy cannot hide inside +ordinary parsing. + +Yuque, webpages, and knowledge platforms use separate Connectors with explicit external identities, versions, +authorization, and exact reads. Databases/very large datasets need selective acquisition protocols rather than unlimited +document expansion. + +Future work may include permanent binary archives, original-text search, and exact-location navigation. Document-level +generation progress must use real per-Family Source coverage/output evidence. Preserve Scope boundaries, snapshot +identity, omissions, and durable receipts in every extension. diff --git a/docs/zh/rfcs/0000-text-material-import.md b/docs/zh/rfcs/0000-text-material-import.md new file mode 100644 index 0000000000..64b7f036b5 --- /dev/null +++ b/docs/zh/rfcs/0000-text-material-import.md @@ -0,0 +1,933 @@ +--- +title: 文本资料导入 +--- + +- Proposal Name: `text_material_import` +- Start Date: 2026-09-18 +- Status: Proposed +- RFC PR: [oceanbase/powercontext#0000](https://github.com/oceanbase/powercontext/pull/0000) +- Design Baseline: [PowerContext master / cab054db（2026-10-19）](https://github.com/oceanbase/powercontext/tree/cab054dbf6d86e2311b079388e5158178257def2) +- Related RFCs: [Source Definition 与 Observation](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1400_source_definition_and_observation_model.md)、[Source 与 Artifact REST API](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1437_source_artifact_rest_api.md)、[Source 访问与生成准入](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1458_artifact_generation_source_access.md)、[Scope 与 Source 发现](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1502_scope_source_discovery.md) + +# Summary + +为 PowerContext 提供覆盖常见文本、办公文档、网页文件、表格、邮件、电子书、代码、图片与归档包的资料导入能力。用户必须选择已有且具有 scope.read + scope.contribute 权限的 Scope 后才能导入；粘贴文本、上传文件，或通过独立 OpenDAL worker 持续采集本地目录和对象存储。输入经过格式识别、编码处理、文档解析、可选 OCR 和有界分段，保存为不可变的 `content` Source，并保存为 `ImportedContent` Artifact（Family 标识 `importedcontent`），返回精确 Revision 回执。 + +采用“广泛格式支持、明确能力分级”的产品契约。轻量文本可直接保存；复杂文件通过隔离的解析 worker 处理；扫描件及图片需要显式启用 OCR。标准文档能力包覆盖 PDF、现代与传统 Office、OpenDocument、HTML、邮件和电子书等常见格式。部署必须公开实际可用的格式、解析器与限制，不能把文件类型识别成功或仅提取到元数据当作正文导入成功。 + +一份逻辑资料对应一个 ImportedContent Artifact，source_kind 固定为 content，统一使用现有文本证据类型，每次接受的新快照对应一个不可变 Revision，正文保存在其精确引用的 ContentSource 中。导入成功包含导入资料制品及证据保存，不承诺 Memory 等下游制品已经生成、原始二进制已归档或原文已建立检索索引。Head 表示 PowerContext 最近接受的版本，不代表外部文件的实时最新版；持续采集不传播删除。本文是拟新增契约，不代表功能已经发布。 + +已保存资料复用 Artifact 表、标准读取、Revision 与 lineage。导入写入由 ImportedContent Family 的专用操作负责；运行进度和私有预览只进入可清理的内部操作账本，不作为制品版本，也不新增 FileImportJob 业务资源。 + +# Motivation + +项目资料不只存在于 TXT 和 Markdown，也广泛存在于 Word、PDF、Excel、PowerPoint、网页导出、邮件与扫描件。只扩展扩展名白名单无法完成导入：二进制格式需要解析，扫描页需要 OCR，表格需要保留行列关系，长文需要分段，解析不完整时还需要用户可见的错误与恢复方式。 + +仓库已有 ContentSource、Scope 权限、Source journal、文本投影、Source 集合与精确读取，以及 OpenDAL evaluation Connector 的 durable receipt 和 checkpoint CAS。这些能力构成保存与恢复基础，但不等于已经实现通用文件解析。 + +现有 OpenDAL evaluation 使用 `text-file-snapshot` 类型，worker 会注册 Definition manifest。它的 UTF-8 文件采集、权限要求和公开读取范围不直接构成本提案的文件支持合同。本提案复用内置 `content` Source，新增独立的资料导入应用服务与可选解析部署，不改变既有 Source Create/Capture 接口。 + +目标是让用户知道：当前部署能读取哪些格式、提取了哪些内容、遗漏了什么、材料保存到哪个 Scope，以及重复提交和失败时如何继续。 + +# Guide-level explanation + +## 支持范围与能力分级 + +以下是本 RFC 的目标支持矩阵。A 为基础能力，B 为标准文档能力包，C 为可选 OCR 或扩展适配器。发布时每一项都需要实际样本验收；未安装依赖显示“未启用”，未完成适配显示“尚不支持”，不得统一标为支持。 + +| 类别 | 文件类型与常见扩展名 | 级别与提取内容 | +| --- | --- | --- | +| 纯文本与日志 | `.txt`、`.text`、`.log`；显式选择的无扩展名文本 | A；正文与行号 | +| Markdown 与轻量标记 | `.md`、`.markdown`、`.mdown`、`.rst`、`.adoc`、`.asciidoc`、`.org`、`.tex` | A；保留标记源码、段落与行号，不运行排版程序 | +| 数据与配置 | `.csv`、`.tsv`、`.json`、`.jsonl`、`.ndjson`、`.yaml`、`.yml`、`.toml`、`.ini`、`.cfg`、`.conf`、`.xml` | A；原文本及可验证的记录、行列定位，不执行对象构造或外部实体 | +| 源代码与构建文件 | `.py`、`.js`、`.ts`、`.jsx`、`.tsx`、`.java`、`.c`、`.h`、`.cpp`、`.hpp`、`.go`、`.rs`、`.sh`、`.bash`、`.zsh`、`.sql`、`.css`、`.scss`、`.vue`、`.svelte`、`.rb`、`.php`、`.swift`、`.kt`、`.kts`、`.scala`、`.r`、`.R`、`.lua`、`.pl`、`.ps1`、`.bat`、`.cmd`、`.proto`、`.graphql`、`.gql`、`.diff`、`.patch`；Dockerfile、Makefile、CMakeLists.txt | A；按文本保留，不运行、编译或安装依赖 | +| 字幕与记录 | `.srt`、`.vtt`、`.ass`、`.ssa`、`.ics`、`.vcf` | A；保留时间标记或原始字段，不调日历或通讯录 API | +| 网页与离线网页 | `.html`、`.htm`、`.xhtml`、`.mhtml`、`.mht` | B；静态 DOM 正文、标题、列表、表格与链接文字，不执行脚本或读取外链；不承诺浏览器计算后的可见性 | +| PDF | `.pdf` | B；文本层、页码及可提取表格。扫描或混合页使用 C 级 OCR | +| Word 与模板 | `.docx`、`.doc`、`.dot`、`.dotx`、`.docm`、`.dotm` | B；段落、标题、表格、脚注；宏不执行 | +| Excel 与模板 | `.xlsx`、`.xls`、`.xlsb`、`.xlt`、`.xltx`、`.xlsm`、`.xltm` | B 目标；工作表、单元格值、公式与已有缓存结果，保留范围位置；XLSB 的完整/仅值 profile 按第 1.4 节分别准入 | +| PowerPoint 与模板 | `.pptx`、`.ppt`、`.pps`、`.pot`、`.ppsx`、`.potx`、`.pptm`、`.ppsm`、`.potm` | B；幻灯片文字、表格和演讲者备注，保留页序 | +| 开放办公与富文本 | `.odt`、`.ott`、`.ods`、`.ots`、`.odp`、`.otp`、`.fodt`、`.fods`、`.fodp`、`.rtf` | B；文档、工作表或演示内容，对应定位规则一致 | +| 电子书 | `.epub`、`.fb2` | B；目录、章节与正文，不处理 DRM 加密内容 | +| 邮件与邮箱导出 | `.eml`、`.msg`、`.mbox` | B；主题、发件信息、正文；邮箱按邮件拆分，附件按显式选择处理 | +| Notebook | `.ipynb` | B；Markdown、代码单元和已保存文本输出，不执行代码或重新计算 | +| XML 文档与订阅 | `.rss`、`.atom`、`.opml`、`.xlf`、`.xliff`、`.tmx`、`.svg` | A 可保留源码；B 提供语义文本提取；SVG 仅提取文字节点，不执行或承诺理解图形 | +| 图片与扫描件 | `.png`、`.jpg`、`.jpeg`、`.tif`、`.tiff`、`.bmp`、`.webp`；静态 `.gif` | C；OCR 文字、图像/页序和可提供的定位，动画 GIF 不静默只取首帧 | +| 批量归档 | `.zip`、`.tar`、`.tar.gz`、`.tgz`、`.gz`、`.bz2`、`.xz` | B；安全展开，逐成员识别与导入，压缩包不是一种正文格式 | +| 扩展归档 | `.7z`、`.rar`、`.tar.bz2`、`.tbz2`、`.tar.xz`、`.txz` | C；需安装并验收对应解包适配器 | +| 专有或长尾文档 | `.pages`、`.numbers`、`.key`、`.wps`、`.et`、`.dps`、`.wpd`、`.hwp`、`.chm`、`.one`、`.pst`、`.mobi`、`.azw`、`.azw3`、`.heic`、`.heif` | C 扩展目标；按格式代际、正文提取能力、解码器及许可证分别准入，不承诺仅安装通用解析器就全部支持 | + +A、B 是正式文件导入的交付范围；C 是同一扩展机制下按部署启用的能力。轻量安装可以只启用 A；标准文档安装必须打包并固定 B 所需依赖。支持 WPS 保存的 DOCX/XLSX/PPTX 不等于已经支持 WPS 专有的 WPS/ET/DPS。识别新一代 iWork 文件也不等于能提取其正文。 + +纯文本兜底不限于表中扩展名：用户可以显式选择“按纯文本读取”，通过编码、控制字符与大小校验后保存原文。目录采集默认仅扫描允许的类型，不能把整个目录内一切可解码文件自动导入；默认排除密钥、凭据文件、`.env`、版本控制目录、依赖目录与构建产物，可由用户明确选择其他规则。 + +首期不做音视频转写、任意数据库转储解析、可执行文件反编译、图形/图表推理，也不破解加密或 DRM。语雀、网页和 Git 链接抓取属于独立 Connector；导入本地 HTML 或语雀导出的 Markdown，不等于支持直接抓取链接。 + +## 用户流程 + +用户先选择已有且可贡献的 Scope，再进入文本或文件导入。全局入口必须明确选择目标；从某个 Scope 内进入时可预选当前 Scope，但仍须显示目标并校验权限。未选择有效目标时不能创建 ImportSession、提交粘贴文本或上传文件；没有可用 Scope 时提示先在现有 Scope 管理流程中准备目标。导入页面不提供创建 Scope 的表单或自动创建行为。 + +选定 Scope 后，页面展示当前部署支持的格式、实际限制和调用方权限。创建 ImportSession 时固定非空 scope_id 与创建者；同一会话的全部输入都属于该 Scope。改变目标必须创建新的会话，不能迁移正在解析或待确认的输入。 + +纯文本校验后可直接确认保存；文件先上传并解析，展示正文预览、页/表/幻灯片数量、分段数、OCR 使用情况及遗漏，再确认保存。确认时重新校验目标 Scope 及写入权限，在同一事务内保存单份资料的 Source、ImportedContent Revision、Head、lineage 和回执。失败、取消和只预览均不产生已提交资料。 + +```mermaid +flowchart TD + A["选择已有 Scope"] --> P["校验 scope.read + scope.contribute；创建固定 scope_id 的 ImportSession"] + P --> B{"输入方式"} + B -->|粘贴文本| C["校验与规范化"] + B -->|上传文件| D["Family prepare 操作：耐久暂存与隔离解析"] + D --> E["预览正文、分段、OCR 与遗漏"] + E -->|无可用正文| X["失败；不保存资料"] + E -->|完整确认或显式接受 partial| F["确认冻结 bundle_digest"] + C --> G["重新鉴权；在会话固定 Scope 内单份资料原子保存"] + F --> G + G --> I["ContentSource 分段 + ImportedContent Revision + lineage + 回执"] + I --> J["后续文件使用同一 Scope；下游处理按既有配置运行"] +``` + +普通导入只要求目标 Scope 的 `scope.read` + `scope.contribute`，不要求 `server.admin`。会话不能替代当前授权；上传后被撤权、目标不存在或不可写时，确认不得继续保存或自动改换目标。管理代操作遵循第 10 节。 + +第一次手动上传或粘贴默认“新建文档”；系统分配新的 artifact_id。更新现有文档必须从该文档进入“更新文件/文本”，明确指定 artifact_id 和预期 Head Revision。文件名相同、正文相似或字节摘要相同都不能自动合并文档；只可提示可能重复并让用户选择。重命名上传文件不影响明确指定的文档身份。 + +多份文件分别呈现待上传、解析中、待确认、已保存、已存在、需处理、版本冲突和结果待核实。文件名、已选择的目标 Scope、新建/更新文档意图始终可见。不用文件名推断既有文档身份；并发保存仍按同一 Scope 下的幂等、配额及 Head 并发规则处理。 + +例如，Word 提取成功、PDF 中含扫描页但未启用 OCR、Excel 的公式缺少缓存值时：Word 可以确认保存；PDF 必须说明受影响页，启用 OCR 后新尝试或显式接受部分文本；Excel 显示公式和“未保存计算结果”，不编造数值。不能把三份都显示为完整提取。 + +标准解析按照已声明范围保存，不能承诺完全还原二进制。界面使用“查看提取文本”；纯文本显示“查看原文”。图片、布局、图表和嵌入对象的处理范围与已知遗漏必须可见。解析完成与预览可见都不等于已经保存 Source 或 ImportedContent Revision。 + +## 格式特有行为 + +- **编码**：支持 UTF-8、带 BOM 的 UTF-16/UTF-32，以及显式指定的 GB18030、GBK、Big5、Windows-1252、ISO-8859-1。自动检测仅提供建议，除 BOM 和严格有效 UTF-8 外不静默采用猜测编码；先让用户确认，禁止用替换字符掩盖解码错误。 +- **表格**：保留工作表名与单元格坐标，不只读取首张表。CSV/TSV 保留原始文本，可另外提供经校验的行列定位。公式不重算，不更新外链；默认纳入隐藏表、行、列并标记隐藏状态，用户可在预览前显式排除，此选择参与快照身份。 +- **演示文稿**：按幻灯片顺序提取，备注单独标记。默认纳入隐藏幻灯片并标记；不把图片中的文字当作已读取，嵌入图像的文字需 OCR。 +- **Word/开放文档**：标题、正文、表格、脚注按声明规则提取；修订采用已接受视图，批注默认不导入并报告数量，用户可选纳入且标明批注作者。解析器不能可靠提供这些视图时应报能力不足。 +- **HTML/Notebook**：HTML 不运行 JavaScript、不抓取 CSS/图片/链接；Notebook 不执行单元。Notebook 默认包含已保存的文本输出，隐藏元数据、富媒体与 widget 不视为已导入正文。 +- **PDF/OCR**：按页选择文本层或 OCR，避免同一区域重复叠加。空白页、图片页、解析失败页分别报告；低质量 OCR 输出是机器识别结果，不能称为精确转录。布局/页数信息不可得时报告未知，不能伪造完整性。 +- **邮件与归档**：邮件正文中的引用历史保留，纯文本与 HTML 二选一并记录规则，避免重复。附件默认列出但不导入；用户选择后逐附件处理。归档默认展开第一层，嵌套包需显式开启且受深度限制；失败或未选成员在清单中保留。 + +## 保存与后续处理 + +一份逻辑文档对应一个 `ImportedContent` Artifact，接受的快照对应 Revision；长文按结构分成多个不可变 ContentSource。单份文档的全部分段、Revision、Head、lineage、待处理登记和回执在同一事务提交。事务仅写入会话固定的已有 Scope，不创建 Scope 或初始化权限。跨文件、归档成员和邮件间允许部分成功。 + +成功表示导入资料制品和正文证据已可靠保存，不需要 LLM 或 Candidate 审批。Memory 等既有处理器是否消费这些 Source、处理多少、产生什么候选或制品，继续由原有配置和领域契约决定。importedcontent Family 不默认注册自动生成处理器，不能因称为 Artifact 就触发递归生成。 + +历史 Revision 与其 Source 保留，不因 Head 变化而覆盖。资料详情可以显示版本历史，以及按精确 SourceRef 或真实 ArtifactRef 反查的下游引用；不在导入时预填未来“关联制品列表”。一次 consumer window 不保证处理全部分段，不虚构全文理解完成状态。 + +原文件仅用于解析暂存，默认不是长期文件档案。保存后保留提取文本、文件摘要、解析版本、定位及遗漏；下载原始二进制和长期存档不属于本期承诺。 + +## 持续采集与 Dashboard + +OpenDAL worker 在存储附近运行，持有读取凭据并把文件字节提交给导入服务；Server 不通过用户提供的路径或 URL 读取任意文件。文本与复杂文件使用同一份能力矩阵。周期执行仍由外部调度器负责。 + +Dashboard 保持既有 Tabler、Jinja2、HTMX、中英文及明暗主题。写入口由 `POWERCONTEXT_SERVER_DASHBOARD_TEXT_IMPORT_ENABLED` 显式开启，默认关闭，继续满足现有静态 Bearer 和 enforced access 前置条件。文件解析另要求解析服务可用，关闭该服务不影响已保存资料阅读。 + +页面结构为“选择已有 Scope → 资料列表 → 选择文本/文件 → 解析预览与问题清单 → 确认保存 → 精确阅读”。切换 Scope 不改变正在运行任务的归属，也不自动提交未完成输入。关闭浏览器后可从任务记录恢复已接受的文件解析；未上传完的本地文件需重新选择。 + +# Reference-level explanation + +## 1. 架构 + +入口、解析和保存之间的责任边界见“用户流程”图。文件通过 ImportedContent Family 的 prepare 操作产生冻结的 ImportedContentBundle，执行过程由内部操作账本和隔离 worker 承担；用户确认或 worker 按显式策略接受后，ContentImportApplication 才执行单文档原子提交。制品处理仍沿用已有配置与进度机制。 + +文件获取、解析、文本保存与制品生成是四个边界。Core 不导入任意用户指定 parser，不运行 Office 宏，不执行代码。正式部署提供独立 parser worker；轻量文本路径不要求文档解析依赖。 + +### 1.1 解析器职责与固定选型 + +采用“原生结构优先、PDF 按布局分流、OCR 显式开启”的组合方案。PowerContext 实现格式识别、结构审计、ImportedContentBundle 归一、证据分段、任务与保存；复用解析库完成格式解码、布局和 OCR，不自研通用 OCR 模型。下述 profile 名是拟新增的产品契约，不是上游已经提供的接口或已完成实现。 + +| 组件 | 本 RFC 中的固定职责 | 不承担的职责 | +| --- | --- | --- | +| 原生文本/容器适配器 | 严格解码，CSV/JSON/XML 等定位，OOXML/ODF 结构审计，邮件/归档发现 | 不执行内容，不用 LLM 重写正文 | +| LiteParse | 简单文本 PDF 的轻量提取、页几何与复杂度信号；标准路径关闭其内置 OCR | 不统一把 Excel/Word/PPT 转 PDF，不用其 Markdown 清理结果代替全部证据 | +| Docling | 复杂布局 PDF、现代 Word/开放文档的结构提取；启用 OCR 时使用指定本地引擎 | 不负责资料身份、提交或索引，不直接使用检索 chunk 作为 Source | +| openpyxl / Apache POI | 工作簿的原生单元格、公式与已有值；POI 同时用于传统 Office 结构适配 | 不计算公式，不刷新数据连接，不将工作簿打印成 PDF | +| python-pptx + OOXML 审计 | 现代演示的文字、表格、备注与隐藏状态 | 不执行宏，不将演示动画和图表理解混入正文 | +| Apache Tika | 传统文档、RTF、MSG 和已准入长尾格式的受控格式解析 | 不作为无限制自动探测/递归/云 OCR 兜底，不把 metadata 视为正文 | +| 本地 RapidOCR | 通过 Docling 处理扫描 PDF/图像，首个目标语言集为简体中文与英文 | 不自动外发；繁体、其他语言与手写须独立验收 | +| MinerU / DeepDoc / LlamaParse | C 级、独立启用的复杂文档替代 profile；LlamaParse 属外部或企业自托管服务 | 不加入标准 profile 的隐式 fallback 链 | +| LlamaIndex | 后续可选 Reader/检索变换参考，索引阶段按需使用 | 不接管导入事务、版本和 Source；不使用默认 Reader 的跳过错误行为 | + +组件取舍基于 [RAGFlow 调研](https://yuque.antfin.com/obopensrc/knowledge_sharing/lmziuic5tzhgny8c)、[Docling/MinerU 调研](https://yuque.antfin.com/obopensrc/knowledge_sharing/rcga2z7gr8r3vwbo)与[LiteParse/LlamaIndex 调研](https://yuque.antfin.com/obopensrc/knowledge_sharing/yl6nfo4yugep4n6a)。这是拟实施的选型，不是解析质量排名。首轮适配基线采用已调研的 LiteParse Python 2.14.7、Docling 2.129.0；MinerU 4.0.5 仅作可选适配验证基线。发布镜像还必须锁定其传递依赖、模型和转换工具，不能使用 latest。其他组件在实施时固定通过样本与安全检查的具体版本及镜像摘要。 + +### 1.2 A 级：原生文本、标记与结构定位 + +以下各组覆盖支持矩阵中该类的全部扩展名；同类不同后缀不产生不同正文规则。A 路径保存解码后的原始文本，结构解析只增加可验证定位，默认不改写正文。 + +| 输入 | 默认 profile 与具体方式 | 提取/定位与失败规则 | +| --- | --- | --- | +| TXT、TEXT、LOG、显式无扩展名文本 | `text-native-v1`：Python 严格 codecs 解码 | 保留换行、缩进、字符和行号；编码不确定要求确认 | +| Markdown、RST、AsciiDoc、Org、TeX | `markup-source-v1`：同一文本解码器；Markdown 可用 markdown-it-py 辅助识别块 | 正文保存标记源码；不渲染 TeX，不执行 include、插件或远程资源 | +| 全部代码、构建文件、diff/patch | `code-source-v1`：原文与行索引 | 不编译，不运行 formatter，不因语言识别失败丢失正文 | +| CSV、TSV | `delimited-text-v1`：原文 + Python csv 读取器 | 固定分隔符、引号和编码；记录逻辑行、字段及原文范围；允许跨物理行字段，分隔符歧义要求确认 | +| JSON、JSONL、NDJSON | `json-source-v1`：有界 JSON 语法扫描、逐记录解析 | 保留原文及重复键顺序，定位记录与字段出现序号；不将 dict 序列化结果替换原文 | +| YAML/YML、TOML、INI/CFG/CONF | `config-source-v1`：保留源码；PyYAML SafeLoader、tomllib、禁用插值的 ConfigParser 仅作可选结构校验 | YAML 限制别名和深度，不构造任意对象；键路径无法唯一定位时只提供原文行号 | +| XML | `xml-source-v1`：禁用 DTD/实体/网络的 XML 读取器 | 保留源码；节点路径、命名空间和可提供的行号，不执行 XInclude/XSLT | +| SRT、VTT、ASS、SSA | `subtitle-source-v1`：文本 + 有界字幕时间标记扫描 | 保留全部原字段，增加时间范围；不重新排序或翻译 | +| ICS、VCF | `record-source-v1`：保留文本并识别折行和记录边界 | 原文行与逻辑记录区分，不调用日历/通讯录 API | +| RSS、Atom、OPML、XLF/XLIFF、TMX、SVG 的源码模式 | `xml-source-v1` | 保存源码；语义提取另走 B profile，不能静默切换 | + +辅助结构校验失败时，返回原文可用、结构定位不可用的明确警告；如果用户选择的是要求结构完整的 profile，则失败或 needs_review，不能改成源码模式冒充成功。A 路径的原文字节/字符保真不以结构解析成功为前提，但仍必须通过编码、控制字符和大小校验。 + +### 1.3 PDF：文字、布局与 OCR 路由 + +默认 `pdf-standard-v1` 是固定版本的复合 profile：先用 LiteParse 做有界页面探测；全文文字层可用且无复杂布局信号时采用 LiteParse 的结构化 text_items；出现多栏、疑似复杂表格、顺序异常或探测不确定时，采用 Docling 标准布局/表格路径。两者均默认关闭 OCR 和生成式增强，路由判断及阈值作为 profile 的版本化配置，通过样本集固定。 + +| PDF 情况 | 解析方式 | 完整性规则 | +| --- | --- | --- | +| 简单文字层、正常阅读顺序 | LiteParse JSON/结构化输出 → PC 有序块归一 | 保留物理页序、显示页标签、页尺寸及可用 bbox；不依赖默认去页眉页脚的 Markdown | +| 多栏、复杂表格、目录/脚注布局 | Docling 标准 pipeline，读取 DoclingDocument 和 provenance | 结构节点与阅读顺序归一；表格保留行列/合并关系和原始页定位，不能只保存渲染 Markdown | +| 扫描、混合、乱码文字层，`ocr_mode=off` | 仅提取可用原生文字，明确标出需要 OCR 的页/区域 | 有正文则 needs_review；只有元数据或无正文则失败，返回 ocr_required/no_extractable_text | +| 扫描、混合，`ocr_mode=auto` | Docling + 固定本地 RapidOCR，对需要的页/区域识别 | 合并时按区域去重；记录实际 OCR、引擎、语言和遗漏，不能用 needs_ocr 代替执行记录 | +| `ocr_mode=force` | Docling + 同一 OCR 引擎，按明确范围强制识别 | 以配置的识别结果为准，不与原生文本双份叠加；强制策略和可能质量变化在预览中显示 | +| 公式、手写、复杂图表仍不可可靠提取 | 原路径保留可用文字和位置，报告不能识别的对象 | 不能根据图像猜正文；可由用户新建尝试选择 C 级 MinerU/DeepDoc/LlamaParse profile | + +`needs_ocr` 与 layout complexity 分开使用；没有 OCR 需求不表示布局简单。源页数、请求范围、返回页、空白页、失败页分别统计。不能将引擎 max_pages 截断解释为完整文档；用户未请求部分范围而触发硬上限时，沿用本 RFC 的超限拒绝规则。 + +首期按整份 PDF 确定一个正文解析后端；该后端内部可以按页使用原生文字或 OCR。不同 PDF 引擎不逐页拼接结果，避免跨页表格、阅读顺序与坐标不一致。复杂度探测结果可暂存复用，但不是跨引擎通用的中间结果。跨页表格无法可靠合并时保留分表及页范围，并报告关系未确定。 + +PDF 默认纳入可提取的页眉页脚并标为相应块；水印、批注、表单和图片文字的纳入/排除必须在 profile 中声明。未开启某类内容不能把数量记为零；已检测存在却未处理的内容列入 omissions。文本位置是出处描述，首期不承诺原始 PDF 永久在线预览。 + +### 1.4 B 级：Office 与开放办公格式 + +原生文档结构优先于打印布局；现代 Office 不默认转 PDF。模板和宏变体通过 MIME、容器内容类型和关系识别；适配器读取原始容器，不能仅改后缀绕过库的限制。 + +| 输入 | 默认 profile / 后端 | 必须保留与准入条件 | +| --- | --- | --- | +| DOCX、DOTX、DOCM、DOTM | `word-ooxml-v1`：Docling DOCX 结构转换 + PC OOXML 审计/补充读取 | 标题、正文、表格、脚注/尾注、修订视图、选中批注；docm/dotm 宏不加载执行；后端不接受的变体由受控 OOXML 适配器读取，未验收则 planned | +| DOC、DOT | `word-legacy-v1`：受控 Tika OfficeParser / POI HWPF，适配器补齐结构审计 | 不以一段纯文本证明脚注、修订和批注完整；无法可靠提供用户选择的视图时 capability_insufficient，禁止静默转 PDF | +| XLSX、XLTX、XLSM、XLTM | `sheet-ooxml-v1`：openpyxl + OOXML 原始单元格/关系读取 | Sheet、A1 坐标、数据类型、合并范围、隐藏状态、公式及已有缓存值；宏只作为被排除对象计数 | +| XLS、XLT | `sheet-biff-v1`：POI HSSF 原生读取 | 读取公式 token 的可用表达及存储的缓存值，不调用 FormulaEvaluator；老 BIFF 代际逐项验收，不满足时拒绝该代际 | +| XLSB | `sheet-xlsb-v1`：POI XSSFB 事件读取 + 明确的结构适配 | 默认“公式与值”契约必须通过验收才 available；只读缓存值的能力另列 profile，见下文，不冒充完整工作簿 | +| PPTX、PPSX、POTX、PPTM、PPSM、POTM | `slides-ooxml-v1`：python-pptx + 原始 OOXML 关系/节点审计 | 幻灯片顺序、文字框/组内文字、表格、备注、隐藏页；修复模板入口需适配器明确处理，不修改原文件 | +| PPT、PPS、POT | `slides-legacy-v1`：POI HSLF，复用 Java worker | 原生幻灯片与备注；不能仅用 Tika 合并正文后推算页界；未知 OLE/图形对象列入遗漏 | +| ODT、OTT、FODT | `odf-text-v1`：Docling + PC 安全 ODF XML 结构读取 | 对打包/Flat XML、标题、表格、注释和修订视图逐项校验；不足时不降为“所有 XML 文本” | +| ODS、OTS、FODS | `odf-sheet-v1`：PC 安全 ODF XML 原生适配器 | 表/行列、重复计数、合并与覆盖单元格、隐藏、公式和已存值；展开重复行列前计算预算 | +| ODP、OTP、FODP | `odf-slides-v1`：PC 安全 ODF XML 原生适配器 | draw:page 顺序、文字框、表格、备注及显示状态；无稳定对象定位时明确降低定位能力 | +| RTF | `rtf-native-v1`：Tika RTFParser 的结构化输出 + PC 归一 | 正文/段落及可提供表格、脚注；嵌入对象只列清单，不执行；不支持的控制组报告 | + +**Word 的补充读取是适配器工作，不是 Docling/python-docx 已完整实现的保证。** OOXML 按关系图审计段落、表格、脚注/尾注、批注和修订节点；“已接受视图”包含插入/moveTo、排除删除/moveFrom，对属性修订和不支持的复杂变更不能猜测。无法可靠生成该视图时拒绝该 profile；用户若选其他视图需新尝试和明确能力支持。补充文本按原锚点入序，不能在正文末尾随意追加造成重复和错序。 + +**工作簿不重算。** XLSX 读取公式视图与 data_only 缓存视图时使用同一输入摘要及 Sheet/坐标对齐；必要时直接读取 ``、``，区分真实空单元格、空字符串缓存和未保存缓存。公式、缓存值、类型与格式化显示是不同字段；外部引用只保留字符串，不访问资源。损坏的 shared/array formula 或动态数据结构报告受影响范围。 + +**XLSB 是已知准入风险。** POI 的公开 XSSFB 文档明确存在公式表达提取限制，不能因支持读取 XLSB 就承诺公式保真。默认 `sheet-xlsb-v1` 在完整契约未实现前保持 planned；可单独发布显式选择的 `sheet-xlsb-values-v1`，仅在 Sheet/单元格/隐藏范围可核验时导入已有值，始终 needs_review，明确“公式表达未提取”，由用户 accept_partial。范围无法核验或无正文时仍失败。该有限 profile 不算完成 B 级完整 XLSB 交付，也不能被自动采集的 complete-only 接受。[POI XLSB 限制](https://poi.apache.org/apidocs/5.0/org/apache/poi/xssf/extractor/XSSFBEventBasedExcelExtractor.html) + +**LibreOffice 不在标准工作簿路径中。** 格式转换可能改变公式缓存、字体、分页和对象语义；因此不通过 LibreOffice 把 XLS/XLSB/ODS 转 XLSX/PDF 来规避“不重算”契约。文档类如确需转换，仅在单独发布、经过无外链/宏/计算副作用验收的扩展 profile 中启用；首期不依赖该方式完成表格结构承诺。 + +### 1.5 B 级：网页、电子书、邮件、Notebook 与归档 + +| 输入 | 默认 profile / 具体方式 | 正文及成员规则 | +| --- | --- | --- | +| HTML、HTM、XHTML | `html-dom-v1`：html5lib 解析 + PC DOM 文本/结构遍历；XHTML 安全 XML | 标题、段落、列表、表格和链接文字；排除 script/style/template 及明确隐藏节点，不执行 CSS/JS；不能声称复现浏览器计算后的可见性 | +| MHTML、MHT | `mhtml-native-v1`：Python email MIME 解析 → 选定 root HTML → html-dom | 根据根部件信息与 content-location 识别正文；无法确定根文档时报告错误；内嵌资源只列清单，禁止 URL 回源 | +| EPUB | `epub-spine-v1`：安全 ZIP → container/OPF → spine 顺序 → html-dom | EPUB 阅读顺序、章节、导航与文本；不按 ZIP 文件名排序；脚本、远程资源、DRM 不处理 | +| FB2 | `fb2-native-v1`：安全 XML → body/section/notes | 正文、章节、脚注引用；二进制图片不作为正文 | +| EML | `mail-rfc822-v1`：Python email BytesParser | 解码 header/MIME；multipart/alternative 优先可用 text/plain，否则 html-dom；CID 图片/附件列清单，不拼接双份正文 | +| MBOX | `mailbox-mbox-v1`：有界 mbox 迭代 + mail-rfc822 | 先发现邮件成员再按选择解析；保留消息序号、可用 Message-ID 和容器链，重复 Message-ID 不当作唯一键 | +| MSG | `mail-msg-v1`：Tika/POI HSMF 受控读取 | 提取主题、发件信息与正文；HTML 走 html-dom，RTF 正文走 rtf-native；完整附件发现不得由自动递归文本合并代替 | +| IPYNB | `notebook-native-v1`：有界 JSON + nbformat 校验 | 原序 Markdown/code/raw cell；默认已保存 text/plain、stdout/stderr 和文字错误输出；HTML 输出显式净化/纳入，富媒体/widget 列为排除 | +| RSS、Atom、OPML | `feed-xml-v1`：安全 XML 的格式专用映射 | feed/channel/entry/item/outline 的标题、正文与链接文字，保留节点路径，不抓正文链接 | +| XLF、XLIFF、TMX | `translation-xml-v1`：按具体 schema 读取 source/target/segment | 单元 ID、语言与原/译文分别标记，不能把两种语言拼成重复正文；不同 XLIFF 代际独立验收 | +| SVG 语义模式 | `svg-text-v1`：安全 XML 提取 text/tspan/title/desc | 不光栅化运行脚本；path 轮廓文字和图形含义不承诺，图像资源不回源 | +| ZIP、TAR、TAR.GZ/TGZ | `archive-native-v1`：Python zipfile/tarfile 有界流式读取 | 先列完整成员，选择后递归分发；不调用 extractall 写任意路径；拒绝链接/设备/重复路径 | +| GZ、BZ2、XZ | `compressed-stream-v1`：对应标准库解压器 | 单流解压后重新识别，是 TAR 则进入归档发现，否则是一份子文档；不能将每个压缩流都称为多文件归档 | + +Python MIME/mailbox 提供基础读取，PowerContext 仍负责资源上限、结构缺陷、选择与稳定来源规则；库返回对象不代表容器清单完整。[email parser](https://docs.python.org/3/library/email.parser.html)、[mailbox](https://docs.python.org/3/library/mailbox.html) + +Tika 只调用注册的格式解析器，关闭自动 OCR、网络和不受控 embedded recursion；附件/嵌入资源通过受控回调参与发现，不能被解析器偷偷加入正文。对于已检测但无法列举完整附件的输入，discovery 失败，不发布可选择的“完整”成员清单。普通文档中的未选嵌入对象则按已声明的排除规则报告。 + +### 1.6 C 级：OCR、替代引擎与长尾格式 + +| 输入或需求 | 指定扩展方式 | 启用条件 / 限制 | +| --- | --- | --- | +| PNG/JPEG/BMP/WebP、静态 GIF | `image-ocr-v1`：Pillow 解码、方向归一 → Docling + 本地 RapidOCR | OCR 显式开启；记录原始与旋转后坐标映射；不识别图表语义 | +| TIFF/TIF 多页 | `image-multipage-ocr-v1`：有界枚举全部图像页 → 同一 OCR profile | 每页计数、像素和 OCR 预算；失败页明确报告；不能只读首帧 | +| 动画 GIF / 多帧变体 | 默认拒绝该变体 | 仅在独立多帧 profile 完成帧语义与重复文字验收后开放 | +| HEIC/HEIF | `heif-ocr-v1`：pillow-heif/libheif 解码 → image OCR | 区分主图、缩略图、辅助图和序列，固定选择范围；只有 metadata 不可用 | +| 复杂 PDF/扫描件替代 | `pdf-mineru-v1` / `pdf-deepdoc-v1` | 分别直接适配 MinerU 结构协议或隔离封装 DeepDoc；固定 tier/模型；不通过整套 RAGFlow 间接转接 | +| 可选服务解析 | `document-llamaparse-v1` | 部署显式启用、用户选择并看到数据流向；固定 API/版本/tier，下载结果后冻结；不作为本地解析失败后的自动外发 | +| 7Z / RAR | `archive-libarchive-v1`：libarchive-c 的已验收解码能力 | 先安全列举再有界读取；具体压缩方法/格式代际单独准入；加密仍拒绝 | +| TAR.BZ2/TBZ2、TAR.XZ/TXZ | `archive-native-extra-v1`：标准库解压 + tarfile | 复用归档限制；保留 C 级部署开关，不因依赖存在自动打开 | +| Pages | `iwork-pages-v1`：Docling format-iwork 路径 | 按容器代际验收正文/结构，未通过保持 planned；不使用 Tika 新代际 metadata 代替正文 | +| Numbers / Keynote | `iwork-legacy-v1`：仅对 Tika 能提取正文且结构验收通过的旧代际启用 | 新代际默认 planned；Numbers 需满足工作表要求，Keynote 需满足演示要求，否则要求用户导出 XLSX/PPTX | +| WPS/ET/DPS 专有格式 | `wps-native-v1` 保留受控扩展接口，首期 planned | 明确要求用户导出 DOCX/XLSX/PPTX；不以改扩展名、通用识别或自动打印成 PDF 宣称原生支持 | +| WPD | `wordperfect-tika-v1`：Tika WordPerfectParser | 仅准入已验收格式代际和内容范围 | +| HWP | `hwp-tika-v1`：Tika HwpV5Parser | HWP v5 单独验收，不扩展解释为所有 HWP/HWPX | +| CHM | `chm-tika-v1`：受控 Tika ChmParser 发现 HTML 成员 → html-dom | 禁止执行帮助程序，章节顺序可验证后开放 | +| ONE | `onenote-tika-v1`：Tika OneNoteParser | 需验证页/节边界和嵌入对象；不将完整笔记同步能力算入文件导入 | +| PST | `pst-tika-v1`：Tika OutlookPSTParser 的受控成员发现 → 邮件适配器 | 完整文件夹/邮件/附件清单及稳定来源规则通过后开放，不自动关联邮箱账号 | +| MOBI/AZW/AZW3 | `ebook-calibre-v1`：隔离的 calibre ebook-convert → EPUB → epub-spine | 仅无 DRM、已验收变体；固定转换版本与选项，标记转换损失，默认不启用 | + +长尾方案依据 [Tika 格式说明](https://tika.apache.org/docs/4.0.x/formats.html)、[Docling 格式表](https://docling-project.github.io/docling/usage/supported_formats/)、[pillow-heif](https://pillow-heif.readthedocs.io/en/stable/pillow-plugin.html)与[calibre 转换接口](https://manual.calibre-ebook.com/generated/en/ebook-convert.html)。具体 profile 是 PowerContext 的实现选择；上述底层能力并不证明已经满足本 RFC 的全部语义。 + +### 1.7 路由、fallback 与冻结规则 + +```mermaid +flowchart TD + A["耐久输入与内容识别"] --> B{"格式族"} + B -->|文本和代码| C["严格解码 + 原文定位"] + B -->|Office / ODF| D["原生结构适配 + 语义审计"] + B -->|PDF| E["文字质量与布局探测"] + E -->|简单原生文字| F["LiteParse；OCR 关闭"] + E -->|复杂布局或探测不确定| G["Docling 布局 / 表格"] + E -->|需要 OCR 且已启用| H["Docling + 固定本地 OCR"] + B -->|图片且 OCR 已启用| H + B -->|邮件 / 归档| I["完整发现清单 → 选择 → 子文档路由"] + B -->|网页 / 电子书 / Notebook| J["专用结构读取"] + C --> K["PC 归一与覆盖审计"] + D --> K + F --> K + G --> K + H --> K + I --> K + J --> K + K --> L{"正文与契约"} + L -->|完整| M["ready 冻结预览"] + L -->|有正文且允许明确部分结果| N["needs_review + 遗漏"] + L -->|空正文 / 硬限制 / 必需语义缺失| O["failed"] +``` + +每份输入在接受时冻结 router/profile 版本与候选链；探测后记录实际分支。`pdf-standard-v1` 只允许按已声明规则从 LiteParse 选择或提升到 Docling，不做结果投票、不默认运行所有引擎。OCR off 不因 fallback 自动开启；能力不足、加密、资源超限和禁止外发不会被另一个引擎绕过。 + +仅在技术性解析故障或结构校验未通过、另一个本地候选满足相同提取契约且 profile 预先声明时，允许一次备用解析；共享原任务截止时间和累计资源预算。已有冻结 bundle 后不再 fallback；用户更换引擎、OCR、视图或接受范围必须新建尝试。缺少部署声明的必需后端时返回 parser_unavailable，不能根据当前机器碰巧安装的库换路由。 + +记录 `selected_profile`、`router_version`、`actual_backend`、后端及模型版本、转换链、路由原因、OCR 实际执行信息、输出 schema 和归一版本。profile digest 覆盖所有会影响选择的策略；bundle/snapshot digest 同时覆盖实际采用的语义处理链。同一输入被两个后端解析出相同文字，也不能隐藏不同的定位、覆盖与处理来源。 + +### 1.8 部署单元与范围收敛 + +基础安装仅包含 A 所需轻量依赖；标准 B worker 固定 LiteParse、Docling、openpyxl、python-pptx、HTML/XML/Notebook 适配器，并以独立 Java worker 承载 Tika/POI。ODF/OOXML 的补充读取由 PowerContext 格式适配模块实现，使用统一的安全 XML 与容器预算，不复制多个无限制 XML 解析入口。OCR 模型、MinerU/DeepDoc、商业服务、专有解码和 calibre/libarchive 扩展独立打包启用。 + +标准 worker 可以预置布局模型；OCR 引擎与语言包只有启用 C 能力时才运行。全部依赖和模型在部署时准备，不在处理用户文件时下载、安装或更新。进程池、并发、PDFium 全局锁、Java 堆和 OCR worker 数分别限流，避免各后端自行开满 CPU。 + +B 级表示目标交付范围而非无条件可用承诺。未达到必需语义的 B profile 必须保持 planned；独立、明确的部分提取 profile 可以用于手动导入，但不能将其计作完整 B 交付。页面展示“正文/公式/批注/定位”等具体能力，避免用一个支持勾选掩盖差异。 + +## 2. 能力发现与识别 + +### 协议兼容与运行能力分层 + +沿用现有 Server 协议发现边界，三个入口回答不同问题，不能用包版本或静态契约推断解析器健康： + +| 入口 | 职责与本 RFC 的扩展 | +| --- | --- | +| `GET /v1/server-info` | 保留部署身份和协议兼容信息；在 OpenAPI 的 `x-powercontext-feature-contracts` 中拟新增 `content.import: {major: 1, minor: 0}`，表示导入协议版本，不表示当前启用 | +| `GET /v1/capabilities` | 拟新增可选 `content_import` 对象，包含 `enabled`、`readiness`（ready/degraded/unavailable）及 `capabilities_endpoint`；缺失时客户端不得假定支持导入。这是当前 Runtime 的汇总，不替代逐格式健康状态或 Scope 授权 | +| `GET /v1/import-capabilities?scope_id=...` | 返回指定已有 Scope 下、当前调用方可用的实际格式、profile、限额及权限；页面据此展示并决定是否允许开始 | + +`content.import` 和 `content_import` 均为拟新增契约，须同步更新 OpenAPI、生成代码、Server 组装和客户端兼容测试。现有两个全局入口继续要求 `server.observe`；仅具有 Scope 权限的普通导入者可以直接查询 Scope 级导入能力，不将全局观测权限新增为导入前提。Scope 级响应也包含导入协议版本与是否启用，客户端不必靠试写探测支持情况。 + +启用由导入服务配置控制，Dashboard 入口开关只控制页面写入口，不能代替 API 服务能力。`readiness=ready` 表示所有已启用且声明可用的 profile 均通过健康检查；部分失效为 degraded，仍可使用健康 profile；禁用或无可用 profile 为 unavailable。未实现的 planned profile 不拉低已启用能力的健康状态。关闭新增导入或解析器不可用，不影响已发布 ImportedContent 和精确 Source 的读取。能力查询是当前快照，prepare、retry 和 confirm 仍须重新检查相关权限、运行状态和冻结结果有效性；confirm 不因某个无关解析器离线而重新解析或丢弃已有冻结正文。 + +### 格式识别与 profile 能力 + +新增 `GET /v1/import-capabilities`；必须携带指向已有 Scope 的 `scope_id` 并要求 `scope.read`；缺失或格式非法返回 422。响应展示是否具备 scope.contribute；开始导入仍须同时具备这两个权限。返回固定的 source_kind=content 以及每种格式的扩展名、规范 MIME、能力级别、状态（available/unavailable/planned）、parser profile、可提取内容、OCR/转换依赖及实际限额。状态来自服务注册和启动自检；不能在 API 上挂载解析器就宣称其 backend 健康。 + +识别顺序结合文件 magic、容器内部结构、声明 MIME 与扩展名。扩展名匹配不区分大小写，但来源路径身份保持原有大小写。扩展名仅是提示；伪装为 `.txt` 的 PDF 进入 PDF 识别或返回类型不一致，不直接解码。DOCX/XLSX/PPTX/EPUB 是专用文档容器，先按专用格式识别，不能当普通 ZIP 拆成一堆内部 XML。 + +未知二进制拒绝。无扩展名或未知后缀的文本，仅在显式 `as_text` 模式下进入纯文本兜底，响应记录其来源。服务端与 worker 选择相同、固定版本的 parser profile。 + +能力响应按“格式代际 + profile”返回 `profile_id`、`profile_digest`、`router_version`、默认标记、后端候选、实际安装版本、`extraction_scope`、`required_features`、`optional_features`、`locator_kinds`、`known_limitations`、依赖、数据流向和限额。公式、缓存值、批注、修订、备注、隐藏内容、附件发现及 OCR 分别声明；依赖未健康为 unavailable，适配/语义验收未完成为 planned，全部前置满足才 available。`available` 表示这个明确范围的 profile 可调用，不代表同扩展名所有 profile 都能完整提取。 + +同一后缀可有一个标准完整 profile 和显式选择的部分 profile；默认路由不得自动从完整切到部分。上传时校验所选选项是否在能力范围内：未知或矛盾组合为 422，识别后发现必需语义不受支持则解析失败 `capability_insufficient`。不能以 accept_partial 绕过格式识别、安全、大小、解密及用户选择的必需语义。仅已声明允许部分结果的 profile，可在有正文且遗漏可解释时生成 needs_review。 + +## 3. 数据模型 + +新增 `ImportedContent` 类型及 `importedcontent` Family（显示名“导入资料”），由 ContentImportApplication 管理写入。一份逻辑文档对应一个 Artifact，各次接受的快照对应不可变 Revision。不另建独立 ImportRecord 作为第二份资料身份与内容权威。ImportedContent 表示已经接受的内容，不表示上传任务、解析中间态或模型生成的 Memory。 + +```mermaid +flowchart TD + I["ImportSession：创建者、固定且非空的已有 scope_id"] --> J["内部 operation / item_id：临时解析与确认状态"] + J --> B["ImportedContentBundle：冻结内容"] + B -->|原子提交| R["ImportedContent Artifact Revision"] + S["Scope"] --> H["Artifact Head：最近接受的版本"] + H --> R + R -->|有序 lineage.sources| C["ContentSource 1..N：正文与定位"] + D["下游 Artifact Revision"] -->|实际使用的精确 SourceRef| C + D -.->|确实以文档版本为输入时才记录 ArtifactRef| R +``` + +| 对象 | 职责与持久化 | +| --- | --- | +| ImportSession | 创建者 Principal、固定且非空的已有 scope_id、期限；承载同一目标下的导入恢复、隔离及并发控制，不授予额外 Scope 权限 | +| 内部操作记录 / item_id | prepare 返回 operation_id；记录临时输入、发现清单、租约、配额、解析与确认状态；item_id 仅在 operation 内稳定，不是资料永久身份，也不是 Artifact | +| ImportedContentBundle | 解析后冻结的正文、语义 manifest、coverage 与 digest；确认针对它，不重新计算预览内容 | +| ImportedContent Artifact | 稳定身份 `(scope_id, family=importedcontent, artifact_id)`；首次手动导入或稳定来源首次出现时分配 | +| ImportedContent Revision | 精确快照 `(scope_id, family, artifact_id, revision)`；含结构化元信息和有序 Source 映射 | +| Artifact Head | 最近成功接受的 Revision；更新通过 expected_head_revision 乐观并发控制，不等同外部实时版本 | +| 来源绑定 / 请求映射 | 稳定来源键到 Artifact 身份、输入解析缓存、幂等键到精确回执；属于索引与操作账本，不复制一份 ImportRecord | + +### 类型命名与来源语义 + +领域/Python 类型为 `ImportedContent`，Family 的持久化及 HTTP wire name 为 `importedcontent`。该 Family 统一承载外部资料的已接受快照，`source_kind` 固定为 `content`，表示正文使用现有 ContentSource，不区分粘贴、文件、网页等输入方式。 + +| 字段或概念 | 含义与约束 | +| --- | --- | +| `family=importedcontent` | 管理整份导入资料的身份、版本与分段关系 | +| `source_kind=content` | ImportedContent 元数据中的固定值；所有正文分段均使用现有 content Source | +| `input_media_type` | 原始输入格式，例如 text/markdown、application/pdf、text/html | +| `origin` | 具体出处的 namespace、key 和可选 display_name，不作为授权依据 | +| `parser` | 实际解析器、配置、版本、OCR 和转换链 | + +ImportedContent Revision、冻结 bundle 和读取响应必须包含 `source_kind: "content"`。texts 与 prepare 请求可省略该字段，由服务端补齐；显式提供时只能为 content，其他值返回 422。先补齐固定值再计算请求指纹、冻结摘要和缓存键，因此省略与显式 content 在其他字段相同时语义一致。OpenAPI 使用单值约束，Python 模型使用 Literal["content"]；默认值不意味着允许调用方选择其他 Source 类型。 + +此字段不重命名或替代已有 `SourceRef.source_type`。保存时校验所有 parts/lineage.sources 的 `source_type` 均为 content，且与 ImportedContent 的 source_kind 一致；不能把其他 Source 类型或系统保留角色混入正文。标准列表不新增 source_kind 筛选或索引，因为该 Family 的取值恒定；出处仍可按 namespace 筛选。 + +输入方式由文本入口、文件入口或获准 Connector 决定，格式和解析 profile 决定如何提取。上传 HTML、PDF、TXT，粘贴文本及将来的网页快照在形成文本证据后都使用 source_kind=content。统一 Source 类型不表示无需解析二进制，也不表示已经支持网页抓取；网页链接获取仍属独立 Connector，本期不提供任意 URL 采集入口。 + +`origin` 使用 namespace、key 及可选 display_name；连接身份、source_authority_id 和稳定来源绑定由获准 Connector 路径生成/校验，客户端填写出处不能证明授权。未来网页采集需由 Connector 提供页面身份、抓取时间和经脱敏的显示 URL；URL 不是服务器任意取数指令，也不默认等同稳定身份。完整凭据、签名 URL、敏感查询参数不得进入长期元数据。归档和邮件成员通过 container_chain 保留父容器及成员身份；不能把无正文的容器伪造为 ImportedContent。 + +固定的 source_kind 作为 schema 字段进入 bundle_digest，并间接进入 snapshot_digest；它不决定解析路由、artifact_id、来源绑定或缓存分区。显式更新同一 Artifact 时仍校验 expected_head_revision，并按完整快照语义决定 reused 或新增 Revision；即使从文件更新为粘贴文本，各版本的 source_kind 都为 content，具体出处、格式与解析信息分别随版本保留。 + +### ImportedContent Revision 的 content + +| 字段 | 约束 | +| --- | --- | +| `schema_version`、`title` | title 去首尾空白后 1–256 个 Unicode 字符 | +| `source_kind`、`origin` | source_kind 固定为 content；origin 包含必填 namespace、key 和可选 display_name（1–256 个 Unicode 字符，无控制字符）；仅作出处，不证明授权 | +| `input_media_type`、`input_bytes`、`input_digest` | 原始输入类型、长度和 SHA-256；粘贴输入为接收的 UTF-8 字节 | +| `text_media_type`、`text_bytes`、`text_digest` | text/plain 或 text/markdown、完整提取文本长度及摘要 | +| `parser` | selected_profile/profile digest、router_version、实际后端及版本、路由原因、转换链、输出 schema、规范化和分段版本、编码、OCR 模型/语言与实际执行信息；这些语义信息纳入冻结摘要 | +| `bundle_digest`、`snapshot_digest` | 冻结解析结果及其接受语义的摘要，不参与逻辑文档身份，不包含运行或保存后分配的 ID | +| `content_manifest` | 有序 ordinal、字节数、分段摘要、全文字符区间和定位，不含 SourceRef、journal position 或临时 blob ID | +| `parts` | ordinal 对应的 SourceRef 与 journal position，和 manifest 一一对应；SourceRef 顺序必须与 lineage.sources 完全一致 | +| `coverage`、`warnings`、`omissions`、`accepted_partial` | 声明范围内的完整性与显式接受决定；已知遗漏长期保留 | +| `container_chain` | 逐层外部文件摘要、成员路径/序号及出处；与跨快照稳定成员身份分开 | + +正文仅由 Source 长期保存，不在 content 中重复保存全文。Source metadata 保存 ImportedContent artifact_id、revision、part ordinal、source_kind、受控来源与定位；其中保存后分配的 artifact_id/revision/SourceRef 不参与内容摘要，source_kind、来源及定位已在冻结 bundle 中参与摘要。普通导入生成普通 captured ContentSource，显式保证 `working_note=False`、`internal=None`、`handoff_receipt=False`；不能设置 lineage_only、handoff receipt、working-note metadata 或其他系统保留角色。`source_kind=content` 只约束类型，不等于生成准入已满足；保存前同时通过既有 Source 类型及角色校验，拒绝客户端借 metadata 伪装内部 Source，不静默接受后再改变其语义。 + +namespace 最长 128 字符、key 最长 1024 字符，均非空且无首尾空白、控制字符。文件 key 为相对 POSIX 路径，拒绝绝对路径、反斜线、`.`、`..` 和空片段，不做大小写折叠。来源不保存凭据、签名 URL 或本机绝对路径。origin 为必填对象，namespace/key 由调用方提供；CLI/页面可根据输入生成并展示出处标签，但不按文件名匹配既有 Artifact；粘贴接口中的 key 是出处标签，不是稳定来源绑定。 + +### 如何判断同一份文档 + +| 输入方式 | 稳定身份规则 | +| --- | --- | +| 首次手动上传 / 粘贴 | 新建 artifact_id；同名、同内容也不自动归并。创建请求同键重放返回原文档 | +| 更新已有文档 | 显式提供 artifact_id 与 expected_head_revision；文件名、格式和正文可改变 | +| OpenDAL 本地目录 | Scope + 稳定 source_authority_id + 相对路径；namespace/显示名不代替 authority | +| OpenDAL 对象存储 | Scope + 稳定 source_authority_id + object key；ETag、mtime 和文件摘要不作逻辑身份 | +| 将来的知识平台 Connector | Scope + 受信平台实例/连接身份 + 原生 document ID；显示标题和可变 URL 不作身份 | +| 容器成员 | 父文档稳定来源键 + 逐层稳定 member key；ZIP 使用规范成员路径且禁止重复覆盖，邮件优先使用经验证唯一且稳定的原生 ID | + +source_authority_id 是持久配置身份，指向固定存储根/桶和连接来源，不是凭据文本，也不是每次执行的 run ID。更换根或桶视为新来源；轮换凭据、修改 filter 或 parser profile 不自动改变文档身份。对新增来源绑定的创建与使用按该 Scope 的连接配置授权控制，任意上传 origin 字段不能接管别人的绑定。 + +成员 ordinal 仅能保证单次 operation 内定位稳定，不能保证跨版本对应。无唯一稳定 ID 的邮箱邮件或同名附件使用“父输入摘要 + 逐层 ordinal”作为快照局部身份；外层内容改变后默认新建文档，或由用户显式指定更新目标，不猜测一一对应。container_chain 保留外层摘要用于溯源，但有稳定 member key 时不把它加入跨版本身份。路径重命名且无原生稳定 ID 时视为新文档；旧文档保留。 + +同一 Scope 内稳定来源绑定使用唯一约束，防止并发首次采集创建两个 Artifact。显式手动新建不受 filename 或 digest 唯一约束。同一内容可以属于两份不同文档；内容缓存去重不能改变文档身份、权限或来源。 + +### 复用现有表、注册与读取能力 + +ImportedContent 的唯一持久资料身份使用 `(scope_id, family=importedcontent, artifact_id)`。复用 `pc_artifacts` 保存 content 与不可变 Revision、`pc_artifact_heads` 保存已接受 Head、`pc_artifact_lineage_sources` 保存有序 SourceRef，正文使用 `pc_sources`。确有上游 Artifact 输入时才使用 `pc_artifact_lineage_artifacts` 的精确版本引用。不新增 ImportedContent 主表、ImportedContent 版本表或独立 ImportRecord。 + +Family 注册覆盖类型、序列化、读取、授权与允许的操作，复用对应的既有 registry、授权 profile 和路由适配点；不能只增加一个 Family 枚举就认为所有通用接口已经适配，也不要求为本期另建全局插件框架。更新 OpenAPI family schema、服务端能力与 Dashboard 映射，明确以下边界： + +| 能力 | 本期行为 | +| --- | --- | +| list/get、精确 Revision、历史、版本比较 | 支持；统一以所在 Scope 的 `scope.read` 授权 | +| texts/prepare/confirm 专用写入 | 支持；要求 `scope.read` + `scope.contribute`，并执行会话创建者隔离和事务内校验 | +| 通用 Artifact POST/PUT 创建或替换 | 不支持；不能绕过确认提交 parts、lineage 或成功状态 | +| generic restore/rollback、merge/undo_merge | 不支持；不注册相应适配器、不加入允许列表，HTTP 与 Runtime 均拒绝，页面不展示动作 | +| 模型生成、Dream/Candidate 发布 | 不支持；不注册 ImportedContent 生成或候选处理器 | +| 独立 Artifact 分享与 owner 授权 | 不支持；已发布资料随 Scope 可见，创建者信息只用于审计 | +| Artifact 全文/语义搜索 | 本期不支持；不注册 searcher，不把 namespace 列表过滤称为正文检索 | + +ImportedContent 是 Scope 内共享资料。现有通用 Artifact 读路由对多数 Family 使用 `artifact.read`，写路由及部分 Family 又有独立策略；接入必须同时适配 HTTP 访问声明/解析、运行时授权 profile、标准列表/精确读取、Source 读取与页面动作判断，不能让列表可见而详情仍要求 Artifact owner，也不能照搬 Prompt 的管理员写入规则。会话创建者约束仅适用于未发布输入和操作恢复，发布后不成为永久 owner；移除 Scope 读权限后，创建者也不能依靠旧回执访问正文。治理动作保持既有独立授权，不由 `scope.contribute` 隐式获得。 + +通用恢复路径根据历史 content 构建新 Artifact lineage，不能保证复制本 Family 的有序 `lineage.sources`，因此不可直接启用。即使只需要恢复旧正文，也须未来另行定义专用动作：校验完整 parts 与精确 Source 可读性，明确复用或新建证据、版本 provenance、Head CAS 及下游 pending/dirty 行为；本期不将通用 PUT 或历史版本页面按钮当作恢复入口。 + +已提交资料列表、当前版本、历史与精确版本读取复用通用 Artifact API；正文按 parts 中的精确 SourceRef 使用已有 Source 阅读能力组合。来源筛选可增加可重建投影/索引；反向 lineage 查询只表示实际引用到的分段或 Artifact 版本，不推断消费了全文。 + +### 操作存储的最小职责 + +异步执行需要耐久状态，但它不构成第二套资料模型。以下表名是拟新增的存储契约，不声称仓库已有通用任务表可以直接复用： + +| 存储 | 必要职责与约束 | +| --- | --- | +| `pc_import_sessions` | 会话创建者、固定非空 scope_id 与期限；创建时校验目标已存在且可贡献，目标不可修改 | +| `pc_import_operations` | operation_id、session_id、family、操作种类、原始输入暂存引用/摘要、冻结配置、执行状态、租约/generation、阶段重试与截止时间；仅服务执行恢复 | +| `pc_import_operation_items` | `(operation_id, item_id)` 唯一；成员清单、冻结目标、选择/解析/确认状态、bundle 暂存引用与摘要、最终精确 ArtifactRef；分成员原子提交 | +| 来源绑定、请求回执与输入索引 | 稳定来源键到 Artifact 的唯一映射、幂等请求到接收/提交结果、可复用解析结果索引；不复制已保存正文或版本权威 | +| 配额账本与可重建查询投影 | 跨进程准入/释放、来源过滤和反查；不能以无锁内存计数替代共享账本 | + +操作/成员记录支持有界状态更新、租约查询和清理；原始文件与待确认正文位于受控 staging store，不塞入 Artifact content 或操作表大 JSON。确认后长期正文、provenance 和来源关系以 Source + ImportedContent Revision 为准，操作记录仅保留恢复所需回执并按 TTL 清理。 + +不提供独立 FileImportJob 的业务 CRUD、永久详情页或与 ImportedContent 重复的资料目录。前端可以恢复“尚未完成的导入操作”，但操作编号不是 ArtifactRef；操作句柄查询是异步协议的一部分，不授予底层状态的任意写入。少量操作表不能消除，但不按输入格式新增任务领域类型。 + +### 写入与可见性边界 + +ContentImportApplication 作为该 Family 的专用写入适配器，持有一个数据库事务并复用 ArtifactRepository 的 create/revise。现有通用制品创建路径生成的命令 Source 不能替代真实正文分段；该适配器必须保证 parts 与 lineage.sources 一一对应。所有入口共用同一提交服务，不能直接调用通用 PUT 绕过确认。 + +新资料在确认前不创建 Artifact/Head/正文 Source；可在操作账本预分配 artifact_id，但普通 Artifact list/get 不可见。已有资料更新期间 Head 始终指向已接受版本，解析失败、重试、取消、进度变化不会新增 Revision 或改变 Head。仅确认成功后原子追加 Revision 并切换 Head，心跳和阶段变化只更新操作记录。 + +`pc_artifact_heads.lifecycle_state` 的 active/deprecated/retired 继续表示制品治理状态，不能塞入 uploading/parsing/failed。预览按会话创建者鉴权；其他具有 scope.read 的普通用户只能读已发布内容。确认时在同一事务中锁定/校验当前 Scope 权限、目标 Head/治理状态和操作有效性，已 retired 或已合并到其他目标的制品按既有不可写规则拒绝；导入不自动复活、不自动跟随合并目标，也不提供恢复已退役制品的入口。 + +ImportSession 从创建起即固定已有 scope_id。会话与内部操作记录用于私有预览、恢复、租约和 TTL；这些可变状态不写成 Artifact Revision。会话目标、权限与幂等规则遵循第 7 节。 + +## 4. 规范化、定位与分段 + +纯文本严格按确定的编码解码，去除一个开头 BOM。保留换行、缩进及 Unicode 字符，不将非法字符替换成 U+FFFD，不做全局 NFKC 或代码格式化。拒绝空白正文、NUL、surrogate 和除 TAB/CR/LF 外的 C0 控制字符；原始控制字符密集的文件不按普通文本强行导入。 + +文档转换输出固定为 UTF-8，换行采用 LF；转换策略由 parser profile 固定，采用第 1 节的逐格式路由；Office 原生语义不默认经过 PDF 转换。CSV、代码和配置的原始标记可作为 text/plain 保留；格式专用结构化输出只有在明确选用并验收时启用，不能因 JSON/XML parse 失败而悄悄改成另一种语义。 + +提取结果先形成有序 ImportedContentBundle：完整正文、结构节点、页/表/幻灯片/单元/邮件定位、coverage 和 omissions。定位必须区分 PDF 页码、演示页码、工作表与 A1 范围、段落序号、代码行、Notebook cell index，不能给没有稳定页码的 Word 文档编页码。OCR 可选 bounding box 同时保存坐标系与图像尺寸。定位还记录所指对象为原始文件或转换产物；转换 PDF 的页码/bbox 不冒充原工作簿单元格或演示对象。未启用、已启用但为空、提取失败和覆盖未知分别表示。首期仅长期保留文字与结构定位，不提供依赖临时截图、转换 PDF 或预签名 URL 的永久跳转；相关资源清理后仍能精确阅读已保存 Source。 + +分段按章节、页、幻灯片、表格行组和段落等自然边界进行，单段最多 256 KiB UTF-8；过长单一段落再按行和 Unicode 字符边界切分。分段算法固定版本，无重叠、不截断。按 ordinal 拼接分段正文必须精确恢复冻结的完整提取文本，text_digest 一致。用于展示的标题和重复表头放 metadata，不能偷偷插入每段正文导致重建不一致。 + +每文档最多 8 MiB 提取文本、128 段，定位与 manifest 最多 2 MiB；超限拒绝该文档而非静默保留前半部分。每段内部 Source serialization 还须满足现有 4 MiB observation 上限。处理器单次只消费部分 Source 时继续遵守自己的进度，不把文档级原子保存误解为原子生成。 + +## 5. 大小、时间与解析限制 + +| 对象 | 初始默认限制 | +| --- | --- | +| 直接粘贴 | 正文 256 KiB，JSON request 2 MiB | +| 单个上传文件 | 50 MiB；按实际流读取计量 | +| Dashboard 一次选择 | 100 文件、合计 200 MiB；最多 4 个上传并发 | +| PDF/演示/多页图像 | 500 页/幻灯片/帧；启用 OCR 时每任务最多 200 页 | +| 工作簿 | 100 工作表、100,000 个实际非空单元格；与提取字节上限同时生效 | +| 归档/邮箱容器 | 最多 1,000 个成员、展开后合计 200 MiB、单成员 50 MiB、显式嵌套深度最多 2 | +| 压缩比 | 展开实际累计字节/已消费压缩字节不超过 100，实际展开字节仍独立计量 | +| 单个文件 operation 的全部子文档 | 累计提取文本最多 32 MiB、分段最多 512；各文档自己的上限同时生效 | +| 单图像解码 | 最大 40 百万像素；不得仅根据压缩文件大小判断成本 | +| 普通解析/OCR 任务 | 120/600 秒，受独立进程内存与 CPU 额度限制 | +| 暂存期 | 24 小时，具体 expires_at 在任务中返回 | +| 目录扫描 | 每轮最多 10,000 个选中输入；有界 listing 和读取 | + +这些是提案默认预算，实施前用真实样本验证；部署可在已验收的硬上限内降低。每种格式返回实际限制，不能只提供一个含义不明的 max_file_size。归档的展开限额同样作用于 Office/EPUB 等内部压缩结构；文档内部节点不按普通归档 1,000 个成员限制，应由专用 XML 节点、页数和输出预算控制。 + +超限错误指出输入大小、解压大小、页数、像素、单元格还是提取结果超限。不得无界载入后才判断大小，不得通过文件格式的声明长度绕过实际流计量。 + +### 服务端累计配额与背压 + +浏览器并发限制只改善交互；所有入口共用服务端配额，不能通过多标签页、CLI 或多个 API 副本绕过。以下是可配置的提案默认值，能力接口返回实际值与当前调用方可见的剩余额度，不公开其他 Principal 的使用详情。 + +| 预算 | 每 Principal、每 Scope、整个部署的默认上限 | +| --- | --- | +| 暂存实际字节（原文件、展开文件、输出缓存合计） | 1 GiB、5 GiB、20 GiB | +| 活跃任务槽（按下表的实际工作状态计量，不只看 aggregate_status） | 20、100、500 | +| 同时运行的普通解析数 | 2、4、8 | +| 同时运行的 OCR 数 | 1、2、4；同时占用普通解析并发槽 | +| 每 UTC 自然日实际尝试的 OCR 页数 | 1,000、5,000、20,000 | + +任务从接收起即归属创建者 Principal 和会话固定的 scope_id；管理员代操作仍记入原归属。所有入口始终同时满足 Principal、目标 Scope 与部署三个层级的配额才准入；同一 Scope 的多个会话共享 Scope 预算，不能通过创建会话绕过累计限制。所有预留、结算、释放和重试均使用会话固定的 scope_id，不设临时 Scope 额度或配额迁移。接收上传前，在共享耐久配额账本中原子预留任务槽和声明字节；未知长度先预留单文件上限，超出预留必须先扩容或停止读取。解析前预留有界展开/输出空间、并发槽与 OCR 页数；流式处理按实际占用结算，任何写入不得先越界后补账。上传准入失败直接拒绝且不创建任务；已接受任务在解析预留失败时保持可恢复的排队状态，并返回 blocked_reason/retry_after，不占用运行槽。 + +任务槽与 parser/OCR 运行槽分别计量。同一 operation 最多占一个任务槽；状态变化与配额账本在同一事务中更新，按以下规则决定是否占槽： + +| 工作状态 | 任务槽 | +| --- | --- | +| uploading、queued、running；含等待解析资源的队列 | 占用 | +| idle 且成员发现完成等待选择，或有选中未提交的 ready/needs_review 文档 | 占用 | +| partially_committed 且仍有排队、解析、待选择或待确认工作 | 占用 | +| failed；或 partially_committed 仅剩失败文档且没有已安排的自动重试 | 不占用，显式 retry 时重新准入 | +| completed、cancelled、expired | 不占用 | + +自动重试已被安排的任务保持 queued 并持续占槽；自动重试耗尽进入 failed 后释放任务槽。释放任务槽不释放仍存在的暂存字节，也不清除可重试资格。任何操作将不占槽任务转回排队或待处理状态，必须在同一事务内重新申请 Principal、Scope 和部署三个层级的任务槽；多个文档一起 retry 仍只申请一个槽。已有槽的任务不得重复计费。配额不足返回 429/503 和 Retry-After，任务、文档、generation 与重试次数保持不变，不入队、不形成成功接收回执；同一请求可在退避后使用原幂等键再次准入。已经成功接收的同键重放仅返回回执，不重复申请槽。 + +并发槽绑定租约并以 fencing 防止失效 worker 继续写入；多副本共用事务或等价原子账本。取消、任务进入无活动工作的失败状态和过期释放未消费的预留；单个文档失败不得释放其他活动文档所需的任务槽或预留，已使用磁盘空间仅在实际删除后释放；任务结束不代表 blob 已删除。OCR 已尝试页数不因失败或取消退款，重试再次消耗预算，仅释放未使用预留。故障恢复定期对账任务、blob 与租约，孤立文件也占用预算直至删除。已保存 Source 的长期存储另按现有存储治理处理。 + +Principal/Scope 配额不足返回 429，部署容量或依赖不足返回 503，均携带 Retry-After；单个输入硬上限仍为 413。已接受任务的等待原因通过查询返回，worker 持久化下次重试时间并带抖动退避，不能热循环重复上传。 + +## 6. 内部执行、私有预览与耐久暂存 + +复杂文件由 ImportedContent Family 的 prepare 操作接受，返回可恢复的 operation_id；执行状态保存在第 3 节的内部操作账本，由独立解析 worker 处理。ImportedContent Revision 仅管理已接受快照。本文下称“任务”均指该操作的内部执行工作，不引入 FileImportJob 领域实体或另一类 Artifact。 + +可以复用经过验证的数据库事务、租约、fencing、背压与 worker 基础组件；不把现有 Artifact Processing Supervisor 的生成任务直接改作导入任务。现有生成调度不自然具备上传暂存、私有预览与人工确认语义；如抽取公共执行基础设施，必须保留本文状态、权限和恢复约束。解析成功只冻结 bundle，不直接触发 Source/Artifact 写入。 + +执行状态、成员解析状态和成员确认状态分别存储，不能共用一个 status 字段；“ready”不是 Artifact 的发布或治理状态。 + +| 层级 | 字段与取值 | +| --- | --- | +| 任务执行 | execution_state：uploading / queued / running / idle / failed / cancelled / expired；idle 表示当前无待执行解析 | +| 任务阶段 | phase：upload / discovery / selection / parsing / confirmation / done；discovery_state：not_required / pending / running / complete / failed | +| 文档选择 | selected：是否纳入本次导入；容器发现完成后通过 selection API 冻结选择 | +| 文档解析 | parse_state：pending / parsing / ready / needs_review / failed；ready 和 needs_review 都对应不可变 bundle | +| 文档保存 | commit_state：uncommitted / committed；提交事务进行中不公开半提交状态 | + +```mermaid +flowchart TD + U["任务：uploading"] --> Q["任务：queued"] + Q --> R["任务：running；发现文档清单"] + R --> I["任务：idle；等待选择或确认"] + I -->|冻结选择，有待解析文档| Q + R -->|暂时故障，最多自动重试三次| Q + R -->|任务级失败或自动重试耗尽| F["任务：failed"] + F -->|发现阶段可重试，原输入有效且配额重新准入| Q + Q -->|取消或到期| X["任务：cancelled / expired"] + R -->|取消或到期| X + I -->|取消或到期| X +``` + +```mermaid +flowchart LR + P["文档：pending"] --> R["文档：parsing"] + R --> A["文档：ready"] + R --> B["文档：needs_review"] + R --> F["文档：failed"] + F -->|可重试、输入有效且配额准入| P + A -->|确认冻结 digest| C["保存：committed"] + B -->|确认 digest 并接受 partial| C +``` + +任务另外返回派生 aggregate_status 与各类计数，按以下优先级计算：cancelled/expired；正在接收输入为 uploading；存在待执行或运行工作时为 queued/parsing；无活动工作且有已提交文档但未全部完成时为 partially_committed；选中非空且全数 committed 为 completed；存在未提交 ready/needs_review 或待选择成员为 awaiting_confirmation；任务级失败或仅剩不可解析成员为 failed。未选成员计入 excluded，不要求其提交。取消/过期仍返回已提交计数与 ImportedContent Revision 引用，不能隐藏部分成功。 + +普通单文档输入自动固定 selected=true。归档、邮箱或带附件文件先有界发现完整成员清单，返回稳定 item_id、container_chain、类型及默认选择建议；附件默认不选，普通归档成员默认选。发现后暂停，调用方通过 selection API 一次冻结明确的成员 ID 集合，再开始正文解析。超限或无法完成发现不能冒充完整清单；返回 discovery 错误。容器结构节点不可选择，未选附件记 excluded。选择不再改变；改变成员集合或解析选项须创建新尝试。临时 item_id 由本 operation 的持久成员映射产生，自动重试不重新分配。discovery_state=complete 与完整成员清单在同一事务中发布，只有此时才允许 selection。发现期间的中间清单不允许选择，不据其推进 checkpoint。临时故障令 discovery_state=failed 并返回阶段错误和 discovery_retryable；自动重试耗尽仍可通过显式 discovery retry 恢复。若完整清单尚未发布，重试可从原始暂存重新发现并复用内部成员身份；已发布清单不可重新发现、改写或重新分配 ID。损坏、加密、超限等永久错误标记不可重试。 + +原始字节耐久写入受控 staging store 并校验摘要后才返回 202 和排队任务。上传中断不留下可执行的半文件，上传占位及孤立 blob 由 GC 对账回收。单机可用受控状态目录，多副本使用共享耐久存储，不能依赖 API 进程的 /tmp。任务保存固定 profile、输入摘要、租约 owner/generation、attempt、deadline、选择版本及冻结 manifest。领取、结果发布、重试、取消和过期均以事务校验状态及 generation；过期 worker 不得覆盖新结果。 + +临时失败最多自动重试 3 次，永久格式/加密/损坏/超限错误不自动重试。任务内重试仅处理尚未冻结的文档，不覆盖 ready、needs_review 或 committed。显式 retry API 区分成员发现与文档解析两个阶段，只恢复尚未冻结且标记可重试的失败工作；两个阶段的自动重试计数分别记录，手动重试不重置计数，仍受截止时间、配额和次数记录约束。冻结 ImportedContentBundle 返回 bundle_digest；确认必须针对该值，不能预览 A 后保存重算 B。ready 只表示声明提取范围内完整且必需语义满足，已知遗漏或无法证明声明范围完整时为 needs_review；必需语义不足且 profile 不允许对应部分结果则 failed；accept_partial=true 的接受决定永久保存。 + +任务 TTL 从上传占位创建起最长 24 小时，且不晚于 session.expires_at；重试和选择不延长。到期或取消先封闭未提交文档写入、撤销租约，再清理临时数据；与正在确认的事务串行化：先成功的确认保留回执，先取消/到期则未提交确认失败。取消不删除已保存资料。任务状态及幂等映射保留至 expires_at 后 30 天：查询返回 200 的 tombstone（状态、时间与耐久回执引用）；过期后的未提交文档预览/确认返回 410，取消后返回 409；同上传键重放返回同一任务或 tombstone，不会隐式新建。超过 idempotency_expires_at 仅承诺通过永久 ImportedContent Revision 查询，客户端必须使用新尝试键。 + +全部选中成员保存完成后的原始文件与解析缓存最迟 1 小时内清理；取消/到期数据也在 1 小时清理窗口内回收，期间继续计入配额。未全部处理的容器保留到 expires_at。已提交预览从 Source 读取，ImportedContent Revision 的回执、定位和遗漏长期保留。staging blob 不公开为任意读取地址，也不承诺原始二进制长期下载。 + +## 7. 身份、幂等与原子保存 + +### 会话与 Scope + +ImportSession 创建必须提供已有 scope_id；服务端校验 Scope 存在且调用方具备 scope.read + scope.contribute 后，保存创建者和不可变 scope_id。该字段非空，不接受延迟解析的目标或创建参数。创建会话本身不创建 Scope、Artifact 或 Source，也不授予新权限。 + +会话、operation、item、暂存配额及最终 Artifact/Source 始终归属同一 Scope。子操作通过 session_id 获取固定目标，不允许请求正文覆盖 scope_id；其他 Scope 的 artifact_id、SourceRef 或来源绑定不能作为当前目标。切换目标需重新创建会话并重新提供输入；已有操作不转移。 + +创建会话、提交文本、接收上传、选择成员、重试与确认均重新校验该 Scope 的当前权限；操作查询/预览要求会话创建者及当前 Scope 访问权限，管理代操作须显式授权并审计。确认事务还须验证 Scope 仍存在且可写、会话/操作未过期或取消，以及目标 Head/治理状态；撤权或目标失效后不保存资料，不自动补建 Scope 或选择其他目标。取消与清理按第 6、10 节执行。 + +会话上传有效期为创建起 24 小时；新 operation 及子操作不延长它，operation.expires_at 不晚于会话期限。会话与操作回执至少保留至最后一个 operation.expires_at 后 30 天，便于结果未知时恢复。会话到期禁止新上传与未提交保存，保留已提交回执；不删除目标 Scope、Artifact 或 Source。 + +### 内容、身份与请求分别建模 + +Idempotency-Key 必填，长度 1–128，允许 ASCII 字母、数字及 -_.。创建会话映射键为稳定 Principal + operation + key，规范化后的 scope_id 必须计入创建请求指纹；同键改换 Scope 返回 409。其余导入写操作为 Principal + session_id + operation + key，session_id 唯一确定不可变目标 Scope。相同键、相同请求重放原结果，不同请求返回 409；每次重放重新授权。重放沿用首次冻结的默认值/profile。文件指纹包含实际输入摘要、规范化 metadata 与目标文档意图；所有指纹包含动作名、路径/请求中的 operation_id 与 item_id（适用时），避免同键操作不同句柄被错误重放；提交指纹另外包含 bundle_digest、accept_partial 及冻结目标/预期 Head。 + +区分四种身份：artifact_id 是逻辑文档；operation_id/item_id/attempt key 是一次尝试；bundle_digest 是冻结解析结果；ArtifactRef 是一次已接受版本。snapshot_digest 表示内容及接受语义，可在不同文档甚至同一文档的多个历史 Revision 中重复，不能加 `(scope_id, snapshot_digest)` 全局唯一约束。 + +bundle_digest = SHA-256(RFC 8785 canonical JSON)，输入为带 schema_version 的不可变对象：source_kind、受控来源、container_chain、title、input_media_type/input_bytes/input_digest、parser/profile/encoding/OCR 与规范化/分段版本、text_media_type/text_bytes/text_digest、有序 content_manifest、coverage、结构化 warnings/omissions。manifest 仅含 ordinal、字节数、分段摘要、全文字符区间及定位，不引用临时 blob。诊断消息、耗时、时间戳、Principal、session/operation/run/item ID、artifact_id/revision、SourceRef、journal position 全部排除。snapshot_digest 对 `{schema_version, bundle_digest, accepted_partial}` 规范化后计算，仅 needs_review 可令 accepted_partial=true。保存事务附加 parts 回执，不改写冻结摘要。 + +### Revision、Head 与重复输入 + +| 情况 | 行为 | +| --- | --- | +| 首次手动新建 | 本次创建意图分配 artifact_id，保存 Revision 1；同名或同内容的另一新建意图仍是另一文档 | +| 更新指定文档，预期 Head 匹配且 snapshot_digest 等于当前 Head | 返回 reused 与当前精确 ArtifactRef，不新增 Source 或 Revision | +| 更新指定文档，预期 Head 匹配但快照不同 | 新增 Revision 与分段，原子推进 Head；旧证据不变 | +| 预期 Head 与实际不符 | 返回 409 revision_conflict；展示当前版本，由调用方重新确认，不静默覆盖，即使内容碰巧相同也不跳过冲突 | +| A → B → A | 创建 R1(A)、R2(B)、R3(A)；不能用全局摘要命中把 Head 偷换回 R1。本期 R3 创建自己的 Source 分段 | +| 已成功请求同键重放 | 返回第一次的精确回执，即使 Head 已推进；不新增版本，也不回退 Head | +| 同输入重新解析 | reparse=true 新建尝试；若当前 Head 与冻结语义相同则 reused，实际输出或影响输出的 parser/profile 变化则新 Revision | + +手动更新在创建 item 时冻结 artifact_id 和 expected_head_revision。持续采集在接收输入时从来源绑定读出目标和预期 Head；新来源冻结“预期不存在”，绑定唯一约束冲突返回可恢复冲突，不能把旧解析结果自动覆盖新版本。会话中明确的新建意图在重试时保持同一分配身份，跨独立新建请求不按文件名或摘要合并。 + +### 解析复用与部分提取 + +任务跨键复用仅允许同创建者、同 session、同明确目标文档与预期 Head、相同原始输入/origin/title、固定 profile/选项和冻结选择。独立手动新建意图不跨键合并。任务须未到期、暂存可用、execution_state 为 queued/running/idle、有未完成工作且没有已提交 partial；failed/cancelled/expired/completed 不参与。合并检查与请求映射写入以事务/锁串行化。同上传键始终恢复原任务或 tombstone；reparse=true 禁止合并。不同 Principal 不返回对方私有任务或预览。 + +耐久输入映射以明确逻辑文档目标、输入摘要、受控来源/title、profile 和完整选项为键；容器包含稳定成员映射与选择。它索引已冻结解析结果及精确历史 ArtifactRef，而不是授予覆盖 Head 的权限。缓存命中可以省略 OCR，但必须按本次目标与预期 Head 决定 reused 或创建新 Revision；不能直接把历史版本当当前版本。手动新建仍创建新文档，跨 Scope 不暴露命中。 + +缓存 coverage=complete 可以复用解析结果，但交互文件仍需本次确认；明确配置 accept-complete 的 worker 才可自动确认。缓存 coverage=partial 时，本次 item 必须为 needs_review/uncommitted，展示 candidate_ref、coverage 与遗漏，不计入 completed；本次显式 accept_partial=true 后才可保存。不能继承其他 Principal 或上次尝试的部分提取接受决定。同一次已成功提交的同键重放保留原决定。 + +reparse=true 必须重新提供原文件并记录实际执行；运行诊断属于临时任务历史，Revision 保存影响内容的 provenance。解析缓存可以指向历史版本，但缓存指针不改变 Head；最近完整解析结果可更新缓存索引，旧版本继续精确读取。 + +### 原子保存边界 + +一份文档的事务包含：锁定会话、操作/成员及必要的来源绑定/Head、当前授权与状态校验、幂等校验、固定 Scope 的配额结算、全部 ContentSource 与 journal position、处理器 durable pending/dirty、ImportedContent Revision、Head CAS、精确有序 lineage、输入映射、item 提交结果及请求回执。reused 分支不新增分段，但仍原子记录本次接受及回执。任何故障全部回滚。 + +Source metadata 中的目标 Revision 在事务内确定，但不进入摘要；验证 parts 顺序、manifest 与 lineage 一致。不可先逐段调用独立 Source HTTP API 再声称整文档原子性。SQLite、embedded seekdb 与 OceanBase 均采用数据库事务和唯一约束/CAS，不能仅依赖进程锁。本期不自动清理持久文档版本及其证据。 + +现有 record capture 会自行开启事务；导入不能循环调用它，再单独保存 Artifact。实现应抽取或复用接受同一 connection 的内部保存单元，由 ContentImportApplication 唯一持有外层事务，保持以下顺序与职责： + +1. 在事务内验证授权、会话/item 状态、请求回执、目标治理状态、来源绑定和 expected_head_revision；按全局一致的锁顺序获取提交所需锁，确定将写入的 Revision。 +2. 校验冻结 bundle、manifest 及全部普通 ContentSource；逐段调用 `SourceRepository.add_with_status(connection, ...)`，取得精确 SourceRef 与 journal position。该 Repository 已在新增 Source 的同一事务内执行处理意图 `mark_dirty`，导入沿用此唯一职责，不另外重复维护一套 dirty 状态。 +3. 对 `created=True` 的分段，沿用现有 capture 的配置规则，为配置的 Source processing bindings 登记 durable pending；对已存在且身份/内容相同的 Source 不重复触发。共享保存单元必须保留 record capture 的既有行为，不能只调用 Repository 后遗漏 pending，也不能给 ImportedContent 本身注册生成消费者。 +4. 以实际写入结果构造有序 parts 和 `lineage.sources`，调用接受该 connection 的 `ArtifactRepository.create/revise`；在同一事务内完成 Head CAS、配额、来源映射、item 结果和幂等回执。任一分段、pending、Artifact 或回执写入失败，全部回滚。 +5. 提交成功后才发送唤醒通知、返回已保存结果；通知丢失仍由已提交的 pending/dirty 恢复。`reused` 和请求重放不再新增 Source、推进 Head 或重复登记消费。 + +锁与 CAS 失败按第 7 节的冲突和重放语义处理，不以第二个事务补写前一事务遗漏的正文或待处理状态。后台生成只能观察到整份文档已提交的分段集合；其具体消费窗口仍由各处理器决定，不承诺一轮生成消费全文。 + +## 8. HTTP、客户端与页面契约 + +持久资料复用标准 Artifact API，family 固定为 `importedcontent`。以下四个读取操作扩展现有 Family 注册与 schema，不新建独立资料查询资源： + +| 现有 operationId | 方法与路径 | ImportedContent 行为 | +| --- | --- | --- | +| `list_artifacts` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent` | 当前可见 Head 的资料列表；支持 namespace 精确过滤 | +| `get_artifact` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent/{artifact_id}` | 当前已接受快照及有序 parts | +| `list_artifact_revisions` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent/{artifact_id}/revisions` | 已接受版本历史 | +| `get_artifact_revision` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent/{artifact_id}/revisions/{revision}` | 精确不可变快照，回执 Location 指向这里 | + +导入是 Family 专用写入行为。会话创建前必须选择已有 Scope，准备和确认通过 session_id 使用该固定目标；会话路由用于恢复同一批次操作，不另外创建可长期管理的 file-import 业务资源。下表操作均为本 RFC 拟新增契约，不把 operation 恢复查询声称为仓库已有接口。 + +| operationId | 方法与路径 | 含义 | +| --- | --- | --- | +| `get_import_capabilities` | `GET /v1/import-capabilities?scope_id=...` | scope_id 必填且须已存在；返回固定 Source 类型 content、格式、权限与限额 | +| `create_import_session` | `POST /v1/import-sessions` | 必填已有 scope_id；校验贡献权限并固定目标及创建者 | +| `list_import_sessions` | `GET /v1/import-sessions` | 当前创建者可恢复的会话，分页 | +| `get_import_session` | `GET /v1/import-sessions/{session_id}` | 固定 scope_id、期限与回执摘要 | +| `import_content_text` | `POST /v1/import-sessions/{session_id}/importedcontent/texts` | source_kind=content；同步校验并确认保存有界正文，201/200 | +| `prepare_imported_content` | `POST /v1/import-sessions/{session_id}/importedcontent/prepare` | source_kind=content；metadata + 单文件，耐久接收后 202，返回 operation_id | +| `list_import_operations` | `GET /v1/import-sessions/{session_id}/operations` | 创建者的操作恢复列表，非资料列表 | +| `get_import_operation` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}` | 执行状态、错误、期限、计数与可执行动作 | +| `list_import_operation_items` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}/items` | 有界成员清单；单文件也有一个 item | +| `get_import_operation_item` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}/items/{item_id}` | manifest、coverage、warnings、omissions 及冻结目标 | +| `get_import_operation_part` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}/items/{item_id}/parts/{ordinal}` | 创建者的有界分段预览；已提交时从 Source 读取 | +| `confirm_imported_content` | `POST /v1/import-sessions/{session_id}/importedcontent/confirm` | operation_id + item_id + bundle_digest + accept_partial，原子确认，201/200 | +| `select_imported_content` | `POST /v1/import-sessions/{session_id}/importedcontent/selection` | operation_id + 成员集合及各自目标；冻结选择后 202 | +| `retry_import_operation` | `POST /v1/import-sessions/{session_id}/operations/{operation_id}/retry` | 调用所属 Family 的受控重试动作，重新准入后 202 | +| `cancel_import_operation` | `POST /v1/import-sessions/{session_id}/operations/{operation_id}/cancel` | 封闭未提交写入，返回原子取消/已完成结果，200 | + +prepare 的 202 响应包含 `{operation_id, session_id, family: "importedcontent", expires_at, status_url}`，Location 指向操作状态 URL，不指向尚不存在的 Artifact。texts 的请求本身即确认；它与 confirm 返回相同的已保存回执，不要求额外创建解析操作。操作及成员不能通过通用 PUT/PATCH 改写状态；确认/选择处理器须校验 operation 与 session、创建者、Family 匹配，跨会话或不可见句柄按现有 404/403 策略处理。通用 Artifact POST/PUT 对 importedcontent 的直接写入按本提案的 Family 能力校验返回 422，不能退回通用命令 Source 路径。 + +创建会话请求为 `{"scope_id":"已有 Scope 的 ID"}`，scope_id 必填、非空且遵循现有 Scope ID 约束。缺失、空值、未知字段或任何创建 Scope 参数返回 422；目标不存在、不可见或无贡献权限时按现有 404/403 可见性规则拒绝，不建立会话或接收文件。成功响应包含 session_id、固定非空 scope_id 与 expires_at。所有写操作携带 Idempotency-Key；不提供修改会话目标的接口。 + +文档目标为 `{"mode":"create"}` 或 `{"mode":"update","artifact_id":"...","expected_head_revision":2}`。手动默认 create，不从 origin 猜测身份；update 必须属于会话固定 Scope 的 importedcontent Family；其他 Scope 的目标按现有可见性规则拒绝。来源采集使用第三种互斥目标 `{"mode":"source","source_authority_id":"...","source_key":"..."}`，仅允许已注册并获准写入该 authority 的 Connector Principal 使用；接收时解析绑定，冻结 artifact_id 和预期 Head/不存在断言。响应返回已解析目标，不能由任意客户端通过 origin 冒充 authority。容器对每个成员独立指定目标，禁止把多个成员更新到同一 artifact_id。 + +直接粘贴请求保持简单： + +```json +{ + "title": "接口约定", + "artifact_target": {"mode": "create"}, + "source_kind": "content", + "media_type": "text/markdown", + "origin": {"namespace": "project-notes", "key": "note-001"}, + "text": "# 接口约定\n\n分页使用不透明游标。\n" +} +``` + +上传 metadata 包含 artifact_target、title、origin、parser_profile，以及可省略且固定为 content 的 source_kind、可选 encoding、as_text、reparse、ocr_mode、language、include_notes、include_comments、include_hidden 与 archive_depth。as_text=false、reparse=false、ocr_mode=off、include_notes=true、include_comments=false、include_hidden=true、archive_depth=1 为默认值；language 不指定时使用 profile 公布的固定语言集，encoding 不指定时按识别规则决定或返回 encoding_required。parser_profile 不指定时先冻结部署的 router/profile 注册表版本，再按识别出的格式选择并冻结对应默认 profile；PDF 的实际后端随后按该 profile 的固定规则解析。归档/邮箱同时冻结子文档路由注册表，成员发现与选择后按实际格式使用同一版规则，不能在中途采用更新的默认值。跨键重试客户端应传已解析的 profile digest。include_notes 仅对支持备注的格式生效，其他格式默认值忽略，不适用的显式非默认值返回 422。archive_depth 取 1 或 2。所有默认值均在能力接口公布。 + +as_text=true 只允许通过严格文本校验的输入，不能绕过二进制识别与资源限制；ocr_mode 取 off/auto/force，auto 仅处理需要 OCR 的页面，force 在预览中标明。reparse=true 通过新上传与新 Idempotency-Key 发起新尝试，客户端需重新提供原始文件；服务端不假定旧暂存尚在，也不改变旧冻结结果。附件由下面的成员选择操作决定。 + +selection 请求 `{operation_id, expected_selection_version: 0, items: [{item_id, artifact_target}, ...]}`，仅在成员发现完成、选择未冻结且任务有效时接受；ID 必须来自同 operation 完整清单且可选、去重、非空、最多 1,000 个。服务端校验各成员目标、禁止同一次选择中重复指向同一 Artifact，原子冻结为版本 1 并排队；再次改变选择返回 409，未知/其他任务 ID 返回 422 且不透露外部内容。单文档无需该操作。retry 请求为必填 stage 的互斥结构：`{stage: "discovery"}` 或 `{stage: "documents", item_ids: [...]}`。discovery 分支禁止携带 item_ids，仅允许 discovery_state=failed、discovery_retryable=true、完整清单尚未发布且原始暂存仍有效的容器任务；接受后转为 discovery_state=pending、phase=discovery、execution_state=queued。documents 分支要求清单已完成或无需发现，item_ids 非空、去重、最多 1,000 个，只允许已选、失败且 retryable 的文档；接受后将指定文档改为 pending 并排队。非法结构返回 422，阶段或状态不匹配整请求返回 409，均不修改冻结结果。两个分支共用配额重新准入规则，只有原子获得所需任务槽且写入队列与幂等回执后才返回 202。cancel 请求空对象；第一次原子取消，重复取消返回相同回执；已 completed 返回原完成状态，已 expired 返回 tombstone，不删除 ImportedContent Revision。 + +上述三个写操作均要求创建者身份，以及会话固定 Scope 的 scope.read + scope.contribute，并携带 Idempotency-Key；重放保持原接收回执，可用 GET 查询后续状态。管理员代操作遵守第 10 节；selection/retry 对 cancelled 返回 409、expired 返回 410。操作响应返回 operation_id、session_id、family，并独立返回 phase、discovery_state、discovery_retryable、阶段错误、execution_state、aggregate_status、task_slot_held、selection_version、counts、expires_at、idempotency_expires_at 和 blocked_reason；文档响应返回 selected、parse_state、commit_state、retryable、coverage、warnings、omissions、bundle_digest，以及可选 candidate_ref 和 artifact_ref（均为含 scope_id 的精确 ArtifactRef）；artifact_ref 仅用于本次已成功确认的回执，candidate_ref 不表示本次已保存。客户端不推断混合 status。 + +每个上传只接收一个 file 和一个 metadata 对象；metadata 上限 64 KiB，未知字段拒绝， multipart 额外字段及文件拒绝，不能绕过单文件上限。任务查询与文档 manifest 不回传原始 blob 地址。 + +确认请求为 `{operation_id, item_id, bundle_digest, accept_partial}`；目标文档及 expected_head_revision 已在 item 上冻结，响应同时回显。请求携带 Idempotency-Key。当前 bundle 已变化返回 409，未提交文档过期返回 410，取消或未解析完成返回 409。已提交文档在任务映射保留期内重放提交返回耐久回执，即使任务随后到期；此后通过通用 Artifact 精确版本读取接口读取。ready 只允许 accept_partial=false；needs_review 必须为 true，否则返回 409。成功返回 `{outcome: created|reused, scope_id, artifact_ref, snapshot_digest, parts}`；created 包括首版及后续新 Revision，artifact_ref 携带 family=importedcontent、artifact_id、revision，Location 指向现有通用 Artifact 精确版本资源。新的导入响应统一使用 `parts[]`，粘贴正文为一段；不改变既有 Source API 的 response。 + +任务/成员/资料列表默认 limit 50、范围 1–100，SQL keyset 分页并冻结首请求高水位；cursor 绑定操作、调用方、稳定 session_id（会话路由）或 scope_id（资料路由）、过滤、limit、高水位及 TTL，每页重新鉴权。任务状态可随时间变化，列表不是状态快照。ImportedContent 资料列表复用 Artifact 列表并扩展 namespace 精确筛选投影，不伪装成原文全文搜索。 + +Python client 暴露对应操作；CLI 提供 text、files、resume、cancel 和 dry-run；新导入必须用 --scope 指定已有目标，缺失时在上传前报错；--session 可恢复此前已选择目标的同一批次，并显示会话固定的 Scope。若同时提供 --scope，必须与会话一致，否则在提交前拒绝；用 --artifact-id 与 --expected-head-revision 明确更新现有文档，并暴露 --as-text、--reparse 与成员选择参数。容器先列出成员并提交明确选择；无人值守必须配置成员选择规则。dry-run 只验证本地可确认事项；离线未运行解析器时必须将文件标为“需服务端解析”，不能宣称可完整导入。CLI 默认在当前部署解析后交互确认；显式 `--accept-complete` 可自动保存 ready 文档，partial 必须额外显式授权，命中已有 partial 快照同样适用，禁止默认悄悄接受。页面和 CLI 根据 phase/discovery_state 提供“重试成员发现”或“重试失败文档”，不要求用户为尚未发现的成员填写 item_ids；收到配额拒绝后按 Retry-After 退避，不将未接收请求显示为已排队。 + +上传任务的重新打开与预览为读操作；重新登录不自动提交。确认按钮携带 frozen digest 和原 key，结果不明时通过任务与回执核实或同键重试。上传进度、解析进度和 Source 保存进度分开表示；解析器不报告页进度时只显示处理中。 + +版本冲突不自动改变冻结目标。调用方读取当前 Head 后，以新的 expected_head_revision、原文件和新幂等键创建新的文件导入尝试；可命中已有解析缓存，但旧 item/回执不可改写。页面展示当前版本差异后再确认。通用 Artifact 写接口不得绕过导入服务任意改写 ImportedContent 的 provenance、parts 或 lineage。 + +### 已发布正文、历史与比较 + +复用现有 Dashboard 的历史分页、Revision 选择及比较页面；为 ImportedContent 增加读取投影，不另建一套版本系统。历史读取不依赖已经过期的 import operation 或 staging 文件，而是按精确 Revision 的 parts 批量读取对应 ContentSource，保持 ordinal 顺序并验证缺失/摘要异常；不能用当前 Head 正文填补旧版本缺段。 + +比较默认呈现正文差异，以及 origin、parser/profile、OCR、coverage、omissions、accepted_partial 等有意义的元信息变化；SourceRef、journal position、摘要和保存时间只作溯源详情，不能用其 JSON 变化代替正文变化。正文按分段延迟加载并设置比较字符数、输出与计算预算；超出预算显示“比较未完整展示”及分段入口，保留完整精确原文，不在后台截断后声称无差异。历史页面不显示恢复、回滚或合并按钮，直接调用相应 API 也不能成功。 + +本期不提供 ImportedContent 正文搜索，既有 `/v1/scopes/{scope_id}/artifacts/{family}/search` 对此 Family 返回统一的不支持结果；列表的 namespace 精确过滤仅查询元数据。后续若增加关键词或语义检索，应在现有 ArtifactSearchService 注册符合契约的 `ArtifactSearcher`，补齐 Scope 权限、版本选择和索引更新规则，不另建重叠搜索 API，也不因导入成功就承诺向量索引已就绪。 + +若使用既有稳定 Artifact handle，只能绑定某个精确不可变 Revision；它不代表随 Head 更新的外部文档身份，不替代第 3 节的稳定来源绑定。 + +### 使用举例 + +1. 粘贴一段 Markdown：先选择已有 Scope，以 {"scope_id":"项目 Scope ID"} 创建会话后调用 importedcontent/texts，source_kind=content。成功返回精确 artifact_ref；页面通过标准 Artifact get 与 Source parts 阅读，操作失败不留下半份制品。 +2. 上传 PDF：调用 importedcontent/prepare，source_kind=content；返回 operation_id 后查询解析状态及分页预览。确认时提交 operation_id、item_id、bundle_digest 和 accept_partial；成功后读取 family=importedcontent 的 Revision。PDF 页数和 OCR 只属于提取元数据，不产生新 Family。 +3. 更新已有资料：prepare/texts 中 artifact_target 显式填写 artifact_id 与 expected_head_revision。解析期间仍可读取原 Head；确认成功才产生新 Revision。失败或取消不创建失败版本，也不改治理状态。 +4. 上传 HTML：source_kind=content，input_media_type=text/html。即使 origin 中附有来源网页说明，也不会触发网络读取；来源说明不改变 content Source 类型。 +5. 将来接入网页：经独立 Connector 验收后，由获准适配器提供冻结页面快照，归一后的文本仍使用 source_kind=content，复用同一 ImportedContent 提交服务与 Artifact 阅读契约。本期未提供链接采集接口，页面不展示可用的链接导入入口；仅填写 source_kind=content 不会启用网络抓取。 + +## 9. OpenDAL 持续采集 + +新增 `opendal-text-imports` 模式，保留原 `opendal-text-files` / `text-file-snapshot` evaluation 的 Definition、名称、权限和 checkpoint 行为。新模式通过公开导入 API 提交，不调用每次会注册 manifest 的旧 RemoteConnectorWorker 路径，向已有 Scope 导入无需 `server.admin`；worker 必须配置已有 scope_id，并具备 scope.read + scope.contribute；缺失目标或权限时拒绝运行，不自动创建 Scope。worker 使用同样的 ImportSession 协议。 + +worker 使用相同能力矩阵和 parser profile。一次扫描获取完整且有界的选中文件清单，实际读取原始字节并计算 input_digest。mtime/ETag 可辅助诊断，不能单独证明未变。逻辑文档身份是 Scope + source_authority_id + 相对 path/object key。profile、include/exclude、编码或 OCR 策略变化需显式创建新的采集配置 binding/checkpoint，但沿用同一 authority 下的文档身份，变化内容提交为新 Revision；root/桶变化则是新 authority。显示 namespace 不能代替持久来源身份，checkpoint 配置 fingerprint 也不能成为文档身份。 + +checkpoint 只保存已完成输入身份、原始摘要、解析 profile 与对应含 Scope 的精确 ImportedContent ArtifactRef。稳定 input_key 由 binding、输入身份、原始摘要、profile 及完整选择/解析策略计算，用于匹配输入结果，不直接作为每次上传键。 + +worker 在独立于成功 checkpoint 的耐久恢复表中保存 `(binding, input_key, session_id, attempt_uuid, operation_id, expected_head_revision, idempotency_expires_at, retry_after)`；上传键由 input_key 与 attempt_uuid 派生，先以事务/CAS 写入 attempt_uuid 再发送请求。响应丢失或 worker 崩溃时恢复同一尝试和键。多个副本竞争同一输入需复用同一持久尝试；不同服务 Principal 不共享私有任务恢复表。 + +查询得到 expired tombstone 时，先保留其中已提交回执，再以 CAS 分配新的 attempt_uuid 与上传键重新提供文件；旧 session 到期时为原有 scope_id 创建新会话并持久记录 session_id；超过幂等保留窗口且无法恢复旧任务时同样新建尝试。新尝试仍使用稳定输入映射复用已冻结解析结果，并核对当前 Head 与预期版本；已成功提交且待恢复的原始回执不重复提交，未保存成员继续解析，不因固定键永久卡在 410。容器发现后 worker 依据明确的选择规则提交稳定成员集合；只有本轮期望成员均有耐久回执才完成输入。恢复表推进不代表成功 checkpoint 推进。管理员或用户主动取消的输入标记为 paused，后续扫描不自动重建任务；只有明确恢复该输入后才以新 attempt key 继续。 + +worker 自动确认的范围必须在部署配置中明确，只接受 coverage=complete 的文档。此校验覆盖新解析结果、已有快照候选及恢复回执：即使存在 ImportedContent Revision，coverage=partial 也不能满足 complete-only，必须标为需处理且禁止据此推进 checkpoint。worker 不代表用户接受历史 partial。发现阶段可重试失败通过 stage=discovery 恢复，文档失败通过 stage=documents 恢复；配额拒绝保留旧状态并持久退避。扫描页缺 OCR、损坏附件、选中成员不支持或部分提取时标为需处理，不自动接受 partial。每个被配置选中的容器成员都获得已保存/已存在回执且整轮 listing 完整，才 CAS 提交 checkpoint。仅创建 operation、解析 ready 或上传成功都不够。 + +其他正常文件可继续提交；失败项不回滚它们，但旧 checkpoint 保留。恢复时在有效尝试内同键查询与重试，过期后更换尝试键并复用已保存结果。并发 CAS 冲突重新加载并扫描，不强行覆盖。删除外部路径只影响后续比较，不删除 ImportedContent、历史 Revision 或 Source；不完整 listing、permission denied 或临时连接失败不能作为删除证据。 + +尽可能固定外部 immutable revision 或使用 conditional read；无法提供这一能力时，仅承诺处理本次实际读取字节，不把时间戳包装成精确外部 revision。存储凭据只存在于 worker，不进入 Source、checkpoint、日志或 parser。 + +## 10. 安全、权限与隔离 + +所有文本/文件导入都要求已选择 Scope 的 `scope.read` + `scope.contribute`；文档列表、历史 Revision 与 Source 阅读要求 `scope.read`。普通导入不需要 `server.admin`，导入服务不执行 Scope 创建或权限初始化。会话和文件任务绑定创建者稳定 Principal；其他普通调用方不能读取未提交预览或代其确认。管理访问须显式 server.admin 且具备目标 Scope 的对应权限，所有代操作审计。每次确认重新鉴权；创建者撤权后不得凭旧会话写入。 + +Parser worker 只获得调度授予的指定 operation、staging blob 和 profile，不获得任意 Scope 写入权限；最终保存由 ContentImportApplication 以确认调用方权限执行。后台解析可以完成已接受输入，但不会自行生成 Source。撤销权限后禁止确认提交，暂存按到期规则清理。 + +Dashboard 所有写路由做同源 Origin 校验，使用现有会话身份转发公开 API；保持 HttpOnly/Secure/SameSite 和 cookie 路径规则，不直接写库、不向 `/v1` 扩展 cookie 权限，不信任未经部署认可的代理头。 + +解析器运行在受限独立进程/容器，限制 CPU、内存、时间、文件数和像素；默认禁用外网，禁止 XML 外部实体、脚本、Office 宏、公式重算、链接更新、Notebook 执行与动态插件安装。用户文件名不用于任意磁盘路径。原始内容与解析文本都视为资料，不作为执行指令。 + +归档拒绝路径穿越、绝对路径、符号/硬链接、设备文件及重复覆盖，实际计量展开预算；不得仅依赖压缩包索引所写尺寸。密码文件不接收解密口令，本期返回 encrypted_document；DRM 不绕过。 + +OCR 默认采用本地引擎和已安装语言包。云 OCR 或视觉模型需要部署显式启用及用户可见的外发说明，不自动将资料传给未知 provider。普通文档提取不应隐式调用生成式模型；OCR、视觉模型与下游 Artifact 用量分开统计。 + +## 11. 错误与可观测性 + +保留既有 error envelope，并区分 unsupported_format、parser_unavailable、capability_insufficient、encoding_required、invalid_encoding、encrypted_document、damaged_document、ocr_required、no_extractable_text、parse_timeout、input_limit_exceeded、output_limit_exceeded、archive_limit_exceeded、partial_extraction、idempotency_conflict、revision_conflict、source_binding_conflict、session_expired、bundle_conflict、selection_conflict、operation_cancelled、operation_expired 与 quota_exceeded。 + +HTTP 401/403/404 沿用可见性策略；非法字段 422、单请求硬上限 413、不支持媒体类型 415、幂等/状态冲突及已取消操作 409、过期的未提交内容操作 410、限流 429、依赖不可用 503。202 后发生的解析失败记录在 operation 中，通过 200 状态查询返回 failed 与错误明细，不能伪造先前上传请求的 HTTP 错误。网络超时始终可能意味着结果未知，同键核实。 + +日志不保存正文、token、完整本机路径、Prompt 或 OCR 图像。指标区分上传字节、识别类型、解析结果、OCR 页数、需确认项、已提交文档/分段、重复命中、解析延迟、提交延迟、checkpoint 冲突与最后成功完成时间。高基数 ID 不作为 metric label。 + +“已保存”只统计取得耐久文档回执的资料；压缩包总览分别显示已保存、失败、未选择、待处理数量。只提取元数据或空正文不算成功。后续生成成本与进度来自处理器自身证据，不能从解析成功推断。 + +## 12. 兼容、实施与验收 + +### 实施边界 + +交付采用第 1 节固定的解析族与分支，不再将“确定每种格式使用何种解析器”留给运行环境选择。交付按职责推进，但 A、B 格式范围都是正式目标：先实现文本保存与统一 manifest，再实现 Family prepare/confirm、内部耐久操作账本和标准解析包，再完成 OCR/长尾适配及 OpenDAL。不能仅完成 A 就宣称整个多格式 RFC 已实现。 + +注册 importedcontent Family、补充操作账本及查询索引，不改变既有 Source/Artifact 引用。历史 Source 不自动包装为 ImportedContent Revision。更新 family allowlist、精确读取、历史、lineage 及 Dashboard 阅读器；不得把 ImportedContent 自动纳入模型生成或 Candidate 流程。旧 OpenDAL binding 不原地换新模式；显式重新导入可生成新的 content Source，不能声称保留旧类型身份。Core Python 支持范围保持不变，较重解析依赖由独立 worker runtime 承担。 + +### 数据库初始化、升级与恢复 + +导入的所有持久结构必须同时覆盖 SQLite、embedded seekdb 和 OceanBase。第 3 节的三张核心操作表，加上实际采用的来源绑定、回执、配额账本及查询索引，都纳入受版本管理的 schema；不能把辅助表留给首次请求临时建表。持久资料仍复用 Artifact/Source 表，不新增 ImportedContent 主表或版本表。 + +现有 `server db-migrate status/plan/apply/verify` 提供迁移基础，但当前注册 bundle 仅管理有限表集，不能把它视为已经覆盖全部业务表。实现须在该机制中登记导入 schema bundle/revision、冻结表/索引 manifest 和后置校验，并明确与现有 bundle 的依赖及兼容版本，不直接修改已有发布 revision。 + +| 项目 | 必须落实的约束 | +| --- | --- | +| 空库与已有库 | 空库初始化和已有部署升级得到相同 schema；升级前核对受管表、索引、历史 revision 和冲突残留，不把任意已有表自动标记为已迁移 | +| 唯一性与查找索引 | 保证会话/操作主键、`(operation_id,item_id)`、稳定来源身份、幂等键分区唯一；按会话创建者、Scope、状态/租约到期、TTL 建有界恢复与清理索引；长 key 按后端索引限制采用有碰撞校验的摘要索引，不静默截断身份 | +| 升级门禁 | API 写入、解析调度、Connector 提交与 GC 均遵守迁移维护门禁；schema 未就绪时不接收导入，不能让一部分副本按新结构写入、另一部分继续按旧结构执行 | +| DDL 中断恢复 | SQLite 按后端能力使用事务；seekdb/OceanBase 的 MySQL 兼容 DDL 可能隐式提交，不能承诺整次 schema 升级可事务回滚。记录步骤和校验结果，中断后先 inspect/plan，再恢复或人工处置，不盲目重复非幂等 DDL | +| TTL 与运行数据 | 清理会话、操作、回执和暂存引用的期限符合第 6、7 节；保留恢复所需 tombstone/回执。稳定来源绑定、已发布 Artifact、Source 和仍被引用的映射不能随操作 TTL 一起删除 | +| 降级与发布 | 功能开关可停止新导入，但不删除持久资料；禁止未经验证的 schema downgrade。注册 bundle 的 verify 成功只证明其受管结构满足后置条件，不替代整库、权限与服务端到端验收 | + +迁移交付必须包含三种后端的空库、已有数据升级、重复执行和中断恢复验收；检查既有 Source/Artifact 与精确引用不变。迁移脚本、manifest、索引及初始化路径必须随功能一同落地，禁止仅在开发 SQLite 自动建表后宣称支持生产后端。 + +### 契约与验收 + +实现以 `openapi/powercontext.yaml` 为契约源，执行 `make api-generate`、`make contract-test`，不手改生成代码；运行 `make check`、相关 Source/Server/Access/Dashboard/Connector 测试及 `make docs-test`。影响 Python 支持范围时运行 `tox`，跨组件验收放入 `tests/e2e/`。本 RFC 被接受不表示验收已通过。 + +| 场景 | 验收要求 | +| --- | --- | +| Scope 授权的完整读写链路 | 仅具有 scope.read + scope.contribute 的用户可完成导入并读列表、详情、历史及正文；无需 Artifact owner 或 Prompt 管理员权限；撤权后所有对应路径失效 | +| 通用恢复/合并、直接 POST/PUT、retired 目标 | API 与 Runtime 同时拒绝，不产生孤立 Revision 或丢失 lineage.sources；页面无恢复/合并入口，导入不复活 retired 目标 | +| 普通 content 的角色与生成准入 | working_note=False、internal=None、handoff_receipt=False；伪造保留 metadata 被拒绝；符合既有准入规则的分段进入已配置处理器 | +| 原子保存、幂等重放与通知丢失 | 分段、dirty/pending、Artifact、Head 和回执共同提交或回滚；重放不重复触发，提交后丢失通知仍可恢复处理 | +| 三后端 schema 初始化/升级/中断恢复 | SQLite、embedded seekdb、OceanBase 的表/索引约束一致；已有数据保持可读；DDL 中断不误报 ready，多副本遵守维护门禁 | +| 静态契约、运行能力与 Scope 级发现 | feature contract 不等于启用；普通导入者无 server.observe 也可查目标能力；部分解析器故障只影响相应 profile，关闭新导入后历史仍可读 | +| 版本正文比较与搜索边界 | 按精确 Source 比较正文和语义元信息，超预算明确提示，历史不依赖 staging；未注册 searcher 时明确不支持，namespace 不冒充全文搜索 | +| 必选已有 Scope、缺失/无效目标 | UI 禁止开始导入；API 缺失或非法参数返回 422，不存在/不可见/无权限按 404/403 拒绝，不接受输入或建立操作 | +| 固定会话目标、切换 Scope、跨 Scope 请求 | 会话 scope_id 非空且不可修改；子操作不能覆盖目标，更新资料和来源绑定均校验归属；切换目标使用新会话 | +| 普通导入权限、上传后撤权或目标失效 | scope.read + scope.contribute 足够，无需 server.admin；确认重新校验，失效后不写入或自动补建目标 | +| 多会话累计配额与并发提交 | 从接收起计入同一目标 Scope，多个会话不绕过累计限额；预留/释放原子且无额度迁移 | +| CLI 与 OpenDAL 目标配置 | 新导入缺少已有 scope_id 时在上传前拒绝；恢复会话保持原目标，同时传入冲突目标时失败 | +| 同名不同文档、同文档改名、独立新建同内容 | 文件名/摘要不决定身份；明确更新保持 artifact_id，独立新建不合并 | +| A→B→A、旧请求重放、Head 并发冲突 | 回归内容生成 R3；旧键返回旧回执不回退 Head；冲突 409 不覆盖 | +| parser/filter 变化、路径重命名、容器成员重排 | 配置变更不误换文档；无稳定 ID 的重命名/成员变化不猜测关联 | +| ImportedContent 元信息、parts、lineage 与反查 | 精确版本/分段一致；不重复正文、不伪造全文消费或预填下游关系 | +| ImportedContent 类型与 importedcontent wire name | 持久化、Family 注册、OpenAPI、客户端和 Dashboard 一致；标准四个读取操作可访问 | +| 统一 content Source 与不同输入方式 | 粘贴文本、上传 HTML/PDF、文件成员及 OpenDAL 输入均保存 source_kind=content；读取分段的 SourceRef.source_type 一致，网页获取仍需独立 Connector | +| source_kind 固定值与请求归一 | 省略字段与显式 content 的请求指纹、摘要及缓存语义一致；其他值返回 422；不新增恒定字段的筛选索引 | +| 跨输入方式更新与历史版本 | 文件改为粘贴仍校验 Head 和完整快照语义；各版本 source_kind 均为 content，具体 origin、格式和 parser 独立保留 | +| 新资料预览与已有资料更新中 | 新资料无 Artifact/Source 可见；旧 Head 在解析、失败、重试、取消后保持不变，只有确认成功推进 | +| 通用 Artifact 写入与内部状态篡改 | POST/PUT 不能伪造 ImportedContent；operation 无任意 PUT/PATCH 状态入口，跨会话句柄被拒绝 | +| 私有操作与 Scope 普通读者 | 同 Scope 的其他读者不能读取/确认待提交预览；只能通过 Artifact API 读已接受内容 | +| 操作清理和治理状态 | 清理后 Artifact/Source 精确版本仍可读；进度不产生 Revision，不把 parsing/failed 写入治理状态 | +| 默认路由、后端不可用和一次备用解析 | 冻结 router/profile；不隐式开启 OCR/外发；备用共享预算，冻结后不得覆盖 | +| LiteParse 页数上限、page_errors、复杂度与 OCR | 不把探测信号当执行，不把缺失页当完整；复杂布局使用固定 Docling 分支 | +| XLSB 完整 profile 与仅值 profile | 公式契约不足保持 planned；仅值须显式选择且 needs_review，complete-only 不接受 | +| OOXML 模板/宏变体、ODF/Flat XML、重复行列 | 原容器识别和关系正确；不靠改后缀，不执行宏；节点及展开预算先于读取生效 | +| Office 源码审计与转换定位 | 修订/脚注/隐藏/备注符合选项；不将转换页号映射成不存在的原生位置 | +| 外部服务任务、预签名资源与临时图片 | 实际版本与外发可追溯；保存文字不依赖过期 URL，不承诺原图长期在线 | +| 每种正式宣称支持的格式与格式代际 | 至少有可核对正文、定位与遗漏的真实夹具；只识别扩展名不通过 | +| TXT/代码/配置、UTF-8/UTF-16/GB18030/Big5 | 严格解码、中文与行号正确，错误编码不静默替换 | +| PDF 文本页、扫描页、混合页、多栏和表格 | 页序、OCR 去重和遗漏正确；缺 OCR 不误报完整 | +| Word 表格/脚注/修订/批注 | 符合明确视图与选择，不能静默丢弃所选内容 | +| XLS/XLSX/XLSB、多表、隐藏行列、公式缓存缺失 | 不只首表、不重算、不虚构值，位置可核对 | +| PPT/PPTX/模板、备注、隐藏页 | 顺序与纳入选项正确，不执行宏 | +| HTML/Notebook/EPUB/EML/MSG/MBOX | 无外链抓取或执行，正文选择和子文档边界正确 | +| OCR 图片、多页 TIFF、超大像素与空白页 | 限额生效，元数据不能代替正文，结果可追溯 | +| Office/EPUB 容器与普通 ZIP/TAR | 正确区分；归档路径穿越、嵌套与解压炸弹被限制 | +| 扩展库未安装或仅返回 metadata | capability 不报 available，不制造空 Source | +| 长文、多分段、超限和单段极长 | 拼接 digest 等于完整提取结果；超限整文档失败 | +| 上传后断线、worker crash、旧租约结果返回 | 原 operation 可恢复，旧 generation 不能覆盖冻结输出 | +| OCR 预览后重试、确认并发、profile 升级 | 确认固定 bundle，相同输出复用，新输出保留旧证据 | +| 保存前后摘要、不同 SourceRef/时间戳 | 内容摘要不受保存回执影响，同一文档目标与预期 Head 的并发提交至多一个成功推进版本 | +| 同 Scope 两个 Principal 上传相同内容 | 不复用对方私有 operation；独立新建仍是不同文档；更新同文档须校验权限与 Head | +| 任务过期及幂等映射清理后 worker 恢复 | 新 attempt key 可推进，已提交资料不重复，checkpoint 不提前更新 | +| 发现、冻结选择、部分提交、取消与 GC 竞争 | 状态字段与聚合规则一致；已提交不回滚，未提交不会越过取消/到期 | +| as_text、reparse、附件选择、retry、cancel | API/CLI/UI 契约一致；非法组合拒绝，同文档当前 Head 语义相同的重解析不重复 Source | +| 多副本、多调用端、配额耗尽与 worker crash | 累计预留原子生效，实际删除后释放存储，OCR 已用额度不退款，持久退避可恢复 | +| 用户 A 接受 partial，用户 B 或新的导入尝试命中同一快照 | 本次仍为 needs_review/uncommitted，显示遗漏并显式确认；同文档当前 Head 相同才 reused，独立新建生成新文档；同一次提交重放不重复确认 | +| OpenDAL 命中已保存 partial 或恢复 partial 回执 | complete-only 拒绝自动完成，checkpoint 不推进，不因存在 ImportedContent Revision 绕过覆盖检查 | +| 相同输入以新键上传，旧任务 expired/cancelled/failed/completed | 不合并到旧任务；新 operation 独立创建，持久记录按 coverage 规则复用；同旧键重放仍返回旧结果 | +| 成员发现阶段临时失败且自动重试耗尽 | 无 item_ids 也可 stage=discovery 重试；完整清单原子发布，已冻结清单及成员 ID 不变 | +| 已释放任务槽的 failed 或 partially_committed 重试 | 配额满时 429/503，状态和计数不变；额度恢复后同键准入只占一槽，不重复入队 | +| partial 候选等待确认、容器仅剩失败成员、单文档失败 | 待确认持续占槽；无活动工作的失败可释放任务槽；其他成员的活动槽与暂存计量不受错误释放 | +| 写入中任一分段/记录/pending 故障 | 整份文档回滚，恢复后不重复 Source | +| 容器部分成员失败、用户确认 partial | 逐文档结果真实，遗漏长期保留，不冒充完整导入 | +| 撤权、跨 Scope、跨站请求和恶意文件 | 不能确认写入、越界读取或执行内容 | +| worker checkpoint 与 profile/filter 改变 | 只对完整已提交结果推进，配置变更不静默跳过 | +| SQLite/embedded seekdb/OceanBase、多副本暂存、过期 GC 竞争 | 恢复与提交一致，GC 不破坏正在确认的 bundle | +| 中英文、明暗主题、小屏、关闭后恢复 | 支持清单、预览、遗漏、逐项重试和阅读全文可用 | + +# Drawbacks + +广泛格式支持显著增加依赖、样本和安全维护成本。Office 传统格式、复杂表格与各代专有文档的能力不一致，必须维护能力矩阵并保留失败原因，不能使用“支持任意文件”的笼统承诺。 + +解析 worker、耐久暂存、租约和分段事务比单份文本捕获复杂；OCR 的速度、准确率和语言覆盖也需要独立预算。保留提取文本和 digest 无法还原原始二进制的所有布局、图像与交互内容。 + +文档原子提交可能同时追加多个 Source，增加单事务成本和后续消费量。8 MiB/128 段等上限必须以三种数据库后端的验收控制，不能无限提高。归档允许部分成功，用户需要理解容器完成与单份文档完成的区别。 + +版本模型保留旧新文本；Head 只表示最近接受的版本，不提供外部实时最新事实选择或删除传播。一次解析正常完成也不能证明模型能准确理解其中所有内容。 + +# Rationale and alternatives + +**导入资料使用 ImportedContent Artifact + Revision。** 复用既有不可变版本、Head、lineage 和读取能力，比另建 ImportRecord 更容易提供版本历史与资料关系查询；代价是必须注册新 Family、提供专用写事务和阅读器，并补齐来源绑定与查询索引。source_kind 固定为 content，复用现有文本 Source；具体出处、文件格式和解析器独立记录。运行状态由内部可清理操作账本承载，不新增 FileImportJob 业务实体，不因每次解析进度创建 Revision。 + +**显式文档身份与固定已有 Scope。** 手动新建和更新资料明确区分，自动采集使用受控稳定来源键，避免同名/同内容误归并。导入前必须选择已有可贡献 Scope；会话只维护固定目标、私有预览和恢复，不负责创建 Scope、初始化权限或迁移配额。这使上传、确认和配额从开始到结束使用同一权限边界。 + +**原生结构与 PDF 布局分开。** Excel、演示、邮件和归档具有原生对象与成员身份，不统一转 PDF；PDF 采用 LiteParse 轻量分支和 Docling 布局/OCR 分支，复杂替代引擎作为独立 profile。接受多种依赖的代价，换取可说明的结构、定位与遗漏。RAGFlow/LlamaIndex 的任务、去重、索引与默认 Reader 不直接承担本 RFC 的保存事务。 + +**采用可插拔解析能力与固定路由。** 一套只接受 UTF-8 的接口无法覆盖 Office/PDF；一个通用解析器也不能保证所有格式的正文与结构。固定并验收每类适配器比依赖隐式 fallback 更可诊断，也允许未来增加长尾格式。 + +**隔离文件解析，复用 ContentSource。** 解析依赖和文件处理风险不进入 Core 请求进程；归一后的文本继续使用已有 Source、Scope、lineage 和 Artifact 处理，无需按每个后缀新增 Source family。 + +**文件异步解析、单份资料同步提交。** OCR 与大文件无法保证在普通 HTTP 超时内完成；Family prepare 返回操作句柄，内部耐久执行记录承担恢复责任。冻结结果后单文档事务可以可靠确认，避免边解析边公开部分 Source。 + +**分段保存而非截断或强迫用户手工拆文件。** 自然边界和定位保留可用性，精确拼接和全量摘要保护完整性。原子保存保证一份文档不出现仅前几页已消费的半提交状态，但不强制所有 Artifact 在同一轮消费整份文档。 + +**提取能力与“理解所有内容”分开。** 正文、表格、OCR 和元数据的覆盖不同。已知缺失必须报告，不能以推测补全文字或把元数据算成正文；用户接受 partial 的决定应留在记录中。 + +**不把库的支持列表直接当产品支持列表。** 产品可用性还取决于格式代际、依赖、选项、输出质量、资源限额与测试。能力发现只声明部署已完成的适配。 + +# Prior art + +本提案主要基于仓库中可核对的现有设计与实现，不依赖未经验证的竞品功能： + +- [RFC 模板](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/0000_example.md)与[RFC 流程](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/README.md)：本文的结构与编号规则。 +- [ContentSource 实现](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/builtin/sources/content.py)：captured text、metadata、内置文本投影与内部角色边界。 +- [关系型 Source 保存](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/builtin/persistence/records.py)：Source 与待处理登记的既有事务路径。 +- [OpenDAL evaluation](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/integrations/opendal/README.md)与[ConnectorLifecycle](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/sources/connectors.py)：快照采集、逐项结果与 checkpoint CAS。 +- [Dashboard 设计原则](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/development/dashboard.md)：Scope 边界、全文阅读、错误恢复与静态身份约束。 +- [Dashboard 补齐制品 RFC](https://yuque.antfin.com/obopensrc/knowledge_sharing/frkh86luw67rkcpt)：相邻页面规划与精确来源阅读；本文不改变其画像和交接导出契约。 +- [SourceRepository](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/builtin/persistence/sources.py)与[Source 生成准入](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/builtin/source_eligibility.py)：connection 内保存、dirty 登记及普通正文角色约束。 +- [Artifact 授权 profile](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/server/authz/profiles.py)与[Server 路由](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/server/app.py)：Family 授权与标准 API 接入边界。 +- [版本历史页面](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/server/dashboard/revisions.py)与[合并恢复实现](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/builtin/artifacts/merge_restoration.py)：复用读取和比较，专用证据模型不能直接启用通用恢复。 +- [Artifact Search RFC](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1858-artifact-search.md)与[Artifact Merge RFC](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1859_artifact_merge.md)：搜索注册、合并及恢复能力边界。 +- [数据库迁移操作说明](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/docs/operate/database-migrations.md)与[统一迁移 RFC](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1771-unified-database-migrations.md)：受管 bundle、三后端迁移与恢复限制。 +- [OpenAPI 契约](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/openapi/powercontext.yaml)与[ServerInfo](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/src/powercontext/server/info.py)：协议版本、feature contracts 和运行能力分离。 + +多格式解析参考以下官方资料；其能力用于评估适配方案，不作为 PowerContext 已完成支持的证明： + +- [Apache Tika 文档格式说明](https://tika.apache.org/docs/4.0.x/formats.html)：提供跨文档类型的文本/元数据解析基础，明确指出识别范围大于正文解析范围；图像元数据不等于 OCR。 +- [Docling 支持格式](https://docling-project.github.io/docling/usage/supported_formats/)：提供统一文档表示和多种文档转换入口，可用于布局与结构提取适配。 +- [Microsoft MarkItDown](https://github.com/microsoft/markitdown)：提供文件到 Markdown 的转换和可选依赖机制，适合作为特定格式的候选适配器。 +- [RAGFlow 调研](https://yuque.antfin.com/obopensrc/knowledge_sharing/lmziuic5tzhgny8c)、[Docling/MinerU 调研](https://yuque.antfin.com/obopensrc/knowledge_sharing/rcga2z7gr8r3vwbo)、[LiteParse/LlamaIndex 调研](https://yuque.antfin.com/obopensrc/knowledge_sharing/yl6nfo4yugep4n6a)：解析平台边界、格式路由、版本与证据约束的综合依据。 +- [LiteParse 固定版本](https://github.com/run-llama/liteparse/tree/python-v2.14.7)：PDF 结构输出、复杂度、页范围及失败信息；不采用其 Office 转 PDF 作为工作簿默认路径。 +- [Docling OCR](https://docling-project.github.io/docling/concepts/OCR/)、[openpyxl 读取选项](https://openpyxl.readthedocs.io/en/stable/api/openpyxl.reader.excel.html)、[POI 工作簿组件](https://poi.apache.org/components/spreadsheet/)、[python-pptx 备注](https://python-pptx.readthedocs.io/en/latest/user/notes.html)：选定底层组件的能力依据；PC 仍需补齐语义审计和逐格式验收。 + +# Unresolved questions + +1. 第 1 节已确定各格式的默认解析族和候选边界；具体发布依赖版本、PDF 路由阈值及性能预算仍需同样本验收后冻结。XLSB 公式、传统 Word 修订视图与长尾代际的适配缺口未补齐前保持 planned，不以部分 profile 宣称完整交付。 +2. 50 MiB 原文件、8 MiB 提取结果、500 页和 128 段等默认预算需要在 SQLite/embedded seekdb/OceanBase 与实际部署资源上验证；调整时同时评估事务与下游消费成本。 +3. C 级 OCR 的首批语言包与专有格式代际需按真实资料验收。Apple/WPS 等长尾格式进入 available 必须有正文样本证明，不能只凭类型检测。 +4. 是否长期保留原始二进制及支持下载，需要单独定义存储配额、保留期、精确证据和删除授权；本提案只保留解析暂存。 +5. 如业务要求下游始终仅消费文档 Head 的 Source，需另行定义版本选择、旧 pending 与历史引用处理；本提案的 ImportedContent Head 不自动改变既有 Source 消费规则。 + +# Future possibilities + +在已有解析注册与能力发现机制下，可以继续接入更多语言、行业文档和专有格式，并单独定义音视频转写、图表理解、公式语义和手写识别。它们的成本和准确性不能隐含在普通文件解析中。 + +语雀、网页和其他知识平台以独立 Connector 接入，明确外部身份、版本、授权与精确读取能力。结构化数据库和超大型数据集应设计有选择条件的采集协议，而不是把任意数据库文件当普通文档无限展开。 + +可进一步支持长期原始文件归档、原文检索及精确位置跳转；资料级生成状态需由各 Family 提供真实 Source 覆盖和产出证据。所有扩展继续保留 Scope 边界、快照身份、遗漏报告和耐久回执。 From d21afbd551604ac4b71e41f5bce1d6a5f3eb4feb Mon Sep 17 00:00:00 2001 From: Frey <370805886@qq.com> Date: Mon, 19 Oct 2026 15:58:59 +0800 Subject: [PATCH 2/4] docs(rfc): assign text import proposal number 2073 --- ...000-text-material-import.md => 2073-text-material-import.md} | 2 +- ...000-text-material-import.md => 2073-text-material-import.md} | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) rename docs/en/rfcs/{0000-text-material-import.md => 2073-text-material-import.md} (99%) rename docs/zh/rfcs/{0000-text-material-import.md => 2073-text-material-import.md} (99%) diff --git a/docs/en/rfcs/0000-text-material-import.md b/docs/en/rfcs/2073-text-material-import.md similarity index 99% rename from docs/en/rfcs/0000-text-material-import.md rename to docs/en/rfcs/2073-text-material-import.md index abf405bec6..367ce3d1a7 100644 --- a/docs/en/rfcs/0000-text-material-import.md +++ b/docs/en/rfcs/2073-text-material-import.md @@ -5,7 +5,7 @@ title: Text Material Import - Proposal Name: `text_material_import` - Start Date: 2026-09-18 - Status: Proposed -- RFC PR: [oceanbase/powercontext#0000](https://github.com/oceanbase/powercontext/pull/0000) +- RFC PR: [oceanbase/powercontext#2073](https://github.com/oceanbase/powercontext/pull/2073) - Design Baseline: [PowerContext master / cab054db (2026-10-19)](https://github.com/oceanbase/powercontext/tree/cab054dbf6d86e2311b079388e5158178257def2) - Related RFCs: [Source Definition and Observation](1400_source_definition_and_observation_model.md), [Source and Artifact REST API](1437_source_artifact_rest_api.md), [Source access and generation eligibility](1458_artifact_generation_source_access.md), [Scope and Source discovery](1502_scope_source_discovery.md) diff --git a/docs/zh/rfcs/0000-text-material-import.md b/docs/zh/rfcs/2073-text-material-import.md similarity index 99% rename from docs/zh/rfcs/0000-text-material-import.md rename to docs/zh/rfcs/2073-text-material-import.md index 64b7f036b5..bc60af295a 100644 --- a/docs/zh/rfcs/0000-text-material-import.md +++ b/docs/zh/rfcs/2073-text-material-import.md @@ -5,7 +5,7 @@ title: 文本资料导入 - Proposal Name: `text_material_import` - Start Date: 2026-09-18 - Status: Proposed -- RFC PR: [oceanbase/powercontext#0000](https://github.com/oceanbase/powercontext/pull/0000) +- RFC PR: [oceanbase/powercontext#2073](https://github.com/oceanbase/powercontext/pull/2073) - Design Baseline: [PowerContext master / cab054db（2026-10-19）](https://github.com/oceanbase/powercontext/tree/cab054dbf6d86e2311b079388e5158178257def2) - Related RFCs: [Source Definition 与 Observation](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1400_source_definition_and_observation_model.md)、[Source 与 Artifact REST API](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1437_source_artifact_rest_api.md)、[Source 访问与生成准入](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1458_artifact_generation_source_access.md)、[Scope 与 Source 发现](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1502_scope_source_discovery.md) From 2fec6680ac78eb2175033bc22eddea585dc69767 Mon Sep 17 00:00:00 2001 From: Frey <370805886@qq.com> Date: Mon, 19 Oct 2026 17:05:09 +0800 Subject: [PATCH 3/4] docs(rfc): align text import HTTP contracts --- docs/en/rfcs/2073-text-material-import.md | 558 ++++++++++++++++------ docs/zh/rfcs/2073-text-material-import.md | 471 +++++++++++++++--- 2 files changed, 800 insertions(+), 229 deletions(-) diff --git a/docs/en/rfcs/2073-text-material-import.md b/docs/en/rfcs/2073-text-material-import.md index 367ce3d1a7..a6f26dea78 100644 --- a/docs/en/rfcs/2073-text-material-import.md +++ b/docs/en/rfcs/2073-text-material-import.md @@ -114,7 +114,7 @@ flowchart TD A["Select an existing Scope"] --> P["Check scope.read + scope.contribute; create ImportSession with fixed scope_id"] P --> B{"Input"} B -->|Paste text| C["Validate and normalize"] - B -->|Upload file| D["Family prepare: durable staging and isolated parsing"] + B -->|Upload file| D["File-operation creation: durable staging and isolated parsing"] D --> E["Preview body, parts, OCR, and omissions"] E -->|No usable body| X["Fail; publish nothing"] E -->|Confirm complete or explicitly accept partial| F["Confirm frozen bundle_digest"] @@ -192,7 +192,7 @@ operations resume after browser closure; incomplete uploads require reselecting ## 1. Architecture -The user-flow diagram separates input, parsing, and persistence. ImportedContent prepare creates a frozen +The user-flow diagram separates input, parsing, and persistence. ImportedContent file-operation creation produces a frozen ImportedContentBundle through an internal ledger and isolated workers. Only user confirmation or a worker's explicit acceptance policy invokes ContentImportApplication's per-document atomic commit. Existing Artifact processing retains its configuration and progress semantics. @@ -445,7 +445,7 @@ parser health. | --- | --- | | `GET /v1/server-info` | Deployment identity and compatibility. Add `content.import: {major: 1, minor: 0}` to OpenAPI `x-powercontext-feature-contracts`; protocol support is not enablement | | `GET /v1/capabilities` | Add optional `content_import` with `enabled`, `readiness` (ready/degraded/unavailable), and `capabilities_endpoint`. Absence cannot imply support. Runtime summary does not replace format health or Scope authorization | -| `GET /v1/import-capabilities?scope_id=...` | Actual formats, profiles, limits, and caller permissions for an existing Scope; drives UI availability | +| `GET /v1/scopes/{scope_id}/import-capabilities` | Actual formats, profiles, limits, and caller permissions for an existing Scope; drives UI availability | `content.import` and `content_import` are new contracts requiring OpenAPI, generated code, Server assembly, and client compatibility tests. The existing global endpoints retain `server.observe`. Ordinary callers with only Scope permissions @@ -456,12 +456,12 @@ Import-service configuration owns API enablement; the Dashboard switch controls means all enabled, declared-available profiles passed health checks; partial failure is degraded while healthy profiles remain usable; disabled/no usable profiles is unavailable. Planned profiles do not degrade enabled capabilities. Disabling imports or losing parser availability does not block published Artifact/Source reads. Discovery is a current -snapshot: prepare, retry, and confirm recheck applicable permission, runtime state, and frozen-result validity. An +snapshot: operation creation, retry, and commit recheck applicable permission, runtime state, and frozen-result validity. An unrelated parser outage cannot cause confirmation to reparse or discard frozen text. ### Format and profile identification -The new `GET /v1/import-capabilities` endpoint requires an existing `scope_id` and `scope.read`; missing/malformed IDs return 422. +The new `GET /v1/scopes/{scope_id}/import-capabilities` endpoint requires an existing `scope_id` and `scope.read`; malformed path IDs return 400; absent/invisible Scopes follow 404/403 policy. It reports `scope.contribute`, but starting import requires both permissions. Return fixed `source_kind=content`, extensions, canonical MIME, capability level, available/unavailable/planned status, parser profiles, extraction scope, OCR/conversion dependencies, and actual limits. Registration plus startup self-checks establish status; an API-mounted @@ -508,7 +508,7 @@ flowchart TD | Object | Responsibility and persistence | | --- | --- | | ImportSession | Creator Principal, immutable nonempty existing scope_id, expiry; recovery/isolation/concurrency for that destination; no new authorization | -| Internal operation / item_id | prepare returns operation_id; staging, discovery, leases, quotas, parsing, confirmation. item_id is stable only within its operation, not permanent identity or an Artifact | +| Internal operation / item_id | file-operation creation returns operation_id; staging, discovery, leases, quotas, parsing, confirmation. item_id is stable only within its operation, not permanent identity or an Artifact | | ImportedContentBundle | Frozen body, semantic manifest, coverage, digest; confirmation uses it without recalculating preview | | ImportedContent Artifact | Stable `(scope_id, family=importedcontent, artifact_id)`, allocated for manual creation or a first stable source | | ImportedContent Revision | Exact `(scope_id, family, artifact_id, revision)` with structured metadata and ordered Source mapping | @@ -528,7 +528,7 @@ says its body uses ContentSource, not whether it was pasted, uploaded, or acquir | `origin` | namespace, key, optional display_name; provenance, not authorization | | `parser` | Actual parser, configuration, version, OCR, conversion chain | -Revisions, frozen bundles, and read responses require `source_kind: "content"`. texts/prepare may omit it; the Server +Revisions, frozen bundles, and read responses require `source_kind: "content"`. texts requests and upload metadata may omit it; the Server supplies it before fingerprints, frozen digests, and cache keys. Explicit other values return 422. Thus omitted/explicit content are semantically identical given equal other fields. OpenAPI uses a single-value constraint and Python `Literal["content"]`, not caller-selectable types. @@ -619,7 +619,7 @@ family schemas, Server capabilities, and Dashboard mappings: | Capability | This release | | --- | --- | | list/get, exact Revision, history, comparison | Supported under Scope `scope.read` | -| texts/prepare/confirm | Supported with scope.read + scope.contribute, creator isolation, transactional validation | +| texts/operations/commit | Supported with scope.read + scope.contribute, creator isolation, transactional validation | | Generic Artifact POST/PUT | Unsupported; cannot bypass confirmation with forged parts/lineage/success | | Generic restore/rollback, merge/undo_merge | Unsupported; no adapter/allowlist entry; HTTP and Runtime reject, UI hides actions | | Model generation, Dream/Candidate publication | Unsupported; no ImportedContent generation/candidate processor | @@ -760,8 +760,8 @@ state: | --- | --- | | uploading/queued/running, including resource waits | Held | | idle after discovery awaiting selection, or selected uncommitted ready/needs_review documents | Held | -| partially_committed with queued/parsing/selection/confirmation work | Held | -| failed, or partially_committed with only failures and no scheduled automatic retry | Released; explicit retry needs readmission | +| Partial commits with queued/parsing/selection/confirmation work | Held | +| failed, including partial commits with only failures and no scheduled automatic retry | Released; explicit retry needs readmission | | completed/cancelled/expired | Released | Scheduled automatic retries remain queued and hold a slot; exhausted retries enter failed and release it. Slot release @@ -784,7 +784,7 @@ instead of hot-loop reuploads. ## 6. Internal execution, private previews, and durable staging -Family prepare accepts complex files and returns recoverable operation_id, backed by Section 3's ledger and isolated +Family operation creation accepts complex files and returns recoverable operation_id, backed by Section 3's ledger and isolated workers. Revisions represent accepted snapshots only. “Task” below means internal operation execution, not FileImportJob or another Artifact type. @@ -797,8 +797,8 @@ Store execution, parse, and commit states separately. ready is not publication o | Level | Field/values | | --- | --- | | Task execution | execution_state: uploading / queued / running / idle / failed / cancelled / expired; idle means no current parse work | -| Phase | phase: upload / discovery / selection / parsing / confirmation / done; discovery_state: not_required / pending / running / complete / failed | -| Selection | selected; container membership frozen through selection API after discovery | +| Internal phase | internal_phase: upload / discovery / selection / parsing / confirmation / done; discovery_state: not_required / pending / running / complete / failed | +| Selection | selected; container membership frozen through select API after discovery | | Document parsing | parse_state: pending / parsing / ready / needs_review / failed; ready/needs_review refer to immutable bundles | | Document persistence | commit_state: uncommitted / committed; no exposed half-commit state | @@ -827,16 +827,11 @@ flowchart LR B -->|Confirm digest and accept partial| C ``` -Derive aggregate_status and counts in priority order: cancelled/expired; uploading while receiving; queued/parsing with -pending/running work; partially_committed when some documents committed, not all complete, and no active work; completed -when nonempty selection is all committed; awaiting_confirmation for uncommitted ready/needs_review or unselected -discovered members; failed for task-level failure or only unparseable members. Unselected members count as excluded, not -required commits. Cancellation/expiry still return committed counts and exact Revisions, preserving partial success -visibility. +These are internal durable states, not HTTP fields. Derive public ImportOperation.status in priority order: preserve completed/cancelled terminal results; expire unfinished work; running work yields running, otherwise queued work yields queued; complete discovery without selection yields awaiting_selection; nonempty all-committed selection yields completed; uncommitted ready/needs_review yields awaiting_commit; task-level failure or only remaining failed members yields failed. Scheduled automatic retries remain queued. Partial success uses progress counts, not a separate partially_committed status. Public phase is discovery/parsing/commit/done: discovery and selection waits map to discovery, body extraction to parsing, confirmation waits to commit, and all-success to done; failure/cancellation/expiry retain their last phase. Internal discovery_state=complete maps to discovery.status=completed; parse_state/commit_state map to parse_status/commit_status. Pre-202 uploading is not a publicly recoverable accepted operation. Unselected members count as excluded_items; cancellation/expiry retain committed counts and receipts. Single-document input automatically fixes selected=true. Archives/mailboxes/attachments require bounded complete discovery, stable item_ids, container_chain, types, and default suggestions. Attachments default unselected; ordinary -archive members default selected. Pause after discovery; selection API freezes one explicit set before body parsing. +archive members default selected. Pause after discovery; select API freezes one explicit set before body parsing. Overflow/incomplete discovery is a discovery error, not a complete list. Structural container nodes are not selectable; omitted attachments are excluded. Changed selection/options require a new attempt. Persistent item mapping preserves IDs through automatic retries. Publish complete manifest and discovery_state=complete atomically; intermediate lists cannot @@ -876,11 +871,11 @@ promise. ### Sessions and Scope -Require an existing scope_id and current scope.read + scope.contribute at session creation. Persist creator and +Require an existing scope_id in the path, an empty object body, and current scope.read + scope.contribute at session creation. Persist creator and immutable nonempty target; no deferred resolution or Scope-creation parameters. Creating a session creates no Scope/Artifact/Source or permission grant. -Session, operation, item, quota, and eventual Artifact/Sources share that Scope. Child routes derive it from session_id; +Session, operation, item, quota, and eventual Artifact/Sources share that Scope. Child routes validate that path scope_id matches the session destination; bodies cannot override scope_id. Cross-Scope artifact_id, SourceRef, or bindings cannot target the session. Changing Scope requires a new session and inputs; existing operations stay put. @@ -946,7 +941,7 @@ Revision. A historical version is not current automatically; manual create remai hidden. Complete cache output still needs this interactive attempt's confirmation; only explicitly configured accept-complete -workers auto-confirm. Partial cache yields needs_review/uncommitted with candidate_ref, coverage, omissions and no +workers auto-confirm. Partial cache yields needs_review/uncommitted with reusable_artifact, coverage, omissions and no completed count. Require this attempt's explicit accept_partial=true, never inherit another Principal's or earlier attempt's decision. Same successful commit-key replay preserves its original decision. @@ -981,143 +976,402 @@ one generation. ## 8. HTTP, client, and page contracts -Published materials reuse standard Artifact APIs with family=importedcontent. Extend registration/schemas for these -existing operations instead of a separate material resource: +### 8.1 Conventions, identities, and operations -| Existing operationId | Method/path | Behavior | +Follow [RFC 1437](1437_source_artifact_rest_api.md) for Scope resource identity, naming, errors, pagination, and standard Artifact reads. Collections use lowercase plural nouns, static multiword segments use kebab-case, JSON/query/operationId use snake_case, and URIs have no trailing slash. `select`, `commit`, `retry`, and `cancel` are explicit POST domain actions, not arbitrary state PATCH/PUT. Validate the entire Scope/session/operation/item ownership chain; never look up only the last ID. Do not accept path identities again in request bodies. + +These are proposed contracts to implement in `openapi/powercontext.yaml`, the single HTTP contract source. Generate Python/JavaScript clients and run contract-test; do not hand-edit generated code. Family is fixed to `importedcontent`; sessions require an existing Scope and cannot create one. API resources do not introduce a separate FileImportJob or ImportedContent master table. + +The following abbreviations are documentation-only; clients send expanded paths, not literal P/S/O/I: + +```text +P = /v1/scopes/{scope_id} +S = P/import-sessions/{session_id} +O = S/operations/{operation_id} +I = O/items/{item_id} +``` + +| operationId | Method and path | Success and meaning | +| --- | --- | --- | +| `get_import_capabilities` | `GET P/import-capabilities` | 200; caller-visible formats, profiles, permissions, effective limits | +| `create_import_session` | `POST P/import-sessions` | 201; fixed Scope and creator | +| `list_import_sessions` | `GET P/import-sessions` | 200; creator's recovery list within this Scope | +| `get_import_session` | `GET S` | 200; session/deadlines, no bodies | +| `import_content_text` | `POST S/texts` | 201 for new Artifact; 200 for update/reuse; request itself confirms | +| `create_import_operation` | `POST S/operations` | 202; one file durably accepted/queued, no published Artifact | +| `list_import_operations` | `GET S/operations` | 200; recovery operations, not a material catalog | +| `get_import_operation` | `GET O` | 200; status, phase, counts, deadlines, actions | +| `list_import_items` | `GET O/items` | 200; members; single documents also have one item | +| `get_import_item` | `GET I` | 200; frozen target, extraction, coverage, diagnostics | +| `get_import_part` | `GET I/parts/{ordinal}` | 200; bounded preview, reading Source after commit | +| `select_import_items` | `POST O/select` | 202; atomically freeze container selection and enqueue | +| `commit_import_item` | `POST I/commit` | 201 for new Artifact; 200 for update/reuse; atomic per document | +| `retry_import_operation` | `POST O/retry` | 202; retry discovery or specified failed documents after readmission | +| `cancel_import_operation` | `POST O/cancel` | 200; cancel uncommitted work or return completed result | + +Published materials reuse existing operations, extending Family registration, schemas, and read projections rather than creating a second detail resource: + +| operationId | Method and path | Behavior | | --- | --- | --- | -| `list_artifacts` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent` | Visible current Heads; exact namespace filter | -| `get_artifact` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent/{artifact_id}` | Current accepted snapshot and ordered parts | -| `list_artifact_revisions` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent/{artifact_id}/revisions` | Accepted history | -| `get_artifact_revision` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent/{artifact_id}/revisions/{revision}` | Exact immutable snapshot; receipt Location points here | +| `list_artifacts` | `GET P/artifacts/importedcontent` | Visible Heads; exact namespace filter, not body search | +| `get_artifact` | `GET P/artifacts/importedcontent/{artifact_id}` | Standard ArtifactRevision, parts, Head ETag; retain If-None-Match/304 | +| `list_artifact_revisions` | `GET P/artifacts/importedcontent/{artifact_id}/revisions` | Accepted revision history | +| `get_artifact_revision` | `GET P/artifacts/importedcontent/{artifact_id}/revisions/{revision}` | Exact immutable revision; no ETag per existing contract | + +Generic Artifact POST/PUT reject direct importedcontent writes with 422 unsupported: they cannot bypass parts/lineage/acceptance validation. Other Families retain generic Replace If-Match behavior: 428 if missing, 412 if mismatched. Import commit is a domain command using frozen expected_head_revision and 409 for conflicts; do not treat an Artifact ETag as an operation/item HTTP validator. + +### 8.2 Common fields, headers, and validation + +| Item | Format and constraints | +| --- | --- | +| scope_id | Existing Scope contract: 1–256 characters including a non-whitespace character; encode as one URL segment | +| session_id, operation_id, item_id | Server-generated opaque strings, 1–128 visible ASCII characters; no prefix parsing or UUID assumption | +| artifact_id | Existing 1–128 visible ASCII characters; must belong to the path Scope and importedcontent Family | +| revision / expected_head_revision | Integer, minimum 1 | +| ordinal | Zero-based integer; the nth part has ordinal n-1 | +| Times | RFC 3339 UTC, e.g. `2026-10-19T08:00:00Z` | +| Digests | `^sha256:[0-9a-f]{64}$`; input, bundle, snapshot, and text digests have separate meanings | +| Sizes / durations | Integer bytes use `_bytes`; integer seconds use `_seconds` | +| Missing / null | Defaults apply only to omitted request fields; reject null unless explicitly nullable. Unknown response counts are null; zero means verified absence | +| source_kind | Optional request field defaulted to content before fingerprinting; only content accepted; always persisted/returned | + +When authentication is enabled, send `Authorization: Bearer `. Every import write requires Idempotency-Key (1–128 ASCII letters/digits or `-_.`); missing/malformed headers return 400. All success/error responses include X-PowerContext-Request-ID. JSON is UTF-8, with no extra code/message/data success envelope. Unknown body fields return 422; unknown queries and invalid path/header/cursor values return 400. Duplicate/extra multipart parts return 422. Reject NUL, invalid Unicode, and C0 controls other than TAB/CR/LF in nonblank text. + +Location targets S for session creation, O for operation creation, and the exact Artifact Revision for text/commit success. Import receipts are not generic Head representations and do not fabricate their ETag. Admission 429/503 includes Retry-After in seconds. Upload/action 202 returns an acceptance-time ImportOperation snapshot; replay preserves it rather than substituting current state, which is available through GET O. + +### 8.3 Deployment capabilities and sessions + +`GET P/import-capabilities` requires scope.read, not server.observe; invisible/missing Scopes follow existing 403/404 policy. + +| Response field | Type and meaning | +| --- | --- | +| scope_id, family, source_kind | Path Scope, importedcontent, content | +| contract_version | `{major:1,minor:0}`, matching the content.import feature contract, not runtime availability | +| enabled, can_import | Booleans for deployment enablement and current caller write permissions, not future authorization grants | +| readiness | ready / degraded / unavailable, as defined in Section 2 | +| limits | Effective max_text_bytes, max_json_bytes, max_file_bytes, max_metadata_bytes, max_document_text_bytes, max_parts_per_document, max_part_bytes, max_selected_items, session_ttl_seconds, operation_ttl_seconds; also Section 5's format-specific and aggregate limits | +| formats[] | format_id, extensions, media_types, level(A/B/C), status(available/unavailable/planned), nullable default_profile_id, profile_ids, extraction_scope, limits, nullable reason | +| profiles[] | profile_id, profile_digest, status, defaults, options_schema, extraction_scope, dependencies; options_schema is a bounded JSON Schema 2020-12 subset with additionalProperties=false, never executable code or arbitrary parser classes | +| quotas | Caller-visible quota limits/remaining amounts, without other Principals' usage | + +Extensions include the dot, e.g. `.pdf`; media_types are canonical MIME values; reason is `{code,message}`. Profile digests cover pinned configuration/dependencies, not a release clock. This is a current snapshot; every write rechecks eligibility. Initial limits follow Section 5, e.g. max_file_bytes=52428800, max_text_bytes=262144, max_document_text_bytes=8388608, max_parts_per_document=128, max_part_bytes=262144, max_metadata_bytes=65536; clients must not hard-code these as deployed values. + +Create a session: + +```http +POST /v1/scopes/project-docs/import-sessions +Content-Type: application/json +Idempotency-Key: session-notes-001 -Imports are Family-specific writes. Select an existing Scope before creating a session; prepare/confirm derive its fixed -target from session_id. Session routes recover a batch, not a new permanent file-import business resource. All -operations below are proposed, not existing recovery APIs: +{} +``` + +Only the empty object is accepted; no body scope_id or Scope-creation parameters. After scope.read + scope.contribute checks, return 201: -| operationId | Method/path | Meaning | +```json +{ + "scope_id": "project-docs", + "session_id": "is_001", + "created_at": "2026-10-19T08:00:00Z", + "expires_at": "2026-10-20T08:00:00Z", + "idempotency_expires_at": "2026-11-19T08:00:00Z" +} +``` + +GET S returns the same ImportSession fields with server-calculated times. Keep session-creation replay mappings for at least 30 days after session expiry, including sessions that never upload. Child deadlines never exceed session expiry. List only creator-visible sessions within the path Scope; cross-Scope session discovery is out of scope. + +### 8.4 Shared target and origin + +artifact_target is a mode-discriminated union, defaulting to `{"mode":"create"}`. Never infer updates from filenames/origin: + +| mode | Fields | Rules | | --- | --- | --- | -| `get_import_capabilities` | `GET /v1/import-capabilities?scope_id=...` | Required existing Scope; content Source type, formats, permissions, limits | -| `create_import_session` | `POST /v1/import-sessions` | Existing scope_id required; validate contribution and fix destination/creator | -| `list_import_sessions` | `GET /v1/import-sessions` | Creator's recoverable sessions, paginated | -| `get_import_session` | `GET /v1/import-sessions/{session_id}` | Fixed scope_id, expiry, receipt summaries | -| `import_content_text` | `POST /v1/import-sessions/{session_id}/importedcontent/texts` | content; synchronously validate/confirm bounded body, 201/200 | -| `prepare_imported_content` | `POST /v1/import-sessions/{session_id}/importedcontent/prepare` | content; metadata + one file; 202 and operation_id after durable acceptance | -| `list_import_operations` | `GET /v1/import-sessions/{session_id}/operations` | Creator's recovery list, not material catalog | -| `get_import_operation` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}` | Execution/errors/deadlines/counts/available actions | -| `list_import_operation_items` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}/items` | Bounded manifest; single files also have one item | -| `get_import_operation_item` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}/items/{item_id}` | Manifest, coverage, warnings, omissions, frozen target | -| `get_import_operation_part` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}/items/{item_id}/parts/{ordinal}` | Creator's bounded preview; committed body from Source | -| `confirm_imported_content` | `POST /v1/import-sessions/{session_id}/importedcontent/confirm` | operation_id + item_id + bundle_digest + accept_partial; atomic save, 201/200 | -| `select_imported_content` | `POST /v1/import-sessions/{session_id}/importedcontent/selection` | operation_id, members and individual targets; 202 after frozen selection | -| `retry_import_operation` | `POST /v1/import-sessions/{session_id}/operations/{operation_id}/retry` | Controlled Family retry; 202 after readmission | -| `cancel_import_operation` | `POST /v1/import-sessions/{session_id}/operations/{operation_id}/cancel` | Fence uncommitted writes; atomic cancellation/completed result, 200 | - -prepare's 202 returns `{operation_id, session_id, family: "importedcontent", expires_at, status_url}`. Location points -to status, not a nonexistent Artifact. texts is itself confirmation and returns the same saved receipt as confirm -without a separate parsing operation. Operations/items have no generic PUT/PATCH state mutation. Confirm/selection -verify operation, session, creator, and Family; cross-session/invisible handles follow existing 404/403 policy. Generic -Artifact POST/PUT for importedcontent returns 422 from Family capability validation, without falling back to command -Sources. - -Session creation accepts `{"scope_id":"existing Scope ID"}`. Require nonempty valid ID; missing, empty, unknown fields, -or Scope-creation parameters return 422. Missing/invisible/unauthorized targets follow existing 404/403 visibility -policy without accepting files or creating sessions. Return session_id, fixed nonempty scope_id, expires_at. Every write -requires Idempotency-Key; no target-mutation API. - -Targets are `{"mode":"create"}` or `{"mode":"update","artifact_id":"...","expected_head_revision":2}`. Manual defaults -to create, never infers identity from origin. Updates must target this Scope and Family; cross-Scope targets follow -visibility rules. Authorized registered Connector Principals may instead use the mutually exclusive -`{"mode":"source","source_authority_id":"...","source_key":"..."}`. Resolve bindings and freeze target/expected Head or -absence at acceptance; return the resolved target. origin cannot impersonate authority. Each container member has its -own target; multiple members cannot update the same artifact_id. - -A paste request stays simple: +| create | mode only | Explicit creation intent; server allocates identity | +| update | Required artifact_id and expected_head_revision | Freeze at acceptance; validate Scope/Family/Head at commit, never silently retarget | +| source | Required source_authority_id and source_key | Authorized Connector Principal only; resolve binding/absence assertion and freeze target at acceptance | + +source_authority_id is 1–128 visible ASCII characters. source_key is 1–1024 Unicode characters, nonempty, without outer whitespace/controls, following its registered authority's normalization. Read-only resolved_target is `{mode,artifact_id,expected_head_revision}`; mode is create/update, create has null expected_head_revision. Unselected container members have null targets. Clients cannot submit resolved_target, sources, artifacts, parts, or writable lineage. + +origin is required: namespace (1–128 characters), key (1–1024), optional display_name (1–256); no controls, and no outer whitespace in namespace/key. File keys are relative POSIX paths, rejecting absolute paths, backslashes, empty components, `.` and `..`. Origin is provenance, not binding authorization or a fetch instruction. Derive/return container-member origins and container_chain using Section 3's stable-member rules; ordinals cannot masquerade as stable cross-version identity. + +### 8.5 Direct text import + +```http +POST /v1/scopes/project-docs/import-sessions/is_001/texts +Content-Type: application/json +Idempotency-Key: text-notes-001 +``` ```json { "title": "API conventions", "artifact_target": {"mode": "create"}, "source_kind": "content", + "origin": {"namespace": "project-notes", "key": "api-conventions"}, "media_type": "text/markdown", - "origin": {"namespace": "project-notes", "key": "note-001"}, "text": "# API conventions\n\nPagination uses opaque cursors.\n" } ``` -Upload metadata contains artifact_target, title, origin, parser_profile, optional fixed-content source_kind, and -optional encoding, as_text, reparse, ocr_mode, language, include_notes, include_comments, include_hidden, archive_depth. -Defaults: as_text=false, reparse=false, ocr_mode=off, include_notes=true, include_comments=false, include_hidden=true, -archive_depth=1. Omitted language uses the profile's fixed language set; omitted encoding follows identification or -returns encoding_required. Without parser_profile, freeze the deployment router/profile registry version, identify -format, and freeze its default profile; PDF chooses its backend under that profile. Containers also freeze child routing -registry so discovered/selected children cannot adopt changed defaults mid-operation. Across-key retries should send the -resolved profile digest. include_notes applies only where supported; irrelevant defaults are ignored, explicitly -inapplicable nondefaults return 422. archive_depth is 1 or 2. Publish all defaults through capabilities. - -as_text=true still requires strict text validation, binary identification, and resource limits. ocr_mode is -off/auto/force; auto handles required regions only, force is disclosed in preview. reparse=true needs new upload and -Idempotency-Key with original bytes; do not assume old staging exists or alter old bundles. Selection controls -attachments. - -selection accepts `{operation_id, expected_selection_version: 0, items: [{item_id, artifact_target}, ...]}` only after -complete discovery, before selection freezes, and while valid. IDs must be selectable, unique, nonempty, from the same -complete manifest, at most 1,000. Validate targets/no duplicate Artifact updates, atomically freeze version 1, and -enqueue. Changed selection returns 409; unknown/foreign IDs return 422 without disclosure. Single documents need no -selection call. - -retry requires one of `{stage: "discovery"}` or `{stage: "documents", item_ids: [...]}`. Discovery forbids item_ids and -requires discovery_state=failed, discovery_retryable=true, unpublished manifest, valid staged container. Acceptance sets -discovery_state=pending, phase=discovery, execution_state=queued. Document retry requires complete/not-required -discovery and 1–1,000 unique selected failed retryable item IDs; set these pending and enqueue. Invalid structures -return 422; phase/state mismatches return 409 for the entire request; neither alters frozen output. Both branches return -202 only after atomic task-slot admission, enqueue, and idempotent receipt. - -cancel accepts an empty object. First cancellation is atomic; repeats return the same receipt. Completed returns its -completion result; expired returns tombstone. No published Revision is deleted. Selection/retry/cancel require creator, -fixed-Scope scope.read + scope.contribute, and Idempotency-Key, with Section 10 administrative exceptions. Replays -retain acceptance receipts; GET shows later progress. Selection/retry on cancelled return 409, expired 410. - -Operation responses independently expose operation_id, session_id, family, phase, discovery_state, discovery_retryable, -phase errors, execution_state, aggregate_status, task_slot_held, selection_version, counts, expires_at, -idempotency_expires_at, blocked_reason. Item responses include selected, parse_state, commit_state, retryable, coverage, -warnings, omissions, bundle_digest, optional candidate_ref and artifact_ref (exact, Scope-bearing ArtifactRefs). -artifact_ref means this attempt committed; candidate_ref does not. Clients must not infer a combined status. - -Each upload accepts exactly one file and one metadata object. metadata ≤64 KiB; reject unknown fields and extra -multipart fields/files. Queries/manifests do not return raw blob addresses. - -confirm accepts `{operation_id, item_id, bundle_digest, accept_partial}` with Idempotency-Key; target and expected Head -are already frozen and echoed in the response. Changed bundle returns 409; expired uncommitted item 410; -cancelled/unparsed item 409. Committed items replay durable receipts throughout mapping retention even if the operation -later expires; thereafter use standard exact Revision reads. ready requires accept_partial=false; needs_review requires -true, otherwise 409. Return `{outcome: created|reused, scope_id, artifact_ref, snapshot_digest, parts}`. created -includes first and subsequent new Revisions. artifact_ref contains family, artifact_id, revision, with Location at the -standard exact Revision resource. New import responses uniformly use parts[]; pasted text has one part. Existing Source -responses are unchanged. - -Task/item/material lists default to limit 50, range 1–100, SQL keyset pagination with a first-request high-water mark. -Bind cursor to operation, caller, stable session_id or scope_id, filters, limit, high-water mark, and TTL; reauthorize -each page. Mutable task states mean lists are not state snapshots. Namespace filtering extends the Artifact-list -projection, not full-text search. - -Python client exposes these operations. CLI provides text, files, resume, cancel, dry-run. New imports require --scope -before uploading; --session resumes its fixed destination and displays it. If both are supplied they must match. ---artifact-id and --expected-head-revision explicitly update; expose --as-text, --reparse, member selection. Containers -list members before selection; unattended use needs explicit rules. dry-run validates only locally knowable facts: -without running a parser offline, report server parsing required rather than full importability. Default CLI behavior -confirms interactively after deployment parsing. Explicit --accept-complete can auto-save ready; partial needs -additional explicit permission, including cached partial snapshots. UI/CLI offer discovery retry or failed-document -retry based on phase/discovery_state, without requiring undiscovered item IDs. Quota rejection backs off per Retry-After -and cannot appear queued. - -Reopening/previewing is read-only; logging in again does not submit. Confirm carries frozen digest/original key. Verify -uncertain outcomes through status/receipts or same-key retry. Separate upload, parsing, and Source persistence progress; -show indeterminate processing if the parser cannot report pages. - -Conflicts do not mutate frozen targets. Read current Head, then create a new file attempt with new expected Head, -original bytes, and new key; parsing cache may help, but old items/receipts stay immutable. Show current-version -differences before confirmation. Generic writes cannot rewrite provenance/parts/lineage. +| Field | Required / default | Meaning | +| --- | --- | --- | +| title | Required | 1–256 Unicode characters after trimming | +| artifact_target | create | Section 8.4 target | +| source_kind | content | Fixed Source type | +| origin | Required | Controlled provenance | +| media_type | text/plain | Only text/plain or text/markdown; saved body representation | +| text | Required | Nonblank string; UTF-8 bytes ≤ max_text_bytes, total JSON ≤ max_json_bytes | + +The request confirms immediately, without a mandatory asynchronous parsing operation; return Section 8.10's ImportCommitReceipt. Pasted HTML/code is stored as source text in the chosen representation, never executed or implicitly extracted. Use file parsing for semantic extraction. No parser, URL, or server-path input is accepted here. + +### 8.6 File uploads and parser parameters + +POST S/operations accepts multipart/form-data with exactly one file part (raw bytes) and one metadata part (application/json, at most 64 KiB). Reject extra files, URLs, and server paths. Filename/client Content-Type are hints; identify actual type from bytes, MIME, and container structure. + +```json +{ + "title": "Product manual", + "artifact_target": {"mode": "create"}, + "source_kind": "content", + "origin": {"namespace": "product-docs", "key": "manual.pdf"}, + "parser": { + "profile_id": "pdf-standard-v1", + "options": {"ocr_mode": "auto"} + }, + "reparse": false +} +``` + +| metadata field | Required / default | Meaning | +| --- | --- | --- | +| title | Optional | Document title follows text-title constraints; omitted uses normalized filename, with the fixed locale-independent value `Untitled material` as empty-name fallback. Container title is display-only; members use manifest titles | +| artifact_target | Single-document default create | Containers accept only omission/create; select assigns update/source per member, never one shared update target | +| source_kind / origin | content / required | Same as text import | +| parser | Optional | ParserSelection below | +| reparse | false | New upload bypasses parsing-result cache and cross-key operation merging; never mutates old work or assumes old staging exists | + +| parser field | Type/default | Semantics | +| --- | --- | --- | +| profile_id | Optional string | Available profile; omission pins router registry and chooses format default | +| profile_digest | Optional SHA-256 | Requires profile_id; pins exact configuration; mismatch is 409 profile_conflict, not silent replacement | +| options | Object, empty by default | Explicit overrides only; reject undeclared profile options | +| options.encoding | Optional string | Capability-listed encoding; omission uses BOM/strict UTF-8 or encoding_required | +| options.as_text | boolean, false | Requires strict text validation, never bypasses binary/resource limits | +| options.ocr_mode | off/auto/force, off | Disabled, selective, forced; actual execution recorded separately | +| options.languages | Optional string[] | Nonempty, unique capability-listed language IDs; omission uses fixed profile set, never guesses/downloads models | +| options.include_notes | boolean, true | Include supported notes | +| options.include_comments | boolean, false | Include supported comments | +| options.include_hidden | boolean, true | Include supported hidden content | +| options.archive_depth | integer, 1 | Only 1 or 2 | + +Explicit inapplicable options return 422; profile defaults apply only to appropriate formats. Return effective options after defaulting. Containers pin their child router/profile registry and apply overrides only to applicable members; an override applicable to no discovered member is a discovery-stage parameter error, not silently ignored. For pinned cross-key requests, send both profile_id/profile_digest. Return actual backend/model/conversion information on items; clients cannot specify arbitrary import classes. + +The following command reads and uploads a local file to a previously selected Scope/session. Add Authorization when enabled; metadata.json contains the preceding example: + +```bash +curl --request POST \ + 'http://127.0.0.1:8000/v1/scopes/project-docs/import-sessions/is_001/operations' \ + --header 'Idempotency-Key: upload-manual-001' \ + --form 'file=@manual.pdf;type=application/pdf' \ + --form 'metadata=@metadata.json;type=application/json' +``` + +Return 202/ImportOperation only after durable bytes, digest verification, quota, queue, and receipt persistence. Interrupted upload never accepts a partial file. Public status has no uploading value; clients display upload progress themselves. + +Example single-document 202 response, with Location `/v1/scopes/project-docs/import-sessions/is_001/operations/io_001`: + +```json +{ + "scope_id": "project-docs", + "session_id": "is_001", + "operation_id": "io_001", + "family": "importedcontent", + "status": "queued", + "phase": "parsing", + "discovery": {"status": "not_required", "retryable": false, "error": null}, + "selection_version": 1, + "progress": {"discovered_items": 1, "selected_items": 1, "parsed_items": 0, "failed_items": 0, "committed_items": 0, "excluded_items": 0}, + "available_actions": ["cancel"], + "created_at": "2026-10-19T08:05:00Z", + "expires_at": "2026-10-20T08:00:00Z", + "idempotency_expires_at": "2026-11-19T08:00:00Z", + "blocked_reason": null, + "retry_after_seconds": null, + "error": null +} +``` + +### 8.7 Operation queries, items, and previews + +ImportOperation uses the same fields for POST acceptance snapshots, GET, and list items: + +| Field | Type and meaning | +| --- | --- | +| scope_id, session_id, operation_id, family | Path identity and importedcontent | +| status | queued/running/awaiting_selection/awaiting_commit/completed/failed/cancelled/expired; server-derived by Section 6 | +| phase | discovery/parsing/commit/done; errors retain failed phase, success uses done | +| discovery | `{status,retryable,error}`; status not_required/pending/running/completed/failed; nullable ErrorDetail | +| selection_version | Nonnegative integer; container 0 before freezing, 1 after; single document automatically 1 | +| progress | discovered_items (null before complete discovery), selected_items (null before selection), parsed_items, failed_items, committed_items, excluded_items; last four nonnegative. parsed_items counts frozen results including committed items, so counts are not additive categories | +| available_actions | select/retry/cancel hints, revalidated at execution | +| created_at, expires_at, idempotency_expires_at | RFC 3339 UTC; retries do not extend deadlines | +| blocked_reason, retry_after_seconds | Nullable `{code,message}` and nonnegative integer for accepted queue waits | +| error | Nullable ErrorDetail for operation-level errors; item failure is not HTTP query failure | + +Internal execution_state, task_slot_held, leases, and fencing generations are not public status fields. Partial success is represented by progress: active work yields queued/running, ready uncommitted results yield awaiting_commit, and only remaining failures yield failed. Cancellation/expiry preserve committed counts and per-item receipts; an operation is not a cross-document transaction. + +ImportItem (GET I and bounded GET O/items entries, excluding body text/full content_manifest): + +| Field | Type and meaning | +| --- | --- | +| scope_id, session_id, operation_id, item_id | Complete parent/child identity | +| title, origin, container_chain, input_media_type | Identified title, provenance, member chain, MIME; unidentified MIME is null | +| selectable, selection_suggested, selected | Booleans; structural nodes are not selectable, suggestions do not mean acceptance | +| artifact_target, resolved_target | Intent and frozen resolved target; nullable before selection | +| parse_status | pending/parsing/ready/needs_review/failed | +| commit_status | uncommitted/committed | +| retryable | Whether the current parse error can be retried | +| parser | Nullable; profile_id/profile_digest/router_version/options plus actual backends/conversions; fully returned after freezing | +| statistics | page_count, sheet_count, slide_count, table_count, part_count, text_bytes; nonnegative integers or null, never zero for unknown/inapplicable | +| ocr | requested_mode, nullable boolean used, nullable integer processed_page_count, nullable string engine, languages; incomplete counts are not final totals | +| coverage | complete/partial/unknown; null before a result exists | +| warnings, omissions | Arrays of `{code,message,locator}` with nullable locator; structured issues affect semantic digests, display messages do not. Bound diagnostics; fail if they cannot be represented within budget rather than hiding omissions | +| input_digest, bundle_digest | SHA-256 or null; no invented digests before stage completion | +| reusable_artifact | Nullable standard `{family,artifact_id,revision}` within top-level Scope; possible cached result, neither a successful commit nor a Candidate | +| receipt | Nullable ImportCommitReceipt, present only after this attempt commits | +| available_actions, error | commit hints; nullable ErrorDetail | + +GET I/parts/{ordinal} reads one stable part, not the full body. Return scope_id/session_id/operation_id/item_id/ordinal/text/media_type/text_bytes/text_digest/bundle_digest/locators. Maximum 256 KiB per part, 128 parts per document; return the complete part without truncation. Sections 4–5 define splitting/overflow. Unknown ordinal is 404, unfinished parsing is 409, uncommitted expired previews are 410 and cancelled previews 409. Committed previews read Sources while mappings remain, then standard Artifact/Source APIs provide permanent access. + +Locations use a kind discriminator, e.g. `{kind:"pdf",page_number:3}`, `{kind:"spreadsheet",sheet_name:"Budget",cell_range:"A1:D20"}`, `{kind:"slide",slide_number:5}`, `{kind:"text",line_start:10,line_end:25}`. Pages/slides/lines are one-based; line ranges are inclusive. Full-text character ranges count Unicode code points as `[start,end)`. PDF bbox requires page_width/page_height, unit=pt, origin=top_left, and bbox=[left,top,right,bottom]; never mix pixels and points. Report unavailable locations rather than guessing. + +### 8.8 Selection, retry, and cancellation + +POST O/select: + +```json +{ + "expected_selection_version": 0, + "items": [ + {"item_id": "ii_001", "artifact_target": {"mode": "create"}}, + {"item_id": "ii_002", "artifact_target": {"mode": "update", "artifact_id": "ic_002", "expected_head_revision": 3}} + ] +} +``` + +expected_selection_version is required and must be 0. items is required/nonempty/unique, at most 1,000; each item_id is required and artifact_target defaults to create. Accept only a complete, not-yet-frozen manifest. Atomically validate every target, reject multiple members updating one Artifact, freeze version 1, and enqueue. Unknown/foreign members are 422 without disclosure; changed frozen selection is 409. Same-key replay returns its original snapshot. Single documents neither need nor permit select again. + +POST O/retry accepts one of two mutually exclusive bodies: + +```json +{"stage": "discovery"} +``` + +```json +{"stage": "documents", "item_ids": ["ii_002", "ii_003"]} +``` + +stage is required. discovery forbids item_ids and requires failed/retryable discovery, unpublished complete manifest, and valid staging. documents requires a nonempty unique item_ids array of at most 1,000 selected, failed, retryable, unfrozen members of this operation. Reject the whole request if any member is ineligible. Never retry ready/needs_review/committed, change inputs/profiles/targets, or extend deadlines. Manual retries do not reset the per-phase maximum of three automatic retries. Return 202 only after atomic quota readmission, queue, and receipt persistence. 429/503 means not accepted; back off and reuse the key. Accepted background resource waits appear through GET O blocked_reason. + +POST O/cancel accepts only `{}`; atomically fence uncommitted writes and return 200 ImportOperation. Repeated cancellation does not repeat effects, completed remains completed, expired returns a tombstone, and published Artifacts survive. select/retry return 409 for cancelled and 410 for expired operations. Every action rechecks creator, current Scope permissions, and ancestry; administrative exceptions follow Section 10. + +### 8.9 Commit and concurrency + +POST I/commit does not repeat operation_id/item_id in its body. The example digest illustrates syntax only; use the actual preview digest: + +```json +{ + "bundle_digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "accept_partial": false +} +``` + +bundle_digest is required; accept_partial is boolean, default false. ready accepts only false; needs_review requires explicit true or returns 409 partial_extraction. Target/title/origin/parser/expected Head are frozen and cannot be edited here. Digest mismatch is 409 bundle_conflict; Head mismatch is 409 revision_conflict even when text happens to match. Cancelled/unparsed is 409; expired uncommitted work is 410. + +One transaction writes all Sources, journal positions, dirty/pending state, Revision/Head CAS/lineage, bindings, quota, and durable receipt; post-commit notification failure remains recoverable. Within mapping retention, recommitting an already committed item with identical acceptance parameters returns its original receipt even after Head advances or operation expiry; changed acceptance parameters conflict. Reauthorize all replay. + +For version conflicts, read current Head, show differences, then create a new attempt with new expected Head, original bytes, and new key. Matching parsing cache may help, but never mutate old frozen results or overwrite concurrent content. This domain conflict protocol does not change generic PUT's 428/412 or GET ETag behavior. + +### 8.10 Shared commit receipt and standard references + +Text and file commits return ImportCommitReceipt with flat identity and standard SourceTypeReference/ArtifactReference lineage, not a Source/Artifact union or writable references: + +```json +{ + "scope_id": "project-docs", + "family": "importedcontent", + "artifact_id": "ic_001", + "revision": 1, + "outcome": "created", + "source_kind": "content", + "accepted_partial": false, + "snapshot_digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "sources": [{"source_type": "content", "source_id": "cs_001"}], + "artifacts": [], + "parts": [{"ordinal": 0, "source_type": "content", "source_id": "cs_001", "journal_position": 42}] +} +``` + +| Field | Type and meaning | +| --- | --- | +| scope_id, family, artifact_id, revision | Standard Artifact identity; exact accepted revision | +| outcome | created: new Artifact, HTTP 201; updated: new revision of existing Artifact, HTTP 200; reused: exact current revision, HTTP 200 | +| source_kind, accepted_partial | content and this acceptance decision, preserved on successful replay | +| snapshot_digest | Frozen bundle plus acceptance semantics; neither HTTP ETag nor standard content_digest | +| sources | Ordered standard `{source_type,source_id}` direct lineage, all content | +| artifacts | Standard `{family,artifact_id,revision}` direct Artifact lineage, [] if none; Scope from top level | +| parts | Ordered `{ordinal,source_type,source_id,journal_position}` mapping to Source bodies, matching sources/manifest one-to-one; pasted text has one part | + +Example digests show syntax only and cannot submit real files. No body text is returned; Location identifies the standard exact Revision followed by Source reads. Source HTTP identity remains source_type, not source_kind/name. Standard ArtifactRevision shape and content_digest calculation are unchanged. + +### 8.11 Pagination, errors, idempotency, and recovery + +New lists use limit (default 50, range 1–100) and opaque cursor. Session lists are Scope-bound by path; operations optionally filter status; items optionally filter selected/parse_status/commit_status. Responses use `{items:[],next_cursor:null}`, null at the end, without fabricated total. Bind cursors to caller, complete ancestry, filters, limit, high-water mark, and TTL; reauthorize every page. High water limits new records, not changing status, so filtered lists are not status snapshots. Invalid/mismatched cursors return 400; expired cursors return 410. Standard Artifact lists retain their shape, adding only importedcontent namespace filtering. + +Use the existing error envelope: + +```json +{ + "error": { + "code": "revision_conflict", + "message": "The document changed. Read its current revision before starting a new import.", + "details": {"artifact_id": "ic_001", "expected_head_revision": 2, "actual_head_revision": 3} + } +} +``` + +| HTTP status | Conditions | +| --- | --- | +| 400 | Invalid path/query/header, missing Idempotency-Key, invalid/mismatched cursor | +| 401 / 403 / 404 | Authentication, permission, visibility, or absence under existing non-disclosure policy | +| 409 | State, revision, bundle, selection, source-binding, profile, or idempotency conflict | +| 410 | Expired uncommitted session/operation work or cursor | +| 413 | Request/file size overflow known at admission | +| 415 | Unsupported format identified at admission | +| 422 | JSON shape, unknown body fields, invalid parameter combinations, unsupported Family capability | +| 429 | Admission quota/concurrency limit, with Retry-After | +| 503 | Required parser temporarily unavailable at admission, with Retry-After | + +Format/extraction/overflow/parser failures after 202 are operation/item errors; successful queries remain 200, not 500. ErrorDetail is `{code,message,details}` with null details when absent, never credentials/private blob addresses. Keep 412/428 for existing conditional requests; import commit does not borrow them. + +Import Idempotency-Key is a new durable domain protocol, not a claim that existing Source/Artifact Create is idempotent. Section 7 defines namespaces. Normalize after defaulting; file requests compare actual byte digest and metadata, not multipart boundaries. Same key/request replays original status and receipt; changed request returns 409. Identical raw replay retains originally frozen defaults despite deployment profile changes. Unaccepted 429/503 writes no success receipt; committed replay returns its original revision without a new write even after Head advances. Retention/tombstones/GC follow Sections 6–7. + +### 8.12 Clients, UI, and call sequences + +Python clients expose the listed operationIds and pass scope_id to every route. CLI provides text/files/resume/cancel/dry-run; new imports require an existing --scope. Resume --session obtains Scope from a caller-held session receipt or explicit --scope; both must agree if present. With neither stored destination nor explicit Scope, require --scope; do not add an unscoped server lookup. UI recovery stays within the current Scope; switching Scope does not migrate work. + +--artifact-id/--expected-head-revision explicitly update, and parser flags map to parser.options. Containers display the complete manifest before select; unattended use requires explicit selection rules. --accept-complete auto-commits only ready; partial needs separate explicit authorization including cached partial results. dry-run verifies local facts only, never claims complete importability without parsing. Back off on 429/503 using Retry-After rather than presenting unaccepted work as queued. + +The UI sends frozen bundle_digest and the original Idempotency-Key. Refresh/login only resumes reads, never confirms automatically. Resolve uncertain results via status/receipt or same-key replay. Clients measure upload progress; show indeterminate parsing unless the parser reports progress; never expose a half-commit state. + +| Scenario | Calls | +| --- | --- | +| Paste Markdown | GET P/import-capabilities → POST P/import-sessions `{}` → POST S/texts → standard Revision/Source reads | +| Scanned PDF | Create session → POST S/operations with parser.options.ocr_mode=auto → GET O/I/parts → POST I/commit → exact Revision | +| ZIP/mailbox | Upload → GET O until complete discovery → GET O/items → POST O/select → preview/commit per document, separate transactions | +| Update | Explicit artifact_id/expected_head_revision target → preview → commit → updated 200; old Head remains readable until success | +| HTML file | Same file flow, detected text/html; webpage provenance does not fetch a URL | +| Future webpage Connector | Separately validated authorized adapter submits a frozen snapshot; no URL import endpoint in this release | ### Published bodies, history, and comparison @@ -1140,14 +1394,6 @@ updates, rather than a duplicate search endpoint or implicit vector-ready promis Existing stable Artifact handles, if used, bind one exact immutable Revision. They are not live external-document identity or a replacement for source bindings. -### Examples - -1. Paste Markdown: select an existing Scope, create a session with its scope_id, then call importedcontent/texts with source_kind=content. Receive exact artifact_ref; read via standard Artifact get and Source parts. Failure leaves no partial Artifact. -2. Upload PDF: importedcontent/prepare returns operation_id for status/paged preview. Confirm operation_id, item_id, bundle_digest, accept_partial; read the resulting importedcontent Revision. PDF counts/OCR are extraction metadata, not another Family. -3. Update: explicitly supply artifact_id/expected_head_revision in prepare/texts artifact_target. Old Head stays readable during parsing; only confirmation creates a Revision. Failure/cancellation creates no failed version or governance change. -4. Upload HTML: source_kind=content, input_media_type=text/html. A webpage description in origin triggers no network access and changes no Source type. -5. Future webpage Connector: after separate validation, an authorized adapter supplies a frozen page snapshot and reuses content evidence, commit service, and Artifact reads. No link-import endpoint/UI is available now; source_kind=content never enables fetching. - ## 9. Continuous collection with OpenDAL Add `opendal-text-imports` while retaining the existing opendal-text-files/text-file-snapshot evaluation Definition, @@ -1251,7 +1497,7 @@ success. ### Implementation boundaries Implement Section 1's fixed parser families/routes, not ad hoc runtime library selection. Stage work by responsibility: -text persistence/unified manifest, Family prepare/confirm plus durable ledger/standard parsing package, then +text persistence/unified manifest, Family operation creation/commit plus durable ledger/standard parsing package, then OCR/long-tail/OpenDAL. Both A and B remain formal targets; A alone does not complete this RFC. Register importedcontent, operation storage, indexes without changing existing Source/Artifact references. Historical @@ -1285,6 +1531,8 @@ the feature; SQLite development auto-creation alone is not production support. ### Contract and acceptance +HTTP acceptance covers Scope paths and ancestry, repeated body identity rejection, standard source_type references and flat Artifact receipts, created=201/updated or reused=200, distinct 400/409/410/422 behavior, original 202 replay, asynchronous failures returned by successful GET 200, parse_status/commit_status mapping, and unchanged generic Artifact ETag/304/412/428 behavior. Both locale operation tables, defaults, examples, and OpenAPI must agree. The long infographic remains a separate explanatory artifact, not an RFC embed. + Use openapi/powercontext.yaml as source of truth, run make api-generate and make contract-test, never hand-edit generated code. Implementation validation includes make check, relevant Source/Server/Access/Dashboard/Connector tests, make docs-test, and tox when Python support changes. Cross-component acceptance belongs in tests/e2e/. RFC acceptance @@ -1345,7 +1593,7 @@ does not mean these feature tests have passed. | OpenDAL cached/recovered partial | complete-only rejects; no checkpoint advance merely because Revision exists | | New key with old expired/cancelled/failed/completed task | New operation, coverage-aware persistent reuse; old key retains old result | | Transient discovery failure/retries exhausted | stage=discovery works without item_ids; publish complete manifest atomically, retain frozen IDs | -| Retry slotless failed/partially_committed | 429/503 leaves state/counters unchanged; later same-key admission holds one slot and enqueues once | +| Retry slotless failed work, including partial commits | 429/503 leaves state/counters unchanged; later same-key admission holds one slot and enqueues once | | Awaiting partial confirmation/only failed members/one failure | Waiting confirmation holds slot; inactive failure may release; do not release other active resources or extant bytes | | Any part/record/pending write failure | Whole document rollback; recovery does not duplicate Sources | | Failed container members/explicit partial | Accurate per-document outcomes and durable omissions, no false completeness | @@ -1391,7 +1639,7 @@ guarantee all semantics. Validated adapters are more diagnosable than implicit f **Isolated parsing, reused ContentSource.** Keep dependencies and file risk outside Core requests; normalized evidence reuses Source/Scope/lineage/processing without a Source family per suffix. -**Asynchronous parsing, synchronous per-document commit.** OCR/large inputs exceed normal HTTP timeouts. prepare returns +**Asynchronous parsing, synchronous per-document commit.** OCR/large inputs exceed normal HTTP timeouts. Operation creation returns durable recovery handles; confirmation atomically commits frozen output instead of exposing partially parsed Sources. **Segmentation instead of truncation/manual splitting.** Natural boundaries/locations remain usable, exact diff --git a/docs/zh/rfcs/2073-text-material-import.md b/docs/zh/rfcs/2073-text-material-import.md index bc60af295a..7e31ca3789 100644 --- a/docs/zh/rfcs/2073-text-material-import.md +++ b/docs/zh/rfcs/2073-text-material-import.md @@ -76,7 +76,7 @@ flowchart TD A["选择已有 Scope"] --> P["校验 scope.read + scope.contribute；创建固定 scope_id 的 ImportSession"] P --> B{"输入方式"} B -->|粘贴文本| C["校验与规范化"] - B -->|上传文件| D["Family prepare 操作：耐久暂存与隔离解析"] + B -->|上传文件| D["文件操作创建：耐久暂存与隔离解析"] D --> E["预览正文、分段、OCR 与遗漏"] E -->|无可用正文| X["失败；不保存资料"] E -->|完整确认或显式接受 partial| F["确认冻结 bundle_digest"] @@ -128,7 +128,7 @@ Dashboard 保持既有 Tabler、Jinja2、HTMX、中英文及明暗主题。写 ## 1. 架构 -入口、解析和保存之间的责任边界见“用户流程”图。文件通过 ImportedContent Family 的 prepare 操作产生冻结的 ImportedContentBundle，执行过程由内部操作账本和隔离 worker 承担；用户确认或 worker 按显式策略接受后，ContentImportApplication 才执行单文档原子提交。制品处理仍沿用已有配置与进度机制。 +入口、解析和保存之间的责任边界见“用户流程”图。文件通过 ImportedContent Family 的文件操作创建接口产生冻结的 ImportedContentBundle，执行过程由内部操作账本和隔离 worker 承担；用户确认或 worker 按显式策略接受后，ContentImportApplication 才执行单文档原子提交。制品处理仍沿用已有配置与进度机制。 文件获取、解析、文本保存与制品生成是四个边界。Core 不导入任意用户指定 parser，不运行 Office 宏，不执行代码。正式部署提供独立 parser worker；轻量文本路径不要求文档解析依赖。 @@ -311,15 +311,15 @@ B 级表示目标交付范围而非无条件可用承诺。未达到必需语义 | --- | --- | | `GET /v1/server-info` | 保留部署身份和协议兼容信息；在 OpenAPI 的 `x-powercontext-feature-contracts` 中拟新增 `content.import: {major: 1, minor: 0}`，表示导入协议版本，不表示当前启用 | | `GET /v1/capabilities` | 拟新增可选 `content_import` 对象，包含 `enabled`、`readiness`（ready/degraded/unavailable）及 `capabilities_endpoint`；缺失时客户端不得假定支持导入。这是当前 Runtime 的汇总，不替代逐格式健康状态或 Scope 授权 | -| `GET /v1/import-capabilities?scope_id=...` | 返回指定已有 Scope 下、当前调用方可用的实际格式、profile、限额及权限；页面据此展示并决定是否允许开始 | +| `GET /v1/scopes/{scope_id}/import-capabilities` | 返回指定已有 Scope 下、当前调用方可用的实际格式、profile、限额及权限；页面据此展示并决定是否允许开始 | `content.import` 和 `content_import` 均为拟新增契约，须同步更新 OpenAPI、生成代码、Server 组装和客户端兼容测试。现有两个全局入口继续要求 `server.observe`；仅具有 Scope 权限的普通导入者可以直接查询 Scope 级导入能力，不将全局观测权限新增为导入前提。Scope 级响应也包含导入协议版本与是否启用，客户端不必靠试写探测支持情况。 -启用由导入服务配置控制，Dashboard 入口开关只控制页面写入口，不能代替 API 服务能力。`readiness=ready` 表示所有已启用且声明可用的 profile 均通过健康检查；部分失效为 degraded，仍可使用健康 profile；禁用或无可用 profile 为 unavailable。未实现的 planned profile 不拉低已启用能力的健康状态。关闭新增导入或解析器不可用，不影响已发布 ImportedContent 和精确 Source 的读取。能力查询是当前快照，prepare、retry 和 confirm 仍须重新检查相关权限、运行状态和冻结结果有效性；confirm 不因某个无关解析器离线而重新解析或丢弃已有冻结正文。 +启用由导入服务配置控制，Dashboard 入口开关只控制页面写入口，不能代替 API 服务能力。`readiness=ready` 表示所有已启用且声明可用的 profile 均通过健康检查；部分失效为 degraded，仍可使用健康 profile；禁用或无可用 profile 为 unavailable。未实现的 planned profile 不拉低已启用能力的健康状态。关闭新增导入或解析器不可用，不影响已发布 ImportedContent 和精确 Source 的读取。能力查询是当前快照，创建 operation、retry 和 commit 仍须重新检查相关权限、运行状态和冻结结果有效性；commit 不因某个无关解析器离线而重新解析或丢弃已有冻结正文。 ### 格式识别与 profile 能力 -新增 `GET /v1/import-capabilities`；必须携带指向已有 Scope 的 `scope_id` 并要求 `scope.read`；缺失或格式非法返回 422。响应展示是否具备 scope.contribute；开始导入仍须同时具备这两个权限。返回固定的 source_kind=content 以及每种格式的扩展名、规范 MIME、能力级别、状态（available/unavailable/planned）、parser profile、可提取内容、OCR/转换依赖及实际限额。状态来自服务注册和启动自检；不能在 API 上挂载解析器就宣称其 backend 健康。 +新增 `GET /v1/scopes/{scope_id}/import-capabilities`；必须携带指向已有 Scope 的 `scope_id` 并要求 `scope.read`；路径格式非法返回 400，不存在或不可见按 404/403 处理。响应展示是否具备 scope.contribute；开始导入仍须同时具备这两个权限。返回固定的 source_kind=content 以及每种格式的扩展名、规范 MIME、能力级别、状态（available/unavailable/planned）、parser profile、可提取内容、OCR/转换依赖及实际限额。状态来自服务注册和启动自检；不能在 API 上挂载解析器就宣称其 backend 健康。 识别顺序结合文件 magic、容器内部结构、声明 MIME 与扩展名。扩展名匹配不区分大小写，但来源路径身份保持原有大小写。扩展名仅是提示；伪装为 `.txt` 的 PDF 进入 PDF 识别或返回类型不一致，不直接解码。DOCX/XLSX/PPTX/EPUB 是专用文档容器，先按专用格式识别，不能当普通 ZIP 拆成一堆内部 XML。 @@ -348,7 +348,7 @@ flowchart TD | 对象 | 职责与持久化 | | --- | --- | | ImportSession | 创建者 Principal、固定且非空的已有 scope_id、期限；承载同一目标下的导入恢复、隔离及并发控制，不授予额外 Scope 权限 | -| 内部操作记录 / item_id | prepare 返回 operation_id；记录临时输入、发现清单、租约、配额、解析与确认状态；item_id 仅在 operation 内稳定，不是资料永久身份，也不是 Artifact | +| 内部操作记录 / item_id | 创建文件操作返回 operation_id；记录临时输入、发现清单、租约、配额、解析与确认状态；item_id 仅在 operation 内稳定，不是资料永久身份，也不是 Artifact | | ImportedContentBundle | 解析后冻结的正文、语义 manifest、coverage 与 digest；确认针对它，不重新计算预览内容 | | ImportedContent Artifact | 稳定身份 `(scope_id, family=importedcontent, artifact_id)`；首次手动导入或稳定来源首次出现时分配 | | ImportedContent Revision | 精确快照 `(scope_id, family, artifact_id, revision)`；含结构化元信息和有序 Source 映射 | @@ -367,7 +367,7 @@ flowchart TD | `origin` | 具体出处的 namespace、key 和可选 display_name，不作为授权依据 | | `parser` | 实际解析器、配置、版本、OCR 和转换链 | -ImportedContent Revision、冻结 bundle 和读取响应必须包含 `source_kind: "content"`。texts 与 prepare 请求可省略该字段，由服务端补齐；显式提供时只能为 content，其他值返回 422。先补齐固定值再计算请求指纹、冻结摘要和缓存键，因此省略与显式 content 在其他字段相同时语义一致。OpenAPI 使用单值约束，Python 模型使用 Literal["content"]；默认值不意味着允许调用方选择其他 Source 类型。 +ImportedContent Revision、冻结 bundle 和读取响应必须包含 `source_kind: "content"`。texts 与文件上传 metadata 可省略该字段，由服务端补齐；显式提供时只能为 content，其他值返回 422。先补齐固定值再计算请求指纹、冻结摘要和缓存键，因此省略与显式 content 在其他字段相同时语义一致。OpenAPI 使用单值约束，Python 模型使用 Literal["content"]；默认值不意味着允许调用方选择其他 Source 类型。 此字段不重命名或替代已有 `SourceRef.source_type`。保存时校验所有 parts/lineage.sources 的 `source_type` 均为 content，且与 ImportedContent 的 source_kind 一致；不能把其他 Source 类型或系统保留角色混入正文。标准列表不新增 source_kind 筛选或索引，因为该 Family 的取值恒定；出处仍可按 namespace 筛选。 @@ -385,7 +385,7 @@ ImportedContent Revision、冻结 bundle 和读取响应必须包含 `source_kin | `source_kind`、`origin` | source_kind 固定为 content；origin 包含必填 namespace、key 和可选 display_name（1–256 个 Unicode 字符，无控制字符）；仅作出处，不证明授权 | | `input_media_type`、`input_bytes`、`input_digest` | 原始输入类型、长度和 SHA-256；粘贴输入为接收的 UTF-8 字节 | | `text_media_type`、`text_bytes`、`text_digest` | text/plain 或 text/markdown、完整提取文本长度及摘要 | -| `parser` | selected_profile/profile digest、router_version、实际后端及版本、路由原因、转换链、输出 schema、规范化和分段版本、编码、OCR 模型/语言与实际执行信息；这些语义信息纳入冻结摘要 | +| `parser` | profile_id/profile_digest、router_version、实际后端及版本、路由原因、转换链、输出 schema、规范化和分段版本、编码、OCR 模型/语言与实际执行信息；这些语义信息纳入冻结摘要 | | `bundle_digest`、`snapshot_digest` | 冻结解析结果及其接受语义的摘要，不参与逻辑文档身份，不包含运行或保存后分配的 ID | | `content_manifest` | 有序 ordinal、字节数、分段摘要、全文字符区间和定位，不含 SourceRef、journal position 或临时 blob ID | | `parts` | ordinal 对应的 SourceRef 与 journal position，和 manifest 一一对应；SourceRef 顺序必须与 lineage.sources 完全一致 | @@ -422,7 +422,7 @@ Family 注册覆盖类型、序列化、读取、授权与允许的操作，复 | 能力 | 本期行为 | | --- | --- | | list/get、精确 Revision、历史、版本比较 | 支持；统一以所在 Scope 的 `scope.read` 授权 | -| texts/prepare/confirm 专用写入 | 支持；要求 `scope.read` + `scope.contribute`，并执行会话创建者隔离和事务内校验 | +| texts/operations/commit 专用写入 | 支持；要求 `scope.read` + `scope.contribute`，并执行会话创建者隔离和事务内校验 | | 通用 Artifact POST/PUT 创建或替换 | 不支持；不能绕过确认提交 parts、lineage 或成功状态 | | generic restore/rollback、merge/undo_merge | 不支持；不注册相应适配器、不加入允许列表，HTTP 与 Runtime 均拒绝，页面不展示动作 | | 模型生成、Dream/Candidate 发布 | 不支持；不注册 ImportedContent 生成或候选处理器 | @@ -501,7 +501,7 @@ ImportSession 从创建起即固定已有 scope_id。会话与内部操作记录 | 预算 | 每 Principal、每 Scope、整个部署的默认上限 | | --- | --- | | 暂存实际字节（原文件、展开文件、输出缓存合计） | 1 GiB、5 GiB、20 GiB | -| 活跃任务槽（按下表的实际工作状态计量，不只看 aggregate_status） | 20、100、500 | +| 活跃任务槽（按下表的实际工作状态计量，不只看公开 status） | 20、100、500 | | 同时运行的普通解析数 | 2、4、8 | | 同时运行的 OCR 数 | 1、2、4；同时占用普通解析并发槽 | | 每 UTC 自然日实际尝试的 OCR 页数 | 1,000、5,000、20,000 | @@ -514,8 +514,8 @@ ImportSession 从创建起即固定已有 scope_id。会话与内部操作记录 | --- | --- | | uploading、queued、running；含等待解析资源的队列 | 占用 | | idle 且成员发现完成等待选择，或有选中未提交的 ready/needs_review 文档 | 占用 | -| partially_committed 且仍有排队、解析、待选择或待确认工作 | 占用 | -| failed；或 partially_committed 仅剩失败文档且没有已安排的自动重试 | 不占用，显式 retry 时重新准入 | +| 部分已提交且仍有排队、解析、待选择或待确认工作 | 占用 | +| failed；包括部分已提交但仅剩失败文档且没有已安排的自动重试 | 不占用，显式 retry 时重新准入 | | completed、cancelled、expired | 不占用 | 自动重试已被安排的任务保持 queued 并持续占槽；自动重试耗尽进入 failed 后释放任务槽。释放任务槽不释放仍存在的暂存字节，也不清除可重试资格。任何操作将不占槽任务转回排队或待处理状态，必须在同一事务内重新申请 Principal、Scope 和部署三个层级的任务槽；多个文档一起 retry 仍只申请一个槽。已有槽的任务不得重复计费。配额不足返回 429/503 和 Retry-After，任务、文档、generation 与重试次数保持不变，不入队、不形成成功接收回执；同一请求可在退避后使用原幂等键再次准入。已经成功接收的同键重放仅返回回执，不重复申请槽。 @@ -526,7 +526,7 @@ Principal/Scope 配额不足返回 429，部署容量或依赖不足返回 503 ## 6. 内部执行、私有预览与耐久暂存 -复杂文件由 ImportedContent Family 的 prepare 操作接受，返回可恢复的 operation_id；执行状态保存在第 3 节的内部操作账本，由独立解析 worker 处理。ImportedContent Revision 仅管理已接受快照。本文下称“任务”均指该操作的内部执行工作，不引入 FileImportJob 领域实体或另一类 Artifact。 +复杂文件由 ImportedContent Family 的文件操作创建接口接受，返回可恢复的 operation_id；执行状态保存在第 3 节的内部操作账本，由独立解析 worker 处理。ImportedContent Revision 仅管理已接受快照。本文下称“任务”均指该操作的内部执行工作，不引入 FileImportJob 领域实体或另一类 Artifact。 可以复用经过验证的数据库事务、租约、fencing、背压与 worker 基础组件；不把现有 Artifact Processing Supervisor 的生成任务直接改作导入任务。现有生成调度不自然具备上传暂存、私有预览与人工确认语义；如抽取公共执行基础设施，必须保留本文状态、权限和恢复约束。解析成功只冻结 bundle，不直接触发 Source/Artifact 写入。 @@ -535,8 +535,8 @@ Principal/Scope 配额不足返回 429，部署容量或依赖不足返回 503 | 层级 | 字段与取值 | | --- | --- | | 任务执行 | execution_state：uploading / queued / running / idle / failed / cancelled / expired；idle 表示当前无待执行解析 | -| 任务阶段 | phase：upload / discovery / selection / parsing / confirmation / done；discovery_state：not_required / pending / running / complete / failed | -| 文档选择 | selected：是否纳入本次导入；容器发现完成后通过 selection API 冻结选择 | +| 内部任务阶段 | internal_phase：upload / discovery / selection / parsing / confirmation / done；discovery_state：not_required / pending / running / complete / failed | +| 文档选择 | selected：是否纳入本次导入；容器发现完成后通过 select API 冻结选择 | | 文档解析 | parse_state：pending / parsing / ready / needs_review / failed；ready 和 needs_review 都对应不可变 bundle | | 文档保存 | commit_state：uncommitted / committed；提交事务进行中不公开半提交状态 | @@ -565,9 +565,9 @@ flowchart LR B -->|确认 digest 并接受 partial| C ``` -任务另外返回派生 aggregate_status 与各类计数，按以下优先级计算：cancelled/expired；正在接收输入为 uploading；存在待执行或运行工作时为 queued/parsing；无活动工作且有已提交文档但未全部完成时为 partially_committed；选中非空且全数 committed 为 completed；存在未提交 ready/needs_review 或待选择成员为 awaiting_confirmation；任务级失败或仅剩不可解析成员为 failed。未选成员计入 excluded，不要求其提交。取消/过期仍返回已提交计数与 ImportedContent Revision 引用，不能隐藏部分成功。 +以上为内部耐久状态，不直接作为 HTTP 字段。公开 ImportOperation.status 由服务端派生，优先级为：已完成或已取消的终态保持；到期的未完成工作为 expired；存在执行工作为 running，否则存在已排队工作为 queued；完整发现后未选择为 awaiting_selection；选中非空且全部 committed 为 completed；存在未提交 ready/needs_review 为 awaiting_commit；任务级失败或仅剩失败成员为 failed。自动重试已安排的失败仍属于 queued。部分已提交只通过 progress 计数表达，不引入覆盖工作状态的 partially_committed。公开 phase 为 discovery/parsing/commit/done：发现及等待选择映射 discovery，正文处理映射 parsing，等待提交映射 commit，全部成功映射 done；失败/取消/过期保留最后阶段。内部 discovery_state=complete 映射公开 discovery.status=completed，parse_state/commit_state 分别映射 parse_status/commit_status。202 之前的 uploading 不公开为可恢复已接受操作。未选成员计入 excluded_items，取消/过期仍返回已提交计数与回执。 -普通单文档输入自动固定 selected=true。归档、邮箱或带附件文件先有界发现完整成员清单，返回稳定 item_id、container_chain、类型及默认选择建议；附件默认不选，普通归档成员默认选。发现后暂停，调用方通过 selection API 一次冻结明确的成员 ID 集合，再开始正文解析。超限或无法完成发现不能冒充完整清单；返回 discovery 错误。容器结构节点不可选择，未选附件记 excluded。选择不再改变；改变成员集合或解析选项须创建新尝试。临时 item_id 由本 operation 的持久成员映射产生，自动重试不重新分配。discovery_state=complete 与完整成员清单在同一事务中发布，只有此时才允许 selection。发现期间的中间清单不允许选择，不据其推进 checkpoint。临时故障令 discovery_state=failed 并返回阶段错误和 discovery_retryable；自动重试耗尽仍可通过显式 discovery retry 恢复。若完整清单尚未发布，重试可从原始暂存重新发现并复用内部成员身份；已发布清单不可重新发现、改写或重新分配 ID。损坏、加密、超限等永久错误标记不可重试。 +普通单文档输入自动固定 selected=true。归档、邮箱或带附件文件先有界发现完整成员清单，返回稳定 item_id、container_chain、类型及默认选择建议；附件默认不选，普通归档成员默认选。发现后暂停，调用方通过 select API 一次冻结明确的成员 ID 集合，再开始正文解析。超限或无法完成发现不能冒充完整清单；返回 discovery 错误。容器结构节点不可选择，未选附件记 excluded。选择不再改变；改变成员集合或解析选项须创建新尝试。临时 item_id 由本 operation 的持久成员映射产生，自动重试不重新分配。discovery_state=complete 与完整成员清单在同一事务中发布，只有此时才允许 selection。发现期间的中间清单不允许选择，不据其推进 checkpoint。临时故障令 discovery_state=failed 并返回阶段错误和 discovery_retryable；自动重试耗尽仍可通过显式 discovery retry 恢复。若完整清单尚未发布，重试可从原始暂存重新发现并复用内部成员身份；已发布清单不可重新发现、改写或重新分配 ID。损坏、加密、超限等永久错误标记不可重试。 原始字节耐久写入受控 staging store 并校验摘要后才返回 202 和排队任务。上传中断不留下可执行的半文件，上传占位及孤立 blob 由 GC 对账回收。单机可用受控状态目录，多副本使用共享耐久存储，不能依赖 API 进程的 /tmp。任务保存固定 profile、输入摘要、租约 owner/generation、attempt、deadline、选择版本及冻结 manifest。领取、结果发布、重试、取消和过期均以事务校验状态及 generation；过期 worker 不得覆盖新结果。 @@ -581,9 +581,9 @@ flowchart LR ### 会话与 Scope -ImportSession 创建必须提供已有 scope_id；服务端校验 Scope 存在且调用方具备 scope.read + scope.contribute 后，保存创建者和不可变 scope_id。该字段非空，不接受延迟解析的目标或创建参数。创建会话本身不创建 Scope、Artifact 或 Source，也不授予新权限。 +ImportSession 创建必须在路径提供已有 scope_id，body 只能为 {}；服务端校验 Scope 存在且调用方具备 scope.read + scope.contribute 后，保存创建者和不可变 scope_id。该字段非空，不接受延迟解析的目标或创建参数。创建会话本身不创建 Scope、Artifact 或 Source，也不授予新权限。 -会话、operation、item、暂存配额及最终 Artifact/Source 始终归属同一 Scope。子操作通过 session_id 获取固定目标，不允许请求正文覆盖 scope_id；其他 Scope 的 artifact_id、SourceRef 或来源绑定不能作为当前目标。切换目标需重新创建会话并重新提供输入；已有操作不转移。 +会话、operation、item、暂存配额及最终 Artifact/Source 始终归属同一 Scope。子操作同时校验路径 scope_id 与 session_id 的固定目标一致，不允许请求正文覆盖 scope_id；其他 Scope 的 artifact_id、SourceRef 或来源绑定不能作为当前目标。切换目标需重新创建会话并重新提供输入；已有操作不转移。 创建会话、提交文本、接收上传、选择成员、重试与确认均重新校验该 Scope 的当前权限；操作查询/预览要求会话创建者及当前 Scope 访问权限，管理代操作须显式授权并审计。确认事务还须验证 Scope 仍存在且可写、会话/操作未过期或取消，以及目标 Head/治理状态；撤权或目标失效后不保存资料，不自动补建 Scope 或选择其他目标。取消与清理按第 6、10 节执行。 @@ -617,7 +617,7 @@ bundle_digest = SHA-256(RFC 8785 canonical JSON)，输入为带 schema_version 耐久输入映射以明确逻辑文档目标、输入摘要、受控来源/title、profile 和完整选项为键；容器包含稳定成员映射与选择。它索引已冻结解析结果及精确历史 ArtifactRef，而不是授予覆盖 Head 的权限。缓存命中可以省略 OCR，但必须按本次目标与预期 Head 决定 reused 或创建新 Revision；不能直接把历史版本当当前版本。手动新建仍创建新文档，跨 Scope 不暴露命中。 -缓存 coverage=complete 可以复用解析结果，但交互文件仍需本次确认；明确配置 accept-complete 的 worker 才可自动确认。缓存 coverage=partial 时，本次 item 必须为 needs_review/uncommitted，展示 candidate_ref、coverage 与遗漏，不计入 completed；本次显式 accept_partial=true 后才可保存。不能继承其他 Principal 或上次尝试的部分提取接受决定。同一次已成功提交的同键重放保留原决定。 +缓存 coverage=complete 可以复用解析结果，但交互文件仍需本次确认；明确配置 accept-complete 的 worker 才可自动确认。缓存 coverage=partial 时，本次 item 必须为 needs_review/uncommitted，展示 reusable_artifact、coverage 与遗漏，不计入 completed；本次显式 accept_partial=true 后才可保存。不能继承其他 Principal 或上次尝试的部分提取接受决定。同一次已成功提交的同键重放保留原决定。 reparse=true 必须重新提供原文件并记录实际执行；运行诊断属于临时任务历史，Revision 保存影响内容的 provenance。解析缓存可以指向历史版本，但缓存指针不改变 Head；最近完整解析结果可更新缓存索引，旧版本继续精确读取。 @@ -639,73 +639,402 @@ Source metadata 中的目标 Revision 在事务内确定，但不进入摘要； ## 8. HTTP、客户端与页面契约 -持久资料复用标准 Artifact API，family 固定为 `importedcontent`。以下四个读取操作扩展现有 Family 注册与 schema，不新建独立资料查询资源： +### 8.1 规范、身份与接口清单 -| 现有 operationId | 方法与路径 | ImportedContent 行为 | +遵循 [RFC 1437](1437_source_artifact_rest_api.md) 的 Scope 资源身份、命名、错误结构、分页与标准 Artifact 读取契约。集合使用小写复数名词，静态多单词路径使用 kebab-case，JSON/query 与 operationId 使用 snake_case，URI 不带末尾斜线。`select`、`commit`、`retry`、`cancel` 是显式 POST 领域动作，不提供任意 PATCH/PUT 状态修改。所有下级路径校验 Scope、session、operation、item 的真实所属关系，不能仅凭最后一个 ID 查询。路径确定的身份不得在 body 重复传入。 + +本节描述待实施的公开契约；实现时以 `openapi/powercontext.yaml` 为唯一契约源，生成 Python/JavaScript 客户端并执行 contract-test。不得手工修改生成代码。Family 固定为 `importedcontent`，会话只接受已有 Scope，不创建 Scope。接口层级不引入独立 FileImportJob 或 ImportedContent 主表。 + +以下缩写只用于阅读，实际 HTTP 路径须完整展开，不能发送字面量 P/S/O/I： + +```text +P = /v1/scopes/{scope_id} +S = P/import-sessions/{session_id} +O = S/operations/{operation_id} +I = O/items/{item_id} +``` + +| operationId | 方法与路径 | 成功状态与含义 | +| --- | --- | --- | +| `get_import_capabilities` | `GET P/import-capabilities` | 200；当前调用者可见的格式、profile、权限与实际限额 | +| `create_import_session` | `POST P/import-sessions` | 201；创建固定 Scope、固定创建者的会话 | +| `list_import_sessions` | `GET P/import-sessions` | 200；当前 Scope 内创建者自己的恢复列表 | +| `get_import_session` | `GET S` | 200；会话及期限，不返回原文 | +| `import_content_text` | `POST S/texts` | 201 新建资料；200 更新或复用；请求本身即确认 | +| `create_import_operation` | `POST S/operations` | 202；单文件耐久接收并排队，尚未发布 Artifact | +| `list_import_operations` | `GET S/operations` | 200；操作恢复列表，不是资料列表 | +| `get_import_operation` | `GET O` | 200；状态、阶段、计数、期限及动作 | +| `list_import_items` | `GET O/items` | 200；成员清单，单文档也有一个 item | +| `get_import_item` | `GET I` | 200；冻结目标、提取结果、覆盖与问题 | +| `get_import_part` | `GET I/parts/{ordinal}` | 200；有界正文预览，已提交时读取 Source | +| `select_import_items` | `POST O/select` | 202；原子冻结容器成员选择并排队 | +| `commit_import_item` | `POST I/commit` | 201 新建资料；200 更新或复用；单文档原子提交 | +| `retry_import_operation` | `POST O/retry` | 202；重新准入后重试发现或指定失败文档 | +| `cancel_import_operation` | `POST O/cancel` | 200；取消未提交部分或返回已完成结果 | + +正式资料复用以下既有接口及 operationId；不另外建立“导入资料详情”资源。仅扩展 Family 注册、schema 和读取投影： + +| operationId | 方法与路径 | 行为 | | --- | --- | --- | -| `list_artifacts` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent` | 当前可见 Head 的资料列表；支持 namespace 精确过滤 | -| `get_artifact` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent/{artifact_id}` | 当前已接受快照及有序 parts | -| `list_artifact_revisions` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent/{artifact_id}/revisions` | 已接受版本历史 | -| `get_artifact_revision` | `GET /v1/scopes/{scope_id}/artifacts/importedcontent/{artifact_id}/revisions/{revision}` | 精确不可变快照，回执 Location 指向这里 | +| `list_artifacts` | `GET P/artifacts/importedcontent` | 当前可见 Head；新增 namespace 精确过滤，不是正文搜索 | +| `get_artifact` | `GET P/artifacts/importedcontent/{artifact_id}` | 标准 ArtifactRevision、正文 parts 与 Head ETag；保留 If-None-Match/304 | +| `list_artifact_revisions` | `GET P/artifacts/importedcontent/{artifact_id}/revisions` | 已接受的历史版本 | +| `get_artifact_revision` | `GET P/artifacts/importedcontent/{artifact_id}/revisions/{revision}` | 精确不可变版本；按既有契约不使用 ETag | + +通用 Artifact POST/PUT 不允许直接写入 importedcontent，返回 422 的不支持错误；不能绕过专用写服务校验 parts、lineage 和接受语义。其他 Family 的通用 Replace 仍使用 If-Match，缺失返回 428、不匹配返回 412。本 Family 导入 commit 是领域命令，使用冻结的 expected_head_revision，冲突返回 409；不把 Artifact ETag 当成 operation/item 的 HTTP 校验值。 + +### 8.2 公共字段、headers 与请求校验 + +| 项目 | 格式与约束 | +| --- | --- | +| `scope_id` | 沿用现有 Scope ID：1–256 字符且含非空白字符；作为单个 URL segment 编码 | +| `session_id`、`operation_id`、`item_id` | 服务端生成的不透明字符串，1–128 个可见 ASCII 字符；客户端不得解析前缀或假定 UUID | +| `artifact_id` | 沿用 Artifact ID 约束：1–128 个可见 ASCII 字符；必须属于路径 Scope 与 importedcontent Family | +| `revision` / `expected_head_revision` | 整数，最小 1 | +| `ordinal` | 整数，从 0 开始；第 n 段的 ordinal=n-1 | +| 时间 | RFC 3339 UTC，例如 `2026-10-19T08:00:00Z` | +| 摘要 | `^sha256:[0-9a-f]{64}$`；input、bundle、snapshot、text 摘要含义独立 | +| 大小 / 时长 | 字节整数使用 `_bytes`；秒整数使用 `_seconds` | +| 缺失与 null | 请求默认值只在字段省略时应用；未声明可空的字段拒绝 null。响应未知统计值为 null，0 表示确认没有 | +| `source_kind` | 请求可省略，先补为 `content` 再计算指纹；显式值只能为 content，保存及读取必须包含 | + +启用认证时携带 `Authorization: Bearer `。全部导入写请求必须有 `Idempotency-Key`（1–128 字符，ASCII 字母、数字及 `-_.`）；缺失或非法 header 返回 400。所有成功/失败响应返回 `X-PowerContext-Request-ID`。JSON 使用 UTF-8；不使用额外 code/message/data 成功信封。未知 body 字段返回 422，未知 query、非法路径/header/cursor 返回 400；重复 multipart part 或额外 part 返回 422。服务端拒绝非空白正文中的 NUL、非法 Unicode 和除 TAB/CR/LF 外的 C0 控制字符。 + +创建会话的 Location 指向 S；创建操作的 Location 指向 O；文本/commit 成功的 Location 指向精确 Artifact Revision。导入回执不是通用 Head 表示，不伪造其 ETag。返回 429/503 时包含秒数形式的 Retry-After。上传和动作的 202 返回接收时的 ImportOperation 快照，重放不替换成最新状态，调用者通过 GET O 查询变化。 + +### 8.3 当前部署能力与会话 + +`GET P/import-capabilities` 要求 scope.read，不要求 server.observe。Scope 不存在或不可见时沿用 404/403 策略。响应字段如下： + +| 字段 | 类型与含义 | +| --- | --- | +| `scope_id`、`family`、`source_kind` | 路径 Scope、importedcontent、content | +| `contract_version` | `{major:1,minor:0}`；与 content.import feature contract 一致，不表示当前可用 | +| `enabled`、`can_import` | boolean；部署是否启用、当前调用者是否具备所需写权限；不是后续授权凭证 | +| `readiness` | ready / degraded / unavailable；含义见第 2 节 | +| `limits` | 实际限额，包含 max_text_bytes、max_json_bytes、max_file_bytes、max_metadata_bytes、max_document_text_bytes、max_parts_per_document、max_part_bytes、max_selected_items、session_ttl_seconds、operation_ttl_seconds；还包含第 5 节按格式及累计配额的限制 | +| `formats[]` | format_id、extensions、media_types、level(A/B/C)、status(available/unavailable/planned)、default_profile_id（可空）、profile_ids、extraction_scope、limits、reason（可空） | +| `profiles[]` | profile_id、profile_digest、status、defaults、options_schema、extraction_scope、dependencies；options_schema 为有界的 JSON Schema 2020-12 子集，additionalProperties=false，不接受执行代码或任意 parser 类名 | +| `quotas` | 当前调用者允许查看的配额上限和剩余量；不泄漏其他 Principal 的使用情况 | + +extensions 含点，例如 `.pdf`；media_types 使用规范 MIME；reason 为 `{code,message}`。profile 摘要覆盖固定配置和依赖版本，不是发布时钟。能力响应是当前快照，每个写动作仍重新校验。初始 limits 来自第 5 节，例如 max_file_bytes=52428800、max_text_bytes=262144、max_document_text_bytes=8388608、max_parts_per_document=128、max_part_bytes=262144、max_metadata_bytes=65536；客户端不得硬编码为部署实际值。 + +创建会话调用示例： + +```http +POST /v1/scopes/project-docs/import-sessions +Content-Type: application/json +Idempotency-Key: session-notes-001 + +{} +``` + +请求体只能为空对象，不再接收 scope_id 或任何创建 Scope 参数。校验 scope.read + scope.contribute 后返回 201： + +```json +{ + "scope_id": "project-docs", + "session_id": "is_001", + "created_at": "2026-10-19T08:00:00Z", + "expires_at": "2026-10-20T08:00:00Z", + "idempotency_expires_at": "2026-11-19T08:00:00Z" +} +``` + +`GET S` 返回相同 ImportSession 字段；时间为服务端计算结果。会话创建重放映射保留到会话期限后至少 30 天，包括从未上传文件的会话；所有子操作期限不晚于会话期限。列表仅列举路径 Scope 内当前创建者可见的会话，不提供本期跨 Scope 会话发现入口。 -导入是 Family 专用写入行为。会话创建前必须选择已有 Scope，准备和确认通过 session_id 使用该固定目标；会话路由用于恢复同一批次操作，不另外创建可长期管理的 file-import 业务资源。下表操作均为本 RFC 拟新增契约，不把 operation 恢复查询声称为仓库已有接口。 +### 8.4 共用目标与来源 -| operationId | 方法与路径 | 含义 | +`artifact_target` 是带 mode 的互斥对象，默认 `{"mode":"create"}`，不从文件名或 origin 推断更新： + +| mode | 字段 | 规则 | | --- | --- | --- | -| `get_import_capabilities` | `GET /v1/import-capabilities?scope_id=...` | scope_id 必填且须已存在；返回固定 Source 类型 content、格式、权限与限额 | -| `create_import_session` | `POST /v1/import-sessions` | 必填已有 scope_id；校验贡献权限并固定目标及创建者 | -| `list_import_sessions` | `GET /v1/import-sessions` | 当前创建者可恢复的会话，分页 | -| `get_import_session` | `GET /v1/import-sessions/{session_id}` | 固定 scope_id、期限与回执摘要 | -| `import_content_text` | `POST /v1/import-sessions/{session_id}/importedcontent/texts` | source_kind=content；同步校验并确认保存有界正文，201/200 | -| `prepare_imported_content` | `POST /v1/import-sessions/{session_id}/importedcontent/prepare` | source_kind=content；metadata + 单文件，耐久接收后 202，返回 operation_id | -| `list_import_operations` | `GET /v1/import-sessions/{session_id}/operations` | 创建者的操作恢复列表，非资料列表 | -| `get_import_operation` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}` | 执行状态、错误、期限、计数与可执行动作 | -| `list_import_operation_items` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}/items` | 有界成员清单；单文件也有一个 item | -| `get_import_operation_item` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}/items/{item_id}` | manifest、coverage、warnings、omissions 及冻结目标 | -| `get_import_operation_part` | `GET /v1/import-sessions/{session_id}/operations/{operation_id}/items/{item_id}/parts/{ordinal}` | 创建者的有界分段预览；已提交时从 Source 读取 | -| `confirm_imported_content` | `POST /v1/import-sessions/{session_id}/importedcontent/confirm` | operation_id + item_id + bundle_digest + accept_partial，原子确认，201/200 | -| `select_imported_content` | `POST /v1/import-sessions/{session_id}/importedcontent/selection` | operation_id + 成员集合及各自目标；冻结选择后 202 | -| `retry_import_operation` | `POST /v1/import-sessions/{session_id}/operations/{operation_id}/retry` | 调用所属 Family 的受控重试动作，重新准入后 202 | -| `cancel_import_operation` | `POST /v1/import-sessions/{session_id}/operations/{operation_id}/cancel` | 封闭未提交写入，返回原子取消/已完成结果，200 | - -prepare 的 202 响应包含 `{operation_id, session_id, family: "importedcontent", expires_at, status_url}`，Location 指向操作状态 URL，不指向尚不存在的 Artifact。texts 的请求本身即确认；它与 confirm 返回相同的已保存回执，不要求额外创建解析操作。操作及成员不能通过通用 PUT/PATCH 改写状态；确认/选择处理器须校验 operation 与 session、创建者、Family 匹配，跨会话或不可见句柄按现有 404/403 策略处理。通用 Artifact POST/PUT 对 importedcontent 的直接写入按本提案的 Family 能力校验返回 422，不能退回通用命令 Source 路径。 - -创建会话请求为 `{"scope_id":"已有 Scope 的 ID"}`，scope_id 必填、非空且遵循现有 Scope ID 约束。缺失、空值、未知字段或任何创建 Scope 参数返回 422；目标不存在、不可见或无贡献权限时按现有 404/403 可见性规则拒绝，不建立会话或接收文件。成功响应包含 session_id、固定非空 scope_id 与 expires_at。所有写操作携带 Idempotency-Key；不提供修改会话目标的接口。 - -文档目标为 `{"mode":"create"}` 或 `{"mode":"update","artifact_id":"...","expected_head_revision":2}`。手动默认 create，不从 origin 猜测身份；update 必须属于会话固定 Scope 的 importedcontent Family；其他 Scope 的目标按现有可见性规则拒绝。来源采集使用第三种互斥目标 `{"mode":"source","source_authority_id":"...","source_key":"..."}`，仅允许已注册并获准写入该 authority 的 Connector Principal 使用；接收时解析绑定，冻结 artifact_id 和预期 Head/不存在断言。响应返回已解析目标，不能由任意客户端通过 origin 冒充 authority。容器对每个成员独立指定目标，禁止把多个成员更新到同一 artifact_id。 - -直接粘贴请求保持简单： +| create | 只有 mode | 本次明确新建意图；身份由服务端生成 | +| update | artifact_id、expected_head_revision，均必填 | 接收时冻结；提交时校验同 Scope/Family 与 Head，冲突不自动改目标 | +| source | source_authority_id、source_key，均必填 | 仅获准 Connector Principal；接收时解析已有绑定/不存在断言，冻结目标 | + +source_authority_id 为 1–128 个可见 ASCII 字符；source_key 为 1–1024 个 Unicode 字符，非空、无首尾空白或控制字符，并符合已注册 authority 的规范化规则。响应的 `resolved_target` 为只读 `{mode,artifact_id,expected_head_revision}`；mode 为 create/update，create 的 expected_head_revision 为 null。容器未选择成员的目标为 null。客户端不能提交 resolved_target、sources、artifacts、parts 或任意可写 lineage。 + +`origin` 为必填对象：namespace（1–128 字符）、key（1–1024 字符）必填，display_name（1–256 字符）可选；均无控制字符，namespace/key 无首尾空白。文件 key 为相对 POSIX 路径，禁止绝对路径、反斜线、空片段、`.`、`..`。origin 仅作出处，不授予来源绑定权限，也不触发远程获取。容器成员的 origin 和 container_chain 由服务端按第 3 节稳定成员规则生成并回显，成员序号不能伪装成跨版本稳定身份。 + +### 8.5 粘贴文本直接保存 + +```http +POST /v1/scopes/project-docs/import-sessions/is_001/texts +Content-Type: application/json +Idempotency-Key: text-notes-001 +``` ```json { "title": "接口约定", "artifact_target": {"mode": "create"}, "source_kind": "content", + "origin": {"namespace": "project-notes", "key": "api-conventions"}, "media_type": "text/markdown", - "origin": {"namespace": "project-notes", "key": "note-001"}, "text": "# 接口约定\n\n分页使用不透明游标。\n" } ``` -上传 metadata 包含 artifact_target、title、origin、parser_profile，以及可省略且固定为 content 的 source_kind、可选 encoding、as_text、reparse、ocr_mode、language、include_notes、include_comments、include_hidden 与 archive_depth。as_text=false、reparse=false、ocr_mode=off、include_notes=true、include_comments=false、include_hidden=true、archive_depth=1 为默认值；language 不指定时使用 profile 公布的固定语言集，encoding 不指定时按识别规则决定或返回 encoding_required。parser_profile 不指定时先冻结部署的 router/profile 注册表版本，再按识别出的格式选择并冻结对应默认 profile；PDF 的实际后端随后按该 profile 的固定规则解析。归档/邮箱同时冻结子文档路由注册表，成员发现与选择后按实际格式使用同一版规则，不能在中途采用更新的默认值。跨键重试客户端应传已解析的 profile digest。include_notes 仅对支持备注的格式生效，其他格式默认值忽略，不适用的显式非默认值返回 422。archive_depth 取 1 或 2。所有默认值均在能力接口公布。 +| 字段 | 必填 / 默认 | 含义 | +| --- | --- | --- | +| title | 必填 | 去首尾空白后 1–256 个 Unicode 字符 | +| artifact_target | 默认 create | 第 8.4 节目标 | +| source_kind | 默认 content | 固定 Source 类型 | +| origin | 必填 | 受控出处 | +| media_type | 默认 text/plain | 只接受 text/plain、text/markdown，描述保存的正文表示 | +| text | 必填 | 非空白字符串；UTF-8 字节数不超过 max_text_bytes，整个 JSON 不超过 max_json_bytes | + +该请求本身即确认，不创建必需的异步解析操作；成功返回第 8.10 节 ImportCommitReceipt。粘贴 HTML/代码按所选正文表示保存原文，不执行或隐式提取；需要格式解析时使用文件入口。此接口不接受 parser、URL 或服务器文件路径。 + +### 8.6 文件上传与解析参数 + +`POST S/operations` 使用 multipart/form-data，恰好一个 `file` part（原始字节）和一个 `metadata` part（application/json，最多 64 KiB）；不接受额外文件、URL 或服务器路径。客户端 filename 与声明 Content-Type 只作为检测线索，服务端按字节、MIME 与容器结构识别真实格式。 + +```json +{ + "title": "产品使用手册", + "artifact_target": {"mode": "create"}, + "source_kind": "content", + "origin": {"namespace": "product-docs", "key": "manual.pdf"}, + "parser": { + "profile_id": "pdf-standard-v1", + "options": {"ocr_mode": "auto"} + }, + "reparse": false +} +``` + +| metadata 字段 | 必填 / 默认 | 含义 | +| --- | --- | --- | +| title | 可省略 | 单文档标题，显式值遵守文本 title 约束；省略时使用规范化文件名，空名使用固定值 `Untitled material`，不随界面语言变化。容器只用作展示名，成员标题来自清单 | +| artifact_target | 单文档默认 create | 容器只接受省略或 create，update/source 由 select 为各成员单独指定；禁止套用一个目标到多个成员 | +| source_kind / origin | 默认 content / 必填 | 与文本接口相同 | +| parser | 可省略 | ParserSelection，字段见下表 | +| reparse | 默认 false | 本次新上传绕过解析结果缓存和跨键任务合并；不修改旧操作，不假设旧暂存仍存在 | + +| parser 字段 | 类型与默认 | 语义 | +| --- | --- | --- | +| profile_id | string，可省略 | 能力接口中可用 profile；省略时冻结 router 注册表并按格式选择默认 profile | +| profile_digest | SHA-256，可省略 | 提供时必须同时提供 profile_id，要求精确配置；不匹配返回 409 profile_conflict，不静默换版本 | +| options | object，默认空对象 | 仅传显式覆盖项；不接受 profile 未声明的选项 | +| options.encoding | string，可省略 | 能力接口公布的编码名称；省略时 BOM/严格 UTF-8 或返回 encoding_required | +| options.as_text | boolean，false | 仅严格文本验证后启用，不能绕过二进制与资源限制 | +| options.ocr_mode | off / auto / force，off | OCR 禁用、按需、强制；实际执行另外记录 | +| options.languages | string[]，可省略 | 非空、去重，元素为 profile 公布的语言标识；省略使用固定语言集，不猜测或在线下载语言模型 | +| options.include_notes | boolean，true | 纳入支持格式的备注 | +| options.include_comments | boolean，false | 纳入支持格式的批注 | +| options.include_hidden | boolean，true | 纳入支持格式的隐藏内容 | +| options.archive_depth | integer，1 | 仅允许 1 或 2 | + +客户端不适用的显式选项返回 422；profile 默认项只对适用格式生效。响应回显补齐默认值后的有效配置。容器冻结子文档 router/profile 注册表；请求覆盖项只作用于其适用成员，明确不适用任何已发现成员的覆盖项作为发现阶段参数错误报告，不能忽略。跨键要求固定 profile 时应同时传 profile_id/profile_digest。模型、转换和实际后端信息在 item 的 parser 结果中返回，客户端不能任意指定导入类名。 + +以下命令读取本地文件并上传到已选择 Scope 的会话；认证部署需另加 Authorization header。`metadata.json` 内容使用上例： + +```bash +curl --request POST \ + 'http://127.0.0.1:8000/v1/scopes/project-docs/import-sessions/is_001/operations' \ + --header 'Idempotency-Key: upload-manual-001' \ + --form 'file=@manual.pdf;type=application/pdf' \ + --form 'metadata=@metadata.json;type=application/json' +``` + +只有原始字节耐久保存、摘要校验、配额及队列/回执落库后返回 202 和 ImportOperation。上传中断不接受半文件；公开状态不包含 uploading，客户端自行展示上传进度。 + +单文档 202 响应示例（Location 为 `/v1/scopes/project-docs/import-sessions/is_001/operations/io_001`）： + +```json +{ + "scope_id": "project-docs", + "session_id": "is_001", + "operation_id": "io_001", + "family": "importedcontent", + "status": "queued", + "phase": "parsing", + "discovery": {"status": "not_required", "retryable": false, "error": null}, + "selection_version": 1, + "progress": {"discovered_items": 1, "selected_items": 1, "parsed_items": 0, "failed_items": 0, "committed_items": 0, "excluded_items": 0}, + "available_actions": ["cancel"], + "created_at": "2026-10-19T08:05:00Z", + "expires_at": "2026-10-20T08:00:00Z", + "idempotency_expires_at": "2026-11-19T08:00:00Z", + "blocked_reason": null, + "retry_after_seconds": null, + "error": null +} +``` + +### 8.7 操作查询、成员与预览 + +ImportOperation（POST 接收快照、GET 及列表项使用同一字段定义）： + +| 字段 | 类型与含义 | +| --- | --- | +| scope_id、session_id、operation_id、family | 路径身份与固定 importedcontent | +| status | queued / running / awaiting_selection / awaiting_commit / completed / failed / cancelled / expired；由服务端按第 6 节派生 | +| phase | discovery / parsing / commit / done；失败保留失败阶段，成功为 done | +| discovery | `{status,retryable,error}`；status 为 not_required/pending/running/completed/failed；error 可空，结构为 ErrorDetail | +| selection_version | 非负整数；容器未冻结为 0，选择后为 1；单文档自动固定为 1 | +| progress | discovered_items（发现未完成为 null）、selected_items（未选择为 null）、parsed_items、failed_items、committed_items、excluded_items；后四项为非负整数，parsed_items 指已冻结结果，包含已提交项，不得把各计数相加当总数 | +| available_actions | 当前允许尝试的 select/retry/cancel 数组；仅提示，执行时重新校验 | +| created_at、expires_at、idempotency_expires_at | RFC 3339 UTC；期限不因重试延长 | +| blocked_reason、retry_after_seconds | 可空的 `{code,message}` 与非负整数；说明已接受任务的排队阻塞 | +| error | 可空 ErrorDetail；任务级错误，不把单个成员失败伪装成 HTTP 查询失败 | + +内部 execution_state、task_slot_held、租约和 fencing generation 不直接成为公开状态。部分已提交通过 progress 表达；仍有活动工作时 status 为 queued/running，有待确认结果时为 awaiting_commit，仅剩失败成员时为 failed。取消或过期仍返回已提交数量，成员详情保留提交回执；整个操作不是跨文档事务。 + +ImportItem（GET I；GET O/items 的列表项包含同样的有界元数据，不含正文或完整 content_manifest）： + +| 字段 | 类型与含义 | +| --- | --- | +| scope_id、session_id、operation_id、item_id | 完整父子身份 | +| title、origin、container_chain、input_media_type | 识别后的标题、出处、成员链与 MIME；尚未识别的 MIME 为 null | +| selectable、selection_suggested、selected | boolean；结构节点不可选，建议不等于已经选择 | +| artifact_target、resolved_target | 选定意图与解析后的冻结目标；选择前可空 | +| parse_status | pending / parsing / ready / needs_review / failed | +| commit_status | uncommitted / committed | +| retryable | boolean；仅表示当前解析错误是否可重试 | +| parser | 可空；profile_id、profile_digest、router_version、options 及实际 backends/conversion 信息；已冻结时完整回显 | +| statistics | page_count、sheet_count、slide_count、table_count、part_count、text_bytes；非负整数或 null，未知/不适用不能当 0 | +| ocr | requested_mode、used（boolean/null）、processed_page_count（integer/null）、engine（string/null）、languages；未执行完的统计不冒充最终值 | +| coverage | complete / partial / unknown；尚无结果为 null | +| warnings、omissions | `{code,message,locator}` 数组；locator 可空。结构化问题参与语义摘要，展示消息不参与；禁止无界诊断堆积，无法有界表达时失败而非隐藏遗漏 | +| input_digest、bundle_digest | SHA-256 或 null；未完成阶段不伪造摘要 | +| reusable_artifact | 可空的标准 `{family,artifact_id,revision}`，Scope 来自顶层；候选缓存命中，不表示本次已保存，也不是 Candidate | +| receipt | 可空 ImportCommitReceipt；只在本次成功确认后存在 | +| available_actions、error | commit 动作提示数组、可空 ErrorDetail | + +`GET I/parts/{ordinal}` 按稳定 ordinal 逐段读，不返回所有正文。响应含 scope_id、session_id、operation_id、item_id、ordinal、text、media_type、text_bytes、text_digest、bundle_digest、locators。段最大 256 KiB，返回完整段，不静默截断；单文档最多 128 段。超限处理遵循第 4、5 节。ordinal 不存在返回 404，尚未解析完成返回 409；未提交预览过期为 410、取消为 409；已提交内容在操作映射保留期间从 Source 读取，此后使用标准 Artifact/Source 接口。 + +定位使用 kind 判别对象，例如 `{kind:"pdf",page_number:3}`、`{kind:"spreadsheet",sheet_name:"预算",cell_range:"A1:D20"}`、`{kind:"slide",slide_number:5}`、`{kind:"text",line_start:10,line_end:25}`。页、幻灯片和行从 1 开始，行范围含两端；全文字符区间使用 Unicode code point 的 `[start,end)`。若有 PDF bbox，必须同时提供 page_width/page_height、unit=pt、origin=top_left，bbox=[left,top,right,bottom]，不能混用像素与点。无法可靠定位时明确说明，不猜坐标。 + +### 8.8 容器选择、重试与取消 + +`POST O/select` 请求： + +```json +{ + "expected_selection_version": 0, + "items": [ + {"item_id": "ii_001", "artifact_target": {"mode": "create"}}, + {"item_id": "ii_002", "artifact_target": {"mode": "update", "artifact_id": "ic_002", "expected_head_revision": 3}} + ] +} +``` + +expected_selection_version 必填且只能为 0；items 必填、非空、去重、最多 1,000 项，每项 item_id 必填，artifact_target 默认 create。仅完整清单已发布且尚未冻结时接受，原子校验全部目标、禁止多个成员更新同一 Artifact，再冻结 selection_version=1 并排队。未知/其他操作成员返回 422 且不泄漏信息，已冻结后的不同选择返回 409。同键重放返回原始接收快照；单文档不需要也不允许重新 select。 + +`POST O/retry` 使用以下两种互斥 JSON： + +```json +{"stage": "discovery"} +``` + +```json +{"stage": "documents", "item_ids": ["ii_002", "ii_003"]} +``` + +stage 必填。discovery 禁止 item_ids，仅允许发现失败且 retryable、完整清单未发布、原始暂存有效的操作。documents 的 item_ids 必填、非空、唯一、最多 1,000 项，必须全部是本操作已选、失败、可重试且尚未冻结的文档；一个不满足即整请求拒绝，不部分重试。不能重试 ready/needs_review/committed，不改变输入/profile/目标或期限。手动重试不重置每阶段最多三次的自动重试计数。只有重新准入配额、队列和幂等回执原子写入后返回 202；429/503 表示本次未接收，可用原 key 退避重试。已接收的后台资源等待通过 GET O 的 blocked_reason 表达。 -as_text=true 只允许通过严格文本校验的输入，不能绕过二进制识别与资源限制；ocr_mode 取 off/auto/force，auto 仅处理需要 OCR 的页面，force 在预览中标明。reparse=true 通过新上传与新 Idempotency-Key 发起新尝试，客户端需重新提供原始文件；服务端不假定旧暂存尚在，也不改变旧冻结结果。附件由下面的成员选择操作决定。 +`POST O/cancel` 请求只能为 `{}`，原子封闭未提交写入并返回 200 ImportOperation。重复取消不重复执行，已完成返回 completed，已过期返回 expired tombstone，已发布 Artifact 不删除。select/retry 对 cancelled 返回 409，对 expired 返回 410。所有动作重新检查创建者、当前 Scope 权限和父子归属；管理员代操作遵循第 10 节。 -selection 请求 `{operation_id, expected_selection_version: 0, items: [{item_id, artifact_target}, ...]}`，仅在成员发现完成、选择未冻结且任务有效时接受；ID 必须来自同 operation 完整清单且可选、去重、非空、最多 1,000 个。服务端校验各成员目标、禁止同一次选择中重复指向同一 Artifact，原子冻结为版本 1 并排队；再次改变选择返回 409，未知/其他任务 ID 返回 422 且不透露外部内容。单文档无需该操作。retry 请求为必填 stage 的互斥结构：`{stage: "discovery"}` 或 `{stage: "documents", item_ids: [...]}`。discovery 分支禁止携带 item_ids，仅允许 discovery_state=failed、discovery_retryable=true、完整清单尚未发布且原始暂存仍有效的容器任务；接受后转为 discovery_state=pending、phase=discovery、execution_state=queued。documents 分支要求清单已完成或无需发现，item_ids 非空、去重、最多 1,000 个，只允许已选、失败且 retryable 的文档；接受后将指定文档改为 pending 并排队。非法结构返回 422，阶段或状态不匹配整请求返回 409，均不修改冻结结果。两个分支共用配额重新准入规则，只有原子获得所需任务槽且写入队列与幂等回执后才返回 202。cancel 请求空对象；第一次原子取消，重复取消返回相同回执；已 completed 返回原完成状态，已 expired 返回 tombstone，不删除 ImportedContent Revision。 +### 8.9 确认保存与并发 -上述三个写操作均要求创建者身份，以及会话固定 Scope 的 scope.read + scope.contribute，并携带 Idempotency-Key；重放保持原接收回执，可用 GET 查询后续状态。管理员代操作遵守第 10 节；selection/retry 对 cancelled 返回 409、expired 返回 410。操作响应返回 operation_id、session_id、family，并独立返回 phase、discovery_state、discovery_retryable、阶段错误、execution_state、aggregate_status、task_slot_held、selection_version、counts、expires_at、idempotency_expires_at 和 blocked_reason；文档响应返回 selected、parse_state、commit_state、retryable、coverage、warnings、omissions、bundle_digest，以及可选 candidate_ref 和 artifact_ref（均为含 scope_id 的精确 ArtifactRef）；artifact_ref 仅用于本次已成功确认的回执，candidate_ref 不表示本次已保存。客户端不推断混合 status。 +`POST I/commit` 不再在 body 重复 operation_id/item_id。示例摘要仅演示格式，实际必须使用预览返回的值： -每个上传只接收一个 file 和一个 metadata 对象；metadata 上限 64 KiB，未知字段拒绝， multipart 额外字段及文件拒绝，不能绕过单文件上限。任务查询与文档 manifest 不回传原始 blob 地址。 +```json +{ + "bundle_digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "accept_partial": false +} +``` -确认请求为 `{operation_id, item_id, bundle_digest, accept_partial}`；目标文档及 expected_head_revision 已在 item 上冻结，响应同时回显。请求携带 Idempotency-Key。当前 bundle 已变化返回 409，未提交文档过期返回 410，取消或未解析完成返回 409。已提交文档在任务映射保留期内重放提交返回耐久回执，即使任务随后到期；此后通过通用 Artifact 精确版本读取接口读取。ready 只允许 accept_partial=false；needs_review 必须为 true，否则返回 409。成功返回 `{outcome: created|reused, scope_id, artifact_ref, snapshot_digest, parts}`；created 包括首版及后续新 Revision，artifact_ref 携带 family=importedcontent、artifact_id、revision，Location 指向现有通用 Artifact 精确版本资源。新的导入响应统一使用 `parts[]`，粘贴正文为一段；不改变既有 Source API 的 response。 +bundle_digest 必填；accept_partial 为 boolean，默认 false。ready 只接受 false；needs_review 必须显式 true，否则 409 partial_extraction。目标、标题、origin、parser 和预期 Head 已冻结，commit 不接受修改。摘要不符为 409 bundle_conflict，目标 Head 不符为 409 revision_conflict；内容碰巧相同也不能跳过版本检查。已取消/未解析完成为 409，未提交结果已过期为 410。 -任务/成员/资料列表默认 limit 50、范围 1–100，SQL keyset 分页并冻结首请求高水位；cursor 绑定操作、调用方、稳定 session_id（会话路由）或 scope_id（资料路由）、过滤、limit、高水位及 TTL，每页重新鉴权。任务状态可随时间变化，列表不是状态快照。ImportedContent 资料列表复用 Artifact 列表并扩展 namespace 精确筛选投影，不伪装成原文全文搜索。 +一个提交事务内写全部 Source、journal positions、dirty/pending、Revision/Head CAS/lineage、来源绑定、配额和耐久回执；事务后通知失败仍能恢复。已提交 item 在映射保留期内以相同接受参数再次提交返回原回执，即使 Head 已推进或操作后来到期；不同接受参数返回冲突。每次重放重新授权。 -Python client 暴露对应操作；CLI 提供 text、files、resume、cancel 和 dry-run；新导入必须用 --scope 指定已有目标，缺失时在上传前报错；--session 可恢复此前已选择目标的同一批次，并显示会话固定的 Scope。若同时提供 --scope，必须与会话一致，否则在提交前拒绝；用 --artifact-id 与 --expected-head-revision 明确更新现有文档，并暴露 --as-text、--reparse 与成员选择参数。容器先列出成员并提交明确选择；无人值守必须配置成员选择规则。dry-run 只验证本地可确认事项；离线未运行解析器时必须将文件标为“需服务端解析”，不能宣称可完整导入。CLI 默认在当前部署解析后交互确认；显式 `--accept-complete` 可自动保存 ready 文档，partial 必须额外显式授权，命中已有 partial 快照同样适用，禁止默认悄悄接受。页面和 CLI 根据 phase/discovery_state 提供“重试成员发现”或“重试失败文档”，不要求用户为尚未发现的成员填写 item_ids；收到配额拒绝后按 Retry-After 退避，不将未接收请求显示为已排队。 +发生版本冲突时读取当前 Head、展示差异，以新 expected_head_revision、原文件和新 key 创建操作；可以复用匹配的解析缓存，但不能修改旧冻结结果或覆盖当前版本。该冲突协议仅适用于导入动作，不改变通用 Artifact PUT 的 428/412 和 GET 的 ETag 规则。 -上传任务的重新打开与预览为读操作；重新登录不自动提交。确认按钮携带 frozen digest 和原 key，结果不明时通过任务与回执核实或同键重试。上传进度、解析进度和 Source 保存进度分开表示；解析器不报告页进度时只显示处理中。 +### 8.10 统一提交回执与标准引用 -版本冲突不自动改变冻结目标。调用方读取当前 Head 后，以新的 expected_head_revision、原文件和新幂等键创建新的文件导入尝试；可命中已有解析缓存，但旧 item/回执不可改写。页面展示当前版本差异后再确认。通用 Artifact 写接口不得绕过导入服务任意改写 ImportedContent 的 provenance、parts 或 lineage。 +文本保存和文件 commit 返回 ImportCommitReceipt；身份平铺，lineage 复用标准 SourceTypeReference/ArtifactReference，不新增 Source/Artifact union 或可写引用： + +```json +{ + "scope_id": "project-docs", + "family": "importedcontent", + "artifact_id": "ic_001", + "revision": 1, + "outcome": "created", + "source_kind": "content", + "accepted_partial": false, + "snapshot_digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "sources": [{"source_type": "content", "source_id": "cs_001"}], + "artifacts": [], + "parts": [{"ordinal": 0, "source_type": "content", "source_id": "cs_001", "journal_position": 42}] +} +``` + +| 字段 | 类型与含义 | +| --- | --- | +| scope_id、family、artifact_id、revision | 与标准 Artifact 身份相同；revision 是本次实际接受的精确版本 | +| outcome | created：首次创建 Artifact，HTTP 201；updated：已有 Artifact 新 Revision，HTTP 200；reused：复用当前精确版本，HTTP 200 | +| source_kind、accepted_partial | 固定 content、此次接受语义；相同成功请求重放保持原值 | +| snapshot_digest | 冻结 bundle 与接受语义的摘要；不是 HTTP ETag，也不是标准 Artifact content_digest | +| sources | 标准 `{source_type,source_id}` 有序直接 lineage；全部为 content | +| artifacts | 标准 `{family,artifact_id,revision}` 直接 Artifact lineage；没有则 []；Scope 来自顶层 | +| parts | `{ordinal,source_type,source_id,journal_position}` 有序数组，正文在 Source；与 sources 顺序及 manifest 一一对应；粘贴为一段 | + +示例 digest 只表示语法，不能用于提交真实文件。回执不返回全文；精确内容由 Location 指向标准 Revision 再读取 Source。已有 Source 接口仍使用 source_type，不改成 source_kind 或 name；标准 ArtifactRevision 外形与 content_digest 计算规则不变。 + +### 8.11 分页、错误、幂等与恢复 + +新增列表 query 使用 limit（默认 50，1–100）与 cursor（不透明字符串）；会话列表已由路径限定 Scope，操作列表可按 status 过滤，成员列表可按 selected/parse_status/commit_status 过滤。响应为 `{items:[],next_cursor:null}`，末页 next_cursor=null，不返回虚假 total。cursor 绑定调用者、完整父路径、过滤、limit、高水位及 TTL；每页重新授权。高水位限制新增记录，不冻结可变状态，因此过滤结果不承诺状态快照。非法/不匹配 cursor 为 400，过期为 410。标准 Artifact 列表沿用现有结构，仅为 importedcontent 增加 namespace 精确过滤。 + +错误使用现有 envelope： + +```json +{ + "error": { + "code": "revision_conflict", + "message": "资料版本已变化，请查看当前版本后重新发起导入。", + "details": {"artifact_id": "ic_001", "expected_head_revision": 2, "actual_head_revision": 3} + } +} +``` + +| HTTP 状态 | 场景 | +| --- | --- | +| 400 | Path/query/header 格式错误、缺失 Idempotency-Key、非法/不匹配 cursor | +| 401 / 403 / 404 | 认证、权限、可见性或不存在；遵循现有策略，不能泄漏跨 Scope 资源 | +| 409 | 状态、版本、冻结摘要、选择、来源绑定、profile 或幂等冲突 | +| 410 | 会话/操作的未提交工作过期，或 cursor 过期 | +| 413 | 接收阶段可确定的请求/文件大小超限 | +| 415 | 接收阶段已识别的不支持格式 | +| 422 | JSON 结构、未知 body 字段、非法参数组合或 Family 能力不支持 | +| 429 | 准入配额或并发限制，携带 Retry-After | +| 503 | 准入时所需解析能力暂不可用，携带 Retry-After | + +202 之后发生的格式、提取、超限或解析错误写入 operation/item.error；成功查询仍为 200，不伪装成 HTTP 500。ErrorDetail 复用 `{code,message,details}`，details 无内容时为 null，不包含凭据或私有 blob 地址。412/428 保留给既有条件请求，本期导入 commit 不借用它们。 + +导入的 Idempotency-Key 是此领域新增的耐久协议，不声称既有 Source/Artifact Create 已幂等。第 7 节定义键空间；补齐默认值后规范化请求，文件按实际字节摘要及 metadata 比较，不按 multipart boundary 比较。同 key 同请求返回原状态码和接收/提交回执，不同请求返回 409。相同 raw 请求重放沿用首次冻结默认值，不受部署 profile 更新影响。未接收的 429/503 不写成功回执；已提交重放即使 Head 改变也返回原版本，不产生新 Revision。期限、tombstone 与 GC 见第 6、7 节。 + +### 8.12 客户端、页面与调用顺序 + +Python client 公开与表格一致的 operationId，并将 scope_id 显式传递到所有路由。CLI 提供 text、files、resume、cancel、dry-run；新导入必须先提供已有 --scope。恢复 --session 时从调用方保存的会话回执读取 Scope 或显式提供 --scope，二者存在时必须相同；没有本地目标信息必须要求 --scope，不增加绕过 Scope 的服务器查询。页面当前 Scope 内恢复会话，切换 Scope 不迁移旧任务。 + +CLI 的 --artifact-id 与 --expected-head-revision 明确更新，解析参数映射到 parser.options；容器先展示完整清单再 select，无人值守必须显式配置选择规则。--accept-complete 仅自动提交 ready，partial 还需单独明确授权，包括缓存命中的 partial。dry-run 只验证本地事实，未运行解析器不能宣称文件可完整导入。收到 429/503 按 Retry-After 退避，不显示为已接收。 + +页面提交按钮发送冻结 bundle_digest 与原 Idempotency-Key；刷新/重新登录只恢复查询，不自动确认。结果不明时读取状态/receipt 或同键重放。上传进度由客户端计量；解析器未报告页进度时显示处理中；保存不公开半提交状态。 + +| 场景 | 调用顺序 | +| --- | --- | +| 粘贴 Markdown | GET P/import-capabilities → POST P/import-sessions `{}` → POST S/texts → 标准 Revision/Source 阅读 | +| 扫描 PDF | 创建会话 → POST S/operations（parser.options.ocr_mode=auto）→ GET O/I/parts → POST I/commit → 精确 Revision | +| ZIP/邮箱 | 上传 → GET O 等完整发现 → GET O/items → POST O/select → 逐文档预览与 commit；各文档独立事务 | +| 更新已有资料 | artifact_target 指定 artifact_id/expected_head_revision → 解析预览 → commit → updated 200；旧 Head 在成功前始终可读 | +| HTML 文件 | 同文件流程，服务端识别 text/html；origin 中的网页说明不触发抓取 | +| 未来网页 Connector | 经独立验收的授权适配器提交冻结快照；本期没有 URL 导入接口 | ### 已发布正文、历史与比较 @@ -717,14 +1046,6 @@ Python client 暴露对应操作；CLI 提供 text、files、resume、cancel 和 若使用既有稳定 Artifact handle，只能绑定某个精确不可变 Revision；它不代表随 Head 更新的外部文档身份，不替代第 3 节的稳定来源绑定。 -### 使用举例 - -1. 粘贴一段 Markdown：先选择已有 Scope，以 {"scope_id":"项目 Scope ID"} 创建会话后调用 importedcontent/texts，source_kind=content。成功返回精确 artifact_ref；页面通过标准 Artifact get 与 Source parts 阅读，操作失败不留下半份制品。 -2. 上传 PDF：调用 importedcontent/prepare，source_kind=content；返回 operation_id 后查询解析状态及分页预览。确认时提交 operation_id、item_id、bundle_digest 和 accept_partial；成功后读取 family=importedcontent 的 Revision。PDF 页数和 OCR 只属于提取元数据，不产生新 Family。 -3. 更新已有资料：prepare/texts 中 artifact_target 显式填写 artifact_id 与 expected_head_revision。解析期间仍可读取原 Head；确认成功才产生新 Revision。失败或取消不创建失败版本，也不改治理状态。 -4. 上传 HTML：source_kind=content，input_media_type=text/html。即使 origin 中附有来源网页说明，也不会触发网络读取；来源说明不改变 content Source 类型。 -5. 将来接入网页：经独立 Connector 验收后，由获准适配器提供冻结页面快照，归一后的文本仍使用 source_kind=content，复用同一 ImportedContent 提交服务与 Artifact 阅读契约。本期未提供链接采集接口，页面不展示可用的链接导入入口；仅填写 source_kind=content 不会启用网络抓取。 - ## 9. OpenDAL 持续采集 新增 `opendal-text-imports` 模式，保留原 `opendal-text-files` / `text-file-snapshot` evaluation 的 Definition、名称、权限和 checkpoint 行为。新模式通过公开导入 API 提交，不调用每次会注册 manifest 的旧 RemoteConnectorWorker 路径，向已有 Scope 导入无需 `server.admin`；worker 必须配置已有 scope_id，并具备 scope.read + scope.contribute；缺失目标或权限时拒绝运行，不自动创建 Scope。worker 使用同样的 ImportSession 协议。 @@ -759,7 +1080,7 @@ OCR 默认采用本地引擎和已安装语言包。云 OCR 或视觉模型需 ## 11. 错误与可观测性 -保留既有 error envelope，并区分 unsupported_format、parser_unavailable、capability_insufficient、encoding_required、invalid_encoding、encrypted_document、damaged_document、ocr_required、no_extractable_text、parse_timeout、input_limit_exceeded、output_limit_exceeded、archive_limit_exceeded、partial_extraction、idempotency_conflict、revision_conflict、source_binding_conflict、session_expired、bundle_conflict、selection_conflict、operation_cancelled、operation_expired 与 quota_exceeded。 +保留既有 error envelope，并区分 unsupported_format、parser_unavailable、capability_insufficient、encoding_required、invalid_encoding、encrypted_document、damaged_document、ocr_required、no_extractable_text、parse_timeout、input_limit_exceeded、output_limit_exceeded、archive_limit_exceeded、partial_extraction、profile_conflict、idempotency_conflict、revision_conflict、source_binding_conflict、session_expired、bundle_conflict、selection_conflict、operation_cancelled、operation_expired 与 quota_exceeded。 HTTP 401/403/404 沿用可见性策略；非法字段 422、单请求硬上限 413、不支持媒体类型 415、幂等/状态冲突及已取消操作 409、过期的未提交内容操作 410、限流 429、依赖不可用 503。202 后发生的解析失败记录在 operation 中，通过 200 状态查询返回 failed 与错误明细，不能伪造先前上传请求的 HTTP 错误。网络超时始终可能意味着结果未知，同键核实。 @@ -771,7 +1092,7 @@ HTTP 401/403/404 沿用可见性策略；非法字段 422、单请求硬上限 4 ### 实施边界 -交付采用第 1 节固定的解析族与分支，不再将“确定每种格式使用何种解析器”留给运行环境选择。交付按职责推进，但 A、B 格式范围都是正式目标：先实现文本保存与统一 manifest，再实现 Family prepare/confirm、内部耐久操作账本和标准解析包，再完成 OCR/长尾适配及 OpenDAL。不能仅完成 A 就宣称整个多格式 RFC 已实现。 +交付采用第 1 节固定的解析族与分支，不再将“确定每种格式使用何种解析器”留给运行环境选择。交付按职责推进，但 A、B 格式范围都是正式目标：先实现文本保存与统一 manifest，再实现 Family 文件操作创建/commit、内部耐久操作账本和标准解析包，再完成 OCR/长尾适配及 OpenDAL。不能仅完成 A 就宣称整个多格式 RFC 已实现。 注册 importedcontent Family、补充操作账本及查询索引，不改变既有 Source/Artifact 引用。历史 Source 不自动包装为 ImportedContent Revision。更新 family allowlist、精确读取、历史、lineage 及 Dashboard 阅读器；不得把 ImportedContent 自动纳入模型生成或 Candidate 流程。旧 OpenDAL binding 不原地换新模式；显式重新导入可生成新的 content Source，不能声称保留旧类型身份。Core Python 支持范围保持不变，较重解析依赖由独立 worker runtime 承担。 @@ -794,6 +1115,8 @@ HTTP 401/403/404 沿用可见性策略；非法字段 422、单请求硬上限 4 ### 契约与验收 +HTTP 验收须覆盖 Scope 路径与父子归属、body 重复身份拒绝、标准 source_type 引用和平铺 Artifact 回执、created=201/updated或reused=200、400/409/410/422 分工、幂等 202 原回执重放、GET 200 的异步失败、parse_status/commit_status 映射，以及通用 Artifact 的 ETag/304/412/428 行为未被导入命令改变。中英文操作清单、参数默认值、请求示例与 OpenAPI 必须一致；长图只作单独说明材料，不嵌入 RFC。 + 实现以 `openapi/powercontext.yaml` 为契约源，执行 `make api-generate`、`make contract-test`，不手改生成代码；运行 `make check`、相关 Source/Server/Access/Dashboard/Connector 测试及 `make docs-test`。影响 Python 支持范围时运行 `tox`，跨组件验收放入 `tests/e2e/`。本 RFC 被接受不表示验收已通过。 | 场景 | 验收要求 | @@ -851,7 +1174,7 @@ HTTP 401/403/404 沿用可见性策略；非法字段 422、单请求硬上限 4 | OpenDAL 命中已保存 partial 或恢复 partial 回执 | complete-only 拒绝自动完成，checkpoint 不推进，不因存在 ImportedContent Revision 绕过覆盖检查 | | 相同输入以新键上传，旧任务 expired/cancelled/failed/completed | 不合并到旧任务；新 operation 独立创建，持久记录按 coverage 规则复用；同旧键重放仍返回旧结果 | | 成员发现阶段临时失败且自动重试耗尽 | 无 item_ids 也可 stage=discovery 重试；完整清单原子发布，已冻结清单及成员 ID 不变 | -| 已释放任务槽的 failed 或 partially_committed 重试 | 配额满时 429/503，状态和计数不变；额度恢复后同键准入只占一槽，不重复入队 | +| 已释放任务槽的 failed（可含部分已提交）重试 | 配额满时 429/503，状态和计数不变；额度恢复后同键准入只占一槽，不重复入队 | | partial 候选等待确认、容器仅剩失败成员、单文档失败 | 待确认持续占槽；无活动工作的失败可释放任务槽；其他成员的活动槽与暂存计量不受错误释放 | | 写入中任一分段/记录/pending 故障 | 整份文档回滚，恢复后不重复 Source | | 容器部分成员失败、用户确认 partial | 逐文档结果真实，遗漏长期保留，不冒充完整导入 | @@ -882,7 +1205,7 @@ HTTP 401/403/404 沿用可见性策略；非法字段 422、单请求硬上限 4 **隔离文件解析，复用 ContentSource。** 解析依赖和文件处理风险不进入 Core 请求进程；归一后的文本继续使用已有 Source、Scope、lineage 和 Artifact 处理，无需按每个后缀新增 Source family。 -**文件异步解析、单份资料同步提交。** OCR 与大文件无法保证在普通 HTTP 超时内完成；Family prepare 返回操作句柄，内部耐久执行记录承担恢复责任。冻结结果后单文档事务可以可靠确认，避免边解析边公开部分 Source。 +**文件异步解析、单份资料同步提交。** OCR 与大文件无法保证在普通 HTTP 超时内完成；创建文件操作返回操作句柄，内部耐久执行记录承担恢复责任。冻结结果后单文档事务可以可靠确认，避免边解析边公开部分 Source。 **分段保存而非截断或强迫用户手工拆文件。** 自然边界和定位保留可用性，精确拼接和全量摘要保护完整性。原子保存保证一份文档不出现仅前几页已消费的半提交状态，但不强制所有 Artifact 在同一轮消费整份文档。 From ed3886ac6dea10a32dd8dc133adfbb44bd53565e Mon Sep 17 00:00:00 2001 From: Frey <370805886@qq.com> Date: Tue, 27 Oct 2026 20:33:53 +0800 Subject: [PATCH 4/4] docs(rfc): integrate source and artifact plugins for text import --- docs/en/rfcs/2073-text-material-import.md | 1141 +++++++-------------- docs/zh/rfcs/2073-text-material-import.md | 783 ++++++-------- 2 files changed, 662 insertions(+), 1262 deletions(-) diff --git a/docs/en/rfcs/2073-text-material-import.md b/docs/en/rfcs/2073-text-material-import.md index a6f26dea78..f77dcdceec 100644 --- a/docs/en/rfcs/2073-text-material-import.md +++ b/docs/en/rfcs/2073-text-material-import.md @@ -9,49 +9,27 @@ title: Text Material Import - Design Baseline: [PowerContext master / cab054db (2026-10-19)](https://github.com/oceanbase/powercontext/tree/cab054dbf6d86e2311b079388e5158178257def2) - Related RFCs: [Source Definition and Observation](1400_source_definition_and_observation_model.md), [Source and Artifact REST API](1437_source_artifact_rest_api.md), [Source access and generation eligibility](1458_artifact_generation_source_access.md), [Scope and Source discovery](1502_scope_source_discovery.md) +- Plugin Baselines: [Source #2210](https://github.com/oceanbase/powercontext/pull/2210) (`6ef5c494d701c355686806b6b1042d5729267690`), [Artifact #2246](https://github.com/oceanbase/powercontext/pull/2246) (`9fe2e68220a6e0aa8ebe4b78900cd12421db5cc3`). + # Summary -Provide text material import for text, office documents, saved webpages, spreadsheets, email, ebooks, source code, -images, and archives. Users must select an existing Scope with `scope.read` and `scope.contribute` before pasting text, -uploading files, or using an independent OpenDAL worker to collect files from directories or object storage. Format -detection, decoding, parsing, optional OCR, and bounded segmentation produce immutable `content` Sources and an -`ImportedContent` Artifact (`family=importedcontent`), with an exact Revision receipt. +Provide import of text, office documents, webpage files, spreadsheets, email, ebooks, code, scans and archives. Users must select an existing Scope with `scope.read` and `scope.contribute`, paste or upload input, inspect a private parsed preview, and confirm. Continuous collection uses Source collect plugins and the same parsing components. -The product contract combines broad format coverage with explicit capability levels. Lightweight text can be saved -directly; complex files use isolated parser workers; scans and images require explicit OCR enablement. The standard -document package targets PDF, modern and legacy Office, OpenDocument, HTML, email, and ebooks. Deployments expose actual -formats, parsers, and limits. Detecting a format or extracting only metadata is not successful body import. +Confirmation has two durable stages. T1 atomically saves all `content` Sources for one document, an immutable commit manifest, publication intent and acceptance receipt. T2 lets the official ImportedContent Artifact plugin deterministically assemble the Artifact, while the framework atomically publishes its Revision, Head, lineage, declared indexes and completion receipt. T1 creates no Artifact. Evidence can be saved while publication is pending; `202` does not mean import completion. Completion requires T2 success or explicit reuse of a published revision. -One logical document has one ImportedContent Artifact. `source_kind` is always `content`; each newly accepted snapshot -becomes an immutable Revision whose exact ContentSource references hold the body. Success means the imported Artifact -and its evidence were saved, not that downstream Memory generation, binary archival, or full-text indexing completed. -Head is the latest version accepted by PowerContext, not necessarily the current external file. Continuous collection -does not propagate deletion. This RFC proposes contracts; it does not claim released functionality. +One logical document has one `ImportedContent`, with family `importedcontent` and fixed `source_kind=content`. Body text lives only in Sources; the Artifact stores document metadata, parser provenance, omissions and ordered exact references. No LLM or Candidate approval is required. There is no separate ImportedContent main/version table or FileImportJob business resource. -Saved materials reuse Artifact tables, standard reads, Revisions, and lineage. ImportedContent-specific operations own -writes. Progress and private previews belong to a reclaimable internal operation ledger, not Artifact Revisions or a -separate FileImportJob business resource. +This RFC separates existing plugin capabilities, required framework extensions and import-specific responsibilities. Both plugin PRs are dependency baselines, not claims of merged or complete import support. Complete-document input assembly, durable handoff and transactional completion are release prerequisites. Parser profiles expose actual available/unavailable/planned deployment status; recognition alone is not successful extraction. The overview infographic remains a separate deliverable, outside this RFC. # Motivation -Project materials extend beyond TXT and Markdown to Word, PDF, Excel, PowerPoint, webpage exports, email, and scans. An -extension allowlist alone cannot decode binary formats, recognize scans, preserve table relationships, segment long -documents, or explain partial extraction and recovery. - -Existing ContentSource, Scope authorization, Source journals, text projections, Source collection/exact reads, and the -OpenDAL evaluation Connector's durable receipts and checkpoint CAS provide persistence foundations, not general file -parsing. +Source plugins unify acquisition, targets and runtime management. Artifact plugins unify generation, revisions, reads and declared search. Import reuses those frameworks while retaining upload, private preview, explicit confirmation, complete-document boundaries and recovery. A Source journal window is not a complete document, and a plugin checkpoint is not a user acceptance receipt. -The existing OpenDAL evaluation uses `text-file-snapshot` and registers a Definition manifest. Its UTF-8 collection -behavior, permissions, and public reads do not define this proposal's support contract. This design reuses the built-in -`content` Source and adds an import application service and optional parser deployments without changing existing Source -Create/Capture APIs. - -Users need to know which formats their deployment can read, what was extracted or omitted, the destination Scope, and -how to recover from retries and failures. +Parse before saving evidence so accepted text matches the preview. Artifact generation consumes accepted exact evidence without downloading the original or repeating OCR. Memory and other downstream processing remain independently configured; neither evidence acceptance nor ImportedContent publication promises their completion. # Guide-level explanation + ## Supported formats and capability levels The following matrix is the target. A is the base capability, B the standard document package, and C optional OCR or @@ -95,55 +73,28 @@ local HTML or exported Markdown does not implement URL fetching. ## User flow -Users select an existing contributable Scope first. Global entry points require explicit selection; an entry inside a -Scope may preselect it but must display and authorize it. Without a valid destination, creating an ImportSession, -pasting, and uploading are disabled. When none is available, direct users to the existing Scope management flow. The -import UI neither offers Scope creation nor creates one automatically. +An entry inside a Scope may preselect it; a global entry requires an existing Scope. If none is available, direct users to existing Scope management. Import does not create Scopes. Sessions bind one Scope and creator; changing the target requires a new session. -After selection, show actual formats, limits, and caller permissions. Creating an ImportSession fixes a nonempty -`scope_id` and its creator. All inputs in that session belong to that Scope. Changing destinations requires a new -session; parsing or unconfirmed inputs cannot migrate. - -Validated plain text can be confirmed immediately. Files are uploaded and parsed first, then previewed with -page/sheet/slide counts, segment counts, actual OCR use, and omissions. Confirmation rechecks Scope and write -permission, and saves one document's Sources, ImportedContent Revision, Head, lineage, and receipt in one transaction. -Failure, cancellation, or preview alone publishes nothing. +Display available formats, limits and parser/publication readiness first. A text request itself confirms its input. Files require a preview of text, page/sheet/slide counts, parts, OCR and omissions before confirmation. Partial extraction requires explicit acceptance. Cancellation, expiry and unconfirmed previews create no official Sources. ```mermaid flowchart TD - A["Select an existing Scope"] --> P["Check scope.read + scope.contribute; create ImportSession with fixed scope_id"] - P --> B{"Input"} - B -->|Paste text| C["Validate and normalize"] - B -->|Upload file| D["File-operation creation: durable staging and isolated parsing"] - D --> E["Preview body, parts, OCR, and omissions"] - E -->|No usable body| X["Fail; publish nothing"] - E -->|Confirm complete or explicitly accept partial| F["Confirm frozen bundle_digest"] - C --> G["Reauthorize; atomically save one document in the session Scope"] - F --> G - G --> I["ContentSource parts + ImportedContent Revision + lineage + receipt"] - I --> J["Subsequent files retain the Scope; configured downstream processing"] + A["Select existing Scope"] --> B["Paste / upload"] + B --> C["Shared parsing; private file preview"] + C --> D["Confirm frozen result"] + E["Source collect plugin"] --> F["Acquire; same parsing; complete-only policy"] + F --> D + D --> T1["T1: Sources + manifest + publication intent + receipt"] + T1 --> Q["Evidence saved; publication pending"] + Q --> P["ImportedContent Artifact plugin"] + P --> T2["T2: Revision + Head + lineage + completion receipt"] + T2 --> V["Import complete; standard Artifact reads"] ``` -Ordinary imports require `scope.read` and `scope.contribute`, not `server.admin`. A session is not an authorization -grant. Revocation, a missing destination, or an unwritable target prevents confirmation; it cannot silently choose -another target. Administrative operations follow Section 10. - -A first manual upload or paste defaults to creating a new document and allocating `artifact_id`. Updating requires the -existing document's update action, explicit `artifact_id`, and expected Head Revision. Matching filenames, similar -bodies, or identical digests never merge identities automatically; duplicate suggestions require user choice. Renaming a -file does not alter an explicitly selected identity. - -Show each file as awaiting upload, parsing, awaiting confirmation, saved, already present, needs attention, version -conflict, or outcome uncertain. Keep filename, selected Scope, and create/update intent visible. Concurrency still obeys -Scope-level quotas, idempotency, and Head checks. +Distinguish parsing, awaiting confirmation, evidence saved/publication pending, publication blocked/failed, and import complete. After T1, evidence is visible in the Scope and eligible downstream processors may consume it. New documents do not appear in Artifact lists until T2. Updates leave the old Head readable until T2 succeeds. -For example, Word may save successfully while a scanned PDF without OCR needs attention and Excel contains formulas -without cached values. Identify affected PDF pages and offer a new OCR attempt or explicit partial acceptance. Display -Excel formulas and “no saved calculation result,” without invented values. Do not report all three as fully extracted. +Names and digests never automatically merge documents. Manual creation assigns a new identity; updates explicitly specify artifact_id and expected_head_revision. At most one nonterminal publication commit exists per document. Recover that commit after failure instead of uploading again. Publication failure or cancellation of remaining input does not roll back accepted evidence. -Use “view extracted text” for parsed formats and “view original text” for plain text. Show supported handling and -omissions for images, layout, charts, and embedded objects. Neither parsing completion nor preview visibility means -Source or Artifact persistence has occurred. ## Format-specific behavior @@ -155,51 +106,31 @@ Source or Artifact persistence has occurred. - **PDF/OCR:** choose native text or OCR per page/region without double counting. Distinguish blank, image-only, and failed pages. OCR is machine recognition, not an exact transcription guarantee. Unavailable layout/page counts are unknown, not proof of completeness. - **Email/archives:** retain quoted history; select one plain-text or HTML alternative and record the rule. List attachments by default but import only selections. Expand the first archive layer by default; nesting requires explicit enablement and depth limits. Keep failed/unselected members in the manifest. -## Saving and downstream processing - -One logical document has one ImportedContent Artifact; accepted snapshots are Revisions. Long bodies become multiple -immutable ContentSources. All parts, the Revision, Head, lineage, pending processing, and receipt commit together in the -session's existing Scope, without creating a Scope or initializing permissions. Different files, archive members, and -messages may succeed independently. - -Success requires neither an LLM nor Candidate approval. Existing processor configuration and domain contracts determine -whether Memory and other processors consume these Sources, how much they consume, and their outputs. ImportedContent -registers no default generation processor or recursive generation behavior. - -Historical Revisions and Sources remain immutable when Head changes. Details may show history and downstream references -based on actual exact SourceRefs or ArtifactRefs; import cannot prepopulate future related Artifacts. A consumer window -need not consume every part, so do not claim full-document understanding. - -Original binaries are parser staging, not a permanent archive. Persist extracted text, digests, parser versions, -locations, and omissions; binary downloads and archival are outside this release. - -## Continuous collection and Dashboard +# Reference-level explanation -An OpenDAL worker near the storage holds read credentials and submits bytes. The Server does not read arbitrary -user-supplied filesystem paths or URLs. Text and complex files use the same capability matrix; an external scheduler -owns periodic execution. +## 1. Architecture and plugin reuse boundaries -Dashboard retains Tabler, Jinja2, HTMX, both languages, and light/dark themes. -`POWERCONTEXT_SERVER_DASHBOARD_TEXT_IMPORT_ENABLED` explicitly enables its write entry point and defaults to false, -preserving static Bearer and enforced-access prerequisites. File parsing also requires its service; disabling it does -not block saved-material reads. +| Capability | Evidence and status | Use in this feature | +| --- | --- | --- | +| Source discovery, loading, Scope authorization, write/collect routing | Reuse #2210 | Admin deployment and ordinary Scope operations; discovery is not parser support | +| collect instances, targets, independent Runtime, lifecycle and checkpoints | Reuse #2210 | OpenDAL becomes a collection plugin; no parallel instance manager | +| Synchronous `content.inline` | Exists in #2210; not the complete-import path | Single-Source capture does not replace document confirmation | +| Artifact Family/schema, ArtifactDraft, Worker and lifecycle | Reuse #2246 | Official importedcontent plugin generates deterministically, without an LLM | +| Revisions, Head CAS, lineage and standard reads | Reuse #2246 and existing Artifact persistence | No separate document version system; explicitly integrate Scope-shared authorization | +| Declared indexes, construction and queries | Reuse #2246 when declared for a deployment | Initially only fields stored in content, such as title/origin; no automatic Source-body dereference | +| Upload, controlled staging, private preview, parsing and confirmation | New in this RFC | Isolated file workers; retain import-specific APIs | +| `document_commit` input, exact part reads and evidence validation | Required framework extension | Replaces journal-window triggering for this family; not an existing InputContract field | +| Durable T1 handoff, transactional T2 completion, receipts and recovery | New integration in this RFC | Connect two transactions; an in-memory notification is insufficient | +| Public collect adapter for import submission and checkpoint acknowledgement | Required Source integration extension | Do not access the private Context client or treat run_batch as document-atomic submission | -The page flow is existing Scope selection, material list, text/file selection, parsing preview and issues, confirmation, -then exact reading. Scope navigation cannot retarget running operations or submit pending inputs. Accepted parsing -operations resume after browser closure; incomplete uploads require reselecting the local file. +Pinned evidence: [Source write](https://github.com/oceanbase/powercontext/blob/6ef5c494d701c355686806b6b1042d5729267690/src/powercontext/source_management/write.py), [Source Context](https://github.com/oceanbase/powercontext/blob/6ef5c494d701c355686806b6b1042d5729267690/src/powercontext/source_management/context.py), [Artifact SDK](https://github.com/oceanbase/powercontext/blob/9fe2e68220a6e0aa8ebe4b78900cd12421db5cc3/src/powercontext/artifact_plugins.py), [Artifact Worker](https://github.com/oceanbase/powercontext/blob/9fe2e68220a6e0aa8ebe4b78900cd12421db5cc3/src/powercontext/builtin/runtime/artifact_plugins.py). These establish the reusable baseline; the new contracts require separate implementation and acceptance. -# Reference-level explanation +Source write executes synchronously inside Server and submits one observation: add JSON is limited to 1 MiB and an observation to 4 MiB. It is not the large-file/OCR path. Custom write plugins cannot override a built-in Definition, so registering content again cannot implement this feature. collect can run remotely; current local collect process management requires Linux, not a claimed macOS/Windows deployment. -## 1. Architecture +Artifact generation currently uses journal windows, a 1 MiB input budget and direct Source evidence limited to the invocation. Artifact reads in GenerationContext do not provide arbitrary historical Source reads. The 8 MiB/128-part document contract needs document_commit input support; increasing max_sources or bypassing validation is insufficient. A trusted Server adapter assembles it without changing other families' window budgets. -The user-flow diagram separates input, parsing, and persistence. ImportedContent file-operation creation produces a frozen -ImportedContentBundle through an internal ledger and isolated workers. Only user confirmation or a worker's explicit -acceptance policy invokes ContentImportApplication's per-document atomic commit. Existing Artifact processing retains -its configuration and progress semantics. +The importedcontent binding uses only document_commit mode, never automatic consumption of every content Source. New plugins start disabled; deployment must load and explicitly enable the plugin and verify schema, permissions, document budgets and adapter readiness. Install parser dependencies at deployment time. Complex parsing runs separately; the Artifact plugin consumes exact saved evidence only. -Acquisition, parsing, text persistence, and Artifact generation are four boundaries. Core does not import arbitrary -user-selected parsers, execute macros, or run code. Production provides independent parser workers; lightweight text -does not require document-parser dependencies. ### 1.1 Parser responsibilities and selected components @@ -486,70 +417,45 @@ options return 422. Required semantics found unsupported after identification fa `accept_partial` cannot bypass identification, safety, size, encryption, or user-required semantics. Only profiles declaring permitted partial output may produce needs_review when usable body and explainable omissions exist. -## 3. Data model +Publication readiness is reported separately. Source capability discovery does not establish file-format support. If document_commit input support, the ImportedContent plugin or its registered schema is unavailable, reads and private previews remain available but confirmation returns 503 publication_unavailable. Transactional admission resolves races with plugin transitions; already accepted commits wait durably. -Add the `ImportedContent` type and `importedcontent` Family (display name “Imported content”), written by -ContentImportApplication. One logical document is one Artifact; accepted snapshots are immutable Revisions. Do not -create an independent ImportRecord as a second identity/content authority. This Artifact represents accepted content, -not upload work, parsing intermediates, or generated Memory. +## 3. Data model and persistence + +ImportedContent is the document identity; Session/operation/item identify input attempts. commit_id identifies one accepted immutable snapshot and its publication handoff. Commit status is not a second document catalog, arbitrary CRUD resource or Artifact Revision. ```mermaid -flowchart TD - I["ImportSession: creator and fixed, nonempty existing scope_id"] --> J["Internal operation / item_id: transient parsing and confirmation"] - J --> B["ImportedContentBundle: frozen content"] - B -->|Atomic commit| R["ImportedContent Artifact Revision"] - S["Scope"] --> H["Artifact Head: latest accepted version"] - H --> R - R -->|Ordered lineage.sources| C["ContentSource 1..N: body and locations"] - D["Downstream Artifact Revision"] -->|Exact SourceRefs actually consumed| C - D -.->|ArtifactRef only when the document Revision was an actual input| R +flowchart LR + S["Existing Scope"] --> I["Private Session / Operation / Item"] + I -->|T1 confirmation| C["Immutable document commit manifest"] + C --> P["Ordered Content SourceRefs"] + C --> Q["Durable publication state and intent"] + Q -->|Plugin T2| R["ImportedContent Revision"] + R -->|Exact lineage| P + H["Artifact Head"] --> R ``` -| Object | Responsibility and persistence | -| --- | --- | -| ImportSession | Creator Principal, immutable nonempty existing scope_id, expiry; recovery/isolation/concurrency for that destination; no new authorization | -| Internal operation / item_id | file-operation creation returns operation_id; staging, discovery, leases, quotas, parsing, confirmation. item_id is stable only within its operation, not permanent identity or an Artifact | -| ImportedContentBundle | Frozen body, semantic manifest, coverage, digest; confirmation uses it without recalculating preview | -| ImportedContent Artifact | Stable `(scope_id, family=importedcontent, artifact_id)`, allocated for manual creation or a first stable source | -| ImportedContent Revision | Exact `(scope_id, family, artifact_id, revision)` with structured metadata and ordered Source mapping | -| Artifact Head | Latest accepted Revision, changed with expected_head_revision optimistic concurrency; not a live external version | -| Source bindings/request mappings | Stable source key → Artifact, parsing cache, idempotency key → exact receipt; indexes/ledgers, not duplicate ImportRecord | +The following are logical table responsibilities. Implementation freezes actual SQL names, constraints and migrations. No business table is added per file extension. -### Naming and source semantics +| Persistent structure | Ownership | Responsibility and constraints | +| --- | --- | --- | +| `pc_sources`, Source journal, existing processing intents | Reuse Core | Immutable bodies, exact references and downstream work registration | +| `pc_artifacts`, `pc_artifact_heads`, Source/Artifact lineage tables | Reuse Artifact | Sole published document and version authority | +| Family/schema, binding/lease/cursor, index definitions/state/projections | Reuse #2246 | No duplicate plugin registration, scheduling or indexing tables; its eight logical framework tables are not import-specific additions | +| collect instance, target and checkpoint state | Reuse #2210 | No duplicate lifecycle; not private previews or complete-document receipts | +| `pc_import_sessions` | New import responsibility | Creator, fixed Scope and expiry | +| `pc_import_operations` | New import responsibility | Durable input, discovery, selection, leases, quotas and parsing state | +| `pc_import_operation_items` | New import responsibility | Unique `(operation_id,item_id)`; frozen parse result/target and commit_id, not permanent body copies | +| `pc_import_source_authorities` | New / connection configuration integration | Trusted connection, fixed root/bucket, allowed Principals and Scope; origin cannot grant authority | +| `pc_import_source_bindings` | New import responsibility | Unique `(scope_id,authority_id,source_key)` to artifact_id; digest indexes retain collision checks against the full key | +| `pc_import_request_receipts` | New import responsibility | Caller, operation, normalized fingerprint, original HTTP status/body for exact replay | +| `pc_import_quota_ledger` | New import responsibility | Shared Principal/Scope/deployment reservations and usage, CAS and reconciliation | +| `pc_import_commits` | New import responsibility | commit_id, Scope, artifact_id, frozen target/expected Head, accepter, full manifest, acceptance semantics and SourceRefs; immutable after T1 | +| `pc_import_publications` | New / framework durable handoff integration | Unique commit_id; state, generation, leases, retries and completed ArtifactRef; document reservation permits one nonterminal commit per target | -The domain/Python type is ImportedContent; persistence and HTTP wire name are `importedcontent`. `source_kind=content` -says its body uses ContentSource, not whether it was pasted, uploaded, or acquired from a webpage. +There are nine new logical import persistence responsibilities. Any physical reuse of quota/connection tables must be explicit in the migration manifest without dropping constraints. Commit manifests contain no duplicate body, only recovery and exact input assembly data. Pending publications are not deleted by staging TTL. Retain a lightweight commit_id-to-ArtifactRef result with the revision so callers and collect can verify outcomes after short-lived request receipts expire. + +source_kind remains content for pasted text, files and future webpages; entry point, MIME, origin and parser carry distinct information. Omitted and explicit content normalize identically; other values return 422. SourceRef still uses source_type. Saved evidence and published documents are Scope-readable; private operations require their creator or an explicitly authorized administrator. -| Field/concept | Meaning | -| --- | --- | -| `family=importedcontent` | Whole-material identity, versions, segment relationships | -| `source_kind=content` | Constant in ImportedContent metadata; every body part is a content Source | -| `input_media_type` | Original format, e.g. text/markdown, application/pdf, text/html | -| `origin` | namespace, key, optional display_name; provenance, not authorization | -| `parser` | Actual parser, configuration, version, OCR, conversion chain | - -Revisions, frozen bundles, and read responses require `source_kind: "content"`. texts requests and upload metadata may omit it; the Server -supplies it before fingerprints, frozen digests, and cache keys. Explicit other values return 422. Thus omitted/explicit -content are semantically identical given equal other fields. OpenAPI uses a single-value constraint and Python -`Literal["content"]`, not caller-selectable types. - -This does not replace `SourceRef.source_type`. Every parts/lineage Source must be content and match source_kind; no -other types or reserved roles in the body. No constant-value source_kind filter/index is needed; namespace filtering -remains useful. - -Input endpoints/authorized Connectors determine acquisition; format/profile determine extraction. HTML/PDF/TXT, pasted -text, and future webpage snapshots all become content evidence, but binary parsing and Connector-based URL acquisition -remain necessary. No arbitrary URL acquisition endpoint is included. - -Authorized Connectors generate/validate connection identity, source_authority_id, and stable bindings. Caller-provided -origin is not proof of authority. Future webpage Connectors supply page identity, acquisition time, and sanitized -display URL; URLs are neither arbitrary server-fetch instructions nor inherently stable identities. Do not persist -credentials, signed URLs, or sensitive query parameters. container_chain records archive/email ancestry; bodyless -containers are not ImportedContent. - -source_kind enters bundle_digest and indirectly snapshot_digest, but does not choose routing, artifact_id, bindings, or -cache partitions. Updates still check expected_head_revision and complete snapshot semantics for reuse/new Revision. A -document can change from file to pasted text while every version retains content plus its own origin/format/parser. ### Revision content @@ -566,12 +472,7 @@ document can change from file to pasted text while every version retains content | `coverage`, `warnings`, `omissions`, `accepted_partial` | Completeness within declared scope and explicit acceptance; preserve known omissions | | `container_chain` | Ancestor digests, member paths/ordinals, provenance; separate from cross-snapshot stable member identity | -Sources alone retain body text; do not duplicate it in Artifact content. Source metadata includes ImportedContent -artifact_id/revision, part ordinal, source_kind, controlled origin, and location. Persistence-assigned IDs/SourceRefs do -not enter content digests; source_kind/origin/locations already entered the frozen bundle. Create ordinary captured -ContentSources with `working_note=False`, `internal=None`, and `handoff_receipt=False`. Reject lineage_only, handoff -receipts, working-note metadata, and reserved roles. content constrains type, not generation eligibility; apply existing -type/role validation and reject forged internal metadata rather than silently changing its meaning. +Body text is stored only in immutable Content Sources. Source metadata records artifact_id, commit_id, ordinal, source_kind, origin and locations; it must not predict a future Artifact revision or be rewritten after publication. The Artifact content also records commit_id. Captures must remain ordinary content: working_note=False, internal=None and handoff_receipt=False; reserved roles or metadata are rejected. namespace is at most 128 characters; key at most 1024; both nonempty, without outer whitespace or controls. File keys are relative POSIX paths; reject absolute paths, backslashes, `.`, `..`, empty components; do not case-fold. Do not @@ -579,6 +480,8 @@ store credentials, signed URLs, or local absolute paths. origin is required and CLI/UI may generate visible labels, but cannot match existing Artifacts by filename. A pasted-text key is a provenance label, not a stable binding. +The Source metadata and digest rules use commit_id rather than a future Artifact revision. The completed commit status provides the exact ArtifactRef; publication never mutates Source metadata. + ### Logical document identity | Input | Stable identity | @@ -605,83 +508,6 @@ Unique stable-binding constraints prevent concurrent first collection from creat manual creation has no filename/digest uniqueness. Equal content may belong to distinct documents; cache deduplication cannot change identity, authorization, or origin. -### Existing tables, registration, and reads - -Use `(scope_id, family=importedcontent, artifact_id)` as the only material identity. Reuse pc_artifacts for -content/immutable Revisions, pc_artifact_heads for accepted Head, pc_artifact_lineage_sources for ordered SourceRefs, -and pc_sources for body. Use exact pc_artifact_lineage_artifacts references only for actual upstream Artifact inputs. No -new ImportedContent master/version table or independent ImportRecord. - -Register type, serialization, reads, authorization, and allowed operations using existing registries, profiles, and -route adapters. Adding an enum alone is insufficient; a new global plugin framework is unnecessary. Update OpenAPI -family schemas, Server capabilities, and Dashboard mappings: - -| Capability | This release | -| --- | --- | -| list/get, exact Revision, history, comparison | Supported under Scope `scope.read` | -| texts/operations/commit | Supported with scope.read + scope.contribute, creator isolation, transactional validation | -| Generic Artifact POST/PUT | Unsupported; cannot bypass confirmation with forged parts/lineage/success | -| Generic restore/rollback, merge/undo_merge | Unsupported; no adapter/allowlist entry; HTTP and Runtime reject, UI hides actions | -| Model generation, Dream/Candidate publication | Unsupported; no ImportedContent generation/candidate processor | -| Independent Artifact sharing/owner authorization | Unsupported; published content is Scope-visible; creator is audit information | -| Full-text/semantic Artifact search | Unsupported initially; no searcher; namespace filtering is not body search | - -ImportedContent is shared within its Scope. Generic reads currently use artifact.read for most Families and writes have -additional policies. Adapt HTTP access declarations/resolution, runtime authorization profiles, standard list/exact -reads, Source reads, and UI actions consistently. Do not expose list entries whose detail still needs Artifact ownership -or copy Prompt's administrator-only write rules. Creator isolation applies to unpublished inputs/recovery, not permanent -ownership. Revoked Scope readers cannot use old receipts to read bodies. Governance retains separate authorization; -scope.contribute does not implicitly grant it. - -Generic restoration builds Artifact lineage from historical content without guaranteeing this Family's ordered -lineage.sources. Do not enable it. Future dedicated restoration must validate complete parts/exact readable Sources and -define evidence reuse/creation, provenance, Head CAS, and downstream pending/dirty. Generic PUT or history buttons -cannot act as restoration in this release. - -Published list/current/history/exact reads reuse Artifact APIs; compose the body from exact SourceRefs in parts using -existing Source reading. Origin filtering may use rebuildable projections/indexes. Reverse lineage reports actually -referenced parts/versions, not assumed full-document consumption. - -### Minimal internal operation storage - -Durable asynchronous state is not a second material model. The following are proposed storage contracts, not claims that -a reusable generic job table already exists: - -| Storage | Responsibility/constraints | -| --- | --- | -| `pc_import_sessions` | Creator, fixed nonempty existing scope_id, expiry; validate contributable destination at creation; immutable target | -| `pc_import_operations` | operation_id, session_id, family, operation kind, staged input ref/digest, frozen configuration, execution state, lease/generation, phase retries, deadline; execution recovery only | -| `pc_import_operation_items` | Unique `(operation_id,item_id)`; manifest, frozen target, selection/parse/commit state, staged bundle/digest, final exact ArtifactRef; per-member atomic commit | -| Bindings, receipts, input indexes | Unique stable source → Artifact, idempotent request → acceptance/commit receipt, reusable parsing lookup; no duplicated body/version authority | -| Quota ledger/rebuildable query projections | Cross-process admission/release, origin filtering/reverse lookup; not unlocked in-memory counts | - -Support bounded updates, lease scans, and cleanup. Original files/unconfirmed bodies live in a controlled staging store, -not Artifact content or giant ledger JSON. After confirmation, Source + Revision are authoritative for -body/provenance/relationships. Operations retain recovery receipts and expire under TTL. - -No FileImportJob CRUD, permanent job detail resource, or duplicated material catalog. The UI can recover unfinished -operations; operation IDs are not ArtifactRefs, and status handles do not permit arbitrary state mutation. Internal -tables remain necessary, but do not create a new task domain per format. - -### Writes and visibility - -ContentImportApplication owns one database transaction and reuses ArtifactRepository create/revise. Command Sources from -generic Artifact creation cannot substitute for actual body parts. Enforce one-to-one parts/lineage.sources. Every input -uses the same commit service; generic PUT cannot bypass confirmation. - -Before confirmation, new materials have no Artifact/Head/body Source. An artifact_id may be preallocated in the ledger -but is invisible to standard reads. Updating preserves the accepted Head during parsing, failure, retries, cancellation, -and progress changes. Only successful confirmation appends a Revision and switches Head; heartbeats/stages update -operations only. - -pc_artifact_heads.lifecycle_state active/deprecated/retired remains governance state, never uploading/parsing/failed. -Preview is creator-authorized; other Scope readers see published content only. In the commit transaction, lock/check -current permission, Head/governance, and operation validity. Reject retired or merged-away targets under existing -immutability rules; do not revive, follow merge destinations, or expose retired restoration. - -Sessions fix their existing scope_id at creation. Mutable preview/recovery/lease/TTL state never becomes Artifact -Revisions. Section 7 defines target, permission, and idempotency rules. - ## 4. Normalization, locations, and segmentation Strictly decode with the selected encoding and remove one leading BOM. Preserve line endings, indentation, and Unicode; @@ -706,7 +532,7 @@ inserted into every part body. Per document: at most 8 MiB extracted text, 128 parts, and 2 MiB locations/manifest. Exceeding limits rejects the document, not its latter half. Each serialized Source must also satisfy the existing 4 MiB observation ceiling. -Processor windows keep their own progress; atomic persistence is not atomic generation. +Other processors retain their own windows. ImportedContent document_commit must validate the complete document; atomic T1 persistence does not imply synchronous T2 completion. ## 5. Size, time, and parsing limits @@ -782,243 +608,134 @@ Principal/Scope exhaustion returns 429; deployment/dependency exhaustion returns hard limits remain 413. Queries expose waits for already accepted tasks. Workers durably schedule jittered backoff instead of hot-loop reuploads. -## 6. Internal execution, private previews, and durable staging +Release the parsing task slot once every selected document reaches T1 and no parsing/preview work remains, even if the operation is publishing. Publication backlog capacity follows Section 6 independently. -Family operation creation accepts complex files and returns recoverable operation_id, backed by Section 3's ledger and isolated -workers. Revisions represent accepted snapshots only. “Task” below means internal operation execution, not FileImportJob -or another Artifact type. +## 6. Parsing execution, state and retention -Reuse validated transactions, leases, fencing, backpressure, and worker primitives. Do not repurpose Artifact Processing -Supervisor generation jobs directly: they lack upload staging, private preview, and human confirmation semantics. Shared -infrastructure must preserve these constraints. Parse success freezes a bundle without writing Source/Artifact. +Return upload 202 only after durable controlled staging, digest verification, quota reservation and queue registration. Multiple replicas require shared durable storage, not API-process /tmp. Interrupted uploads never execute partial files; orphan blobs count against bytes until reconciliation deletes them. Isolated parser workers use frozen profiles, digests, selection versions, lease generations and deadlines; expired workers cannot replace results. -Store execution, parse, and commit states separately. ready is not publication or governance state. +Select ordinary single documents automatically. Containers first publish a complete bounded member inventory and stable item IDs atomically, then wait for one selection freezing members and individual targets. Incomplete discovery cannot be selected or checkpointed; attachments default to unselected. New members/options/targets require a new attempt; retry cannot renumber a published inventory. -| Level | Field/values | -| --- | --- | -| Task execution | execution_state: uploading / queued / running / idle / failed / cancelled / expired; idle means no current parse work | -| Internal phase | internal_phase: upload / discovery / selection / parsing / confirmation / done; discovery_state: not_required / pending / running / complete / failed | -| Selection | selected; container membership frozen through select API after discovery | -| Document parsing | parse_state: pending / parsing / ready / needs_review / failed; ready/needs_review refer to immutable bundles | -| Document persistence | commit_state: uncommitted / committed; no exposed half-commit state | +Parse states are pending/parsing/ready/needs_review/failed. Ready means required semantics and the declared extraction scope are complete. Known omissions require needs_review and explicit accept_partial. Empty text, encryption, safety failures, limits and missing required semantics cannot be bypassed by accepting partial output. Freeze bundle_digest: never preview A and save recomputed B. -```mermaid -flowchart TD - U["Task: uploading"] --> Q["Task: queued"] - Q --> R["Task: running; discover documents"] - R --> I["Task: idle; awaiting selection or confirmation"] - I -->|Frozen selection with pending documents| Q - R -->|Transient failure; at most three automatic retries| Q - R -->|Task-level failure or retries exhausted| F["Task: failed"] - F -->|Retryable discovery; valid input and quota readmission| Q - Q -->|Cancel or expire| X["Task: cancelled / expired"] - R -->|Cancel or expire| X - I -->|Cancel or expire| X -``` +Transient parsing errors have at most three automatic retries; permanent failures have none. Explicit parsing retry applies only to unfrozen retryable discovery/documents, re-admits quotas and does not reset automatic attempts. Parsing and publication retries are separate: after T1, retry generation for the same commit without reparsing or replacing evidence. ```mermaid -flowchart LR - P["Document: pending"] --> R["Document: parsing"] - R --> A["Document: ready"] - R --> B["Document: needs_review"] - R --> F["Document: failed"] - F -->|Retryable; valid input and quota admission| P - A -->|Confirm frozen digest| C["Persistence: committed"] - B -->|Confirm digest and accept partial| C +stateDiagram-v2 + [*] --> parsing + parsing --> ready + parsing --> needs_review + parsing --> parse_failed + parse_failed --> parsing: explicit retry with valid input + ready --> pending: T1 confirmation + needs_review --> pending: accept partial and confirm T1 + pending --> running: Artifact Worker claim + running --> completed: atomic T2 publication + running --> failed: recoverable generation failure + running --> blocked: plugin or authorization unavailable + running --> conflicted: Head or governance conflict + failed --> pending: publication retry + blocked --> pending: explicit retry after resolution ``` -These are internal durable states, not HTTP fields. Derive public ImportOperation.status in priority order: preserve completed/cancelled terminal results; expire unfinished work; running work yields running, otherwise queued work yields queued; complete discovery without selection yields awaiting_selection; nonempty all-committed selection yields completed; uncommitted ready/needs_review yields awaiting_commit; task-level failure or only remaining failed members yields failed. Scheduled automatic retries remain queued. Partial success uses progress counts, not a separate partially_committed status. Public phase is discovery/parsing/commit/done: discovery and selection waits map to discovery, body extraction to parsing, confirmation waits to commit, and all-success to done; failure/cancellation/expiry retain their last phase. Internal discovery_state=complete maps to discovery.status=completed; parse_state/commit_state map to parse_status/commit_status. Pre-202 uploading is not a publicly recoverable accepted operation. Unselected members count as excluded_items; cancellation/expiry retain committed counts and receipts. - -Single-document input automatically fixes selected=true. Archives/mailboxes/attachments require bounded complete -discovery, stable item_ids, container_chain, types, and default suggestions. Attachments default unselected; ordinary -archive members default selected. Pause after discovery; select API freezes one explicit set before body parsing. -Overflow/incomplete discovery is a discovery error, not a complete list. Structural container nodes are not selectable; -omitted attachments are excluded. Changed selection/options require a new attempt. Persistent item mapping preserves IDs -through automatic retries. Publish complete manifest and discovery_state=complete atomically; intermediate lists cannot -be selected or advance checkpoints. Transient failure records discovery_state=failed, phase error, discovery_retryable. -Explicit discovery retry remains possible after automatic retries exhaust. Before publication it may rediscover staged -input while reusing internal identities; published manifests/IDs cannot be rediscovered or rewritten. -Corruption/encryption/overflow are permanent, nonretryable errors. - -Return 202 and queue only after bytes are durable in controlled staging and digest-verified. Interrupted uploads never -enqueue partial files; GC reconciles placeholders/orphans. Single-node staging may use a controlled state directory; -replicas need shared durable storage, not API-process /tmp. Persist pinned profile, digest, lease owner/generation, -attempt, deadline, selection version, frozen manifest. Claims, publication, retries, cancellation, and expiry -transactionally check state/generation; stale workers cannot overwrite results. - -Automatically retry transient failures at most three times; permanent format/encryption/corruption/overflow failures are -not retried. Within an operation, retry only unfrozen documents, never overwrite ready/needs_review/committed. Explicit -retry separates discovery and document phases, only recovering unfrozen retryable failures. Track automatic counters per -phase; manual retries do not reset them and retain deadline/quota/attempt accounting. Confirm the returned -bundle_digest; do not preview A and save recalculated B. ready means complete within declared scope with required -semantics satisfied. Known omissions or unprovable completeness require review; missing required semantics outside -allowed partial behavior fails. Persist accept_partial=true permanently. - -Operation TTL is at most 24 hours from upload-placeholder creation and never beyond session.expires_at; retry/selection -do not extend it. Expiry/cancellation first fences uncommitted writes and leases, then cleans staging. Serialize with -confirmation: a winning commit retains its receipt; winning cancellation/expiry rejects uncommitted confirmation. -Cancellation never deletes published content. Keep status/idempotency mappings until 30 days after expires_at; status -returns a 200 tombstone with state/times/durable receipt references. Uncommitted preview/confirm after expiry returns -410; after cancellation, 409. Same upload key returns original operation/tombstone, never implicit creation. After -idempotency_expires_at only permanent Revision reads are guaranteed; use new attempt keys. - -Clean originals/parsing caches within one hour after all selected members commit, or after cancellation/expiry; bytes -remain charged until deletion. Incomplete containers retain staging until expires_at. Committed preview reads Sources; -receipts, locations, omissions remain durable. Staging has no arbitrary public read URL or long-term binary download -promise. - -## 7. Identity, idempotency, and atomic persistence - -### Sessions and Scope - -Require an existing scope_id in the path, an empty object body, and current scope.read + scope.contribute at session creation. Persist creator and -immutable nonempty target; no deferred resolution or Scope-creation parameters. Creating a session creates no -Scope/Artifact/Source or permission grant. - -Session, operation, item, quota, and eventual Artifact/Sources share that Scope. Child routes validate that path scope_id matches the session destination; -bodies cannot override scope_id. Cross-Scope artifact_id, SourceRef, or bindings cannot target the session. Changing -Scope requires a new session and inputs; existing operations stay put. - -Reauthorize session creation, text, upload, selection, retry, and confirmation. Queries/previews require creator plus -current Scope access, with explicitly authorized/audited administrative exceptions. Confirmation also checks destination -existence/writability, session/operation expiry/cancellation, Head, and governance. Invalid targets or revoked -permissions cannot trigger repair/creation or alternate destinations. Cancellation/cleanup follow Sections 6 and 10. - -Sessions accept uploads for 24 hours from creation; new operations/children do not extend this and operation expiry -cannot exceed it. Session/operation receipts remain at least 30 days after the last operation expiry. Session expiry -prevents uploads/uncommitted saves but preserves committed receipts and the Scope/Artifacts/Sources. - -### Separate content, identity, and requests - -Require Idempotency-Key, 1–128 ASCII letters/digits or `-_.`. Session creation uses stable Principal + operation + key, -with normalized scope_id in its fingerprint; same key/different Scope returns 409. Other writes use Principal + -session_id + operation + key. Equal request/key replays the original result; different requests conflict, and all -replays reauthorize. Retain initially frozen defaults/profile. File fingerprints cover actual digest, normalized -metadata, and target intent. Every fingerprint includes action and applicable path/body operation_id/item_id; -confirmation additionally includes bundle_digest, accept_partial, frozen target/expected Head. - -Four identities differ: artifact_id is logical document; operation_id/item_id/attempt key is an attempt; bundle_digest -is frozen extraction; ArtifactRef is an accepted version. snapshot_digest covers content/acceptance and may recur across -documents or historical versions. Never impose global `(scope_id,snapshot_digest)` uniqueness. - -Compute bundle_digest as SHA-256 of RFC 8785 canonical JSON over schema_version, source_kind, controlled origin, -container_chain, title, input type/bytes/digest, parser/profile/encoding/OCR and normalization/segmentation versions, -text type/bytes/digest, ordered content_manifest, coverage, structured warnings/omissions. Manifest includes only -ordinal, bytes, part digest, full-text interval, locations; no staging refs. Exclude diagnostic messages, elapsed time, -timestamps, Principal, session/operation/run/item IDs, artifact_id/revision, SourceRef, journal position. Compute -snapshot_digest over canonical `{schema_version, bundle_digest, accepted_partial}`; accepted_partial=true only for -needs_review. Persistence adds parts receipts without changing frozen digests. - -### Revision, Head, and repeated inputs - -| Situation | Behavior | -| --- | --- | -| First manual creation | Allocate artifact_id, save Revision 1; another create intent with equal name/content is another document | -| Update with matching expected Head and current snapshot_digest | reused + current exact ArtifactRef; no new Source/Revision | -| Update with matching Head but different snapshot | New Revision/parts and atomic Head advance; preserve old evidence | -| Expected Head mismatch | 409 revision_conflict; display current version and require reconfirmation, even if content matches | -| A → B → A | R1(A), R2(B), R3(A); no global-digest shortcut back to R1. R3 creates its own Sources in this release | -| Replay successful key | First exact receipt even if Head advanced; no new version or Head rollback | -| Reparse same input | reparse=true creates attempt; equal current frozen semantics reuses, changed output or semantically relevant parser/profile creates Revision | - -Manual updates freeze target/expected Head at item creation. Collection resolves binding/expected Head at acceptance; -new sources freeze expected absence. Binding uniqueness races return recoverable conflicts, never automatically -overwrite a newer version with old parsing. A retried create intent retains its allocated identity; independent creates -never merge by filename/digest. - -### Parsing reuse and partial extraction - -Cross-key operation reuse requires same creator/session, explicit target/expected Head, original input/origin/title, -pinned profile/options, and frozen selection. Independent manual create intents never merge. Eligible work is unexpired -with valid staging, queued/running/idle, unfinished, and no committed partial. Exclude -failed/cancelled/expired/completed. Serialize reuse and request mapping. Same upload key always restores its original -operation/tombstone; reparse=true disables merging. Never return another Principal's private task/preview. - -Durable input mappings key explicit logical target, input digest, controlled origin/title, profile, and complete -options; containers include stable member mapping/selection. They index frozen extraction and exact historical -ArtifactRefs, not overwrite authority. Cache may skip OCR but current target/expected Head still determines reuse/new -Revision. A historical version is not current automatically; manual create remains new, and cross-Scope hits stay -hidden. - -Complete cache output still needs this interactive attempt's confirmation; only explicitly configured accept-complete -workers auto-confirm. Partial cache yields needs_review/uncommitted with reusable_artifact, coverage, omissions and no -completed count. Require this attempt's explicit accept_partial=true, never inherit another Principal's or earlier -attempt's decision. Same successful commit-key replay preserves its original decision. - -reparse=true requires resubmitted bytes and recorded actual execution. Temporary operation history owns runtime -diagnostics; Revisions retain content-affecting provenance. Cache pointers may refer to history but never change Head. -The latest complete parse may update the cache index while historical versions remain exact-readable. - -### Atomic commit boundary - -One document transaction includes session/operation/item and necessary binding/Head locks, current authorization/state -checks, idempotency, fixed-Scope quota settlement, all ContentSources/journal positions, durable pending/dirty, -Revision, Head CAS, ordered lineage, input mappings, item result, and receipt. reused adds no parts but atomically -records acceptance/receipt. Any failure rolls back everything. - -Determine Source metadata's target Revision inside the transaction, outside digests; verify manifest/parts/lineage -order. Independent per-part Source HTTP calls cannot provide document atomicity. SQLite, embedded seekdb, and OceanBase -require transactions and uniqueness/CAS, not process locks. Persistent document versions/evidence are not automatically -collected here. - -Existing record capture opens its own transaction, so do not loop it and separately save Artifact. Extract/reuse an -internal connection-taking save unit with ContentImportApplication owning the outer transaction: - -1. Validate authorization, session/item, receipts, governance, binding, expected Head; acquire locks in a consistent global order and determine target Revision. -2. Validate frozen bundle/manifest and ordinary Sources; call `SourceRepository.add_with_status(connection, ...)` per part for exact refs/positions. This Repository already calls processing-intent `mark_dirty` for inserted Sources in the transaction; retain that single responsibility rather than another dirty implementation. -3. For created=True, preserve capture's configured Source-processing-binding rules and durable pending registration. Existing identical Sources do not trigger again. The shared unit must retain existing capture behavior, not omit pending after Repository calls or add an ImportedContent generation consumer. -4. Build parts and lineage.sources from actual results; call connection-taking `ArtifactRepository.create/revise`. Commit Head CAS, quotas, bindings, item result, idempotency receipt together. Any part/pending/Artifact/receipt failure rolls back all. -5. Notify and return success only after commit. Durable pending/dirty recover lost notifications. reused/replays add no Sources, Head advances, or processing registrations. - -Lock/CAS failures follow conflict/replay rules, not a second transaction that patches missing evidence or pending work. -Background processors can see only committed document parts, but their windows need not consume the whole document in -one generation. +Sessions/unconfirmed operations expire within 24 hours; retries do not extend them. Cancellation/expiry serialize with T1: a winning confirmation survives; expiry first returns 410 and cancellation first returns 409 for unconfirmed input. Cancellation only closes unconfirmed input, never accepted commits or Sources. Retain parsing tombstones and request mappings for 30 days after expires_at; replay preserves original HTTP status/body. -## 8. HTTP, client, and page contracts +Accepted commits recover independently of sessions, items and staging. T1 persists all generation metadata/provenance, and Sources hold text. Delete original blobs/caches within one hour after all selected documents are accepted, cancellation or expiry; unfinished containers remain until expiry. Never TTL-delete pending/running/blocked/failed commits or required bindings. Conflicted results retain evidence and verification data without retracting Sources. Lightweight completed mappings and exact Artifact revisions remain durable. Original files are not a permanent archive. + +Parsing and publication slots are separate. Section 5 governs parsing quotas; T1 additionally reserves durable publication backlog capacity, defaulting to 20/100/500 per Principal/Scope/deployment. Pending/running/blocked/failed commits hold capacity until completed/conflicted, preventing unbounded acceptance during plugin downtime. Failure does not release backlog capacity; retry does not double-charge. Artifact workers use framework concurrency. Discovery exposes all actual limits. + + +## 7. Identity, idempotency and two-stage publication + +### Sessions, digests and request identity + +Session creation requires an existing Scope, scope.read + scope.contribute and `{}`. Session, operation, item, commit, Source and Artifact remain in that Scope; reject body overrides and cross-Scope references. Reauthorize every access; unconfirmed inputs additionally require their creator. Saved Sources and Artifacts use Scope authorization. + +Idempotency-Key is required: 1–128 ASCII letters, digits or `-_.`. Session creation keys are Principal+operation+key with Scope in the fingerprint. Other session operations use Principal+session_id+operation+key; publication retry uses Principal+scope_id+commit_id+operation+key. Fingerprints include full path, normalized body, actual file digest, frozen target and acceptance semantics. Replayed raw input uses the initial defaults/profile. Same key/input replays original status/body; different input returns 409. Reauthorize replays. Unaccepted 429/503 creates no success receipt. + +artifact_id is logical identity, commit_id an accepted snapshot, operation/item an input attempt, and ArtifactRef an exact published version. bundle_digest hashes RFC 8785 canonical JSON containing schema_version, source_kind, origin/container_chain, title, input digest/MIME/size, parser/profile/encoding/OCR, normalization/segmentation versions, full-text digest, content_manifest, coverage and structured omissions. Exclude runtime diagnostics, display messages, timestamps, Principal, execution IDs, commit_id, artifact_id, Revision, SourceRef and journal position. snapshot_digest hashes `{schema_version,bundle_digest,accepted_partial}`. Digests may recur across documents or A→B→A history; no globally unique digest substitutes for document identity. + +Caches reuse parsing, never acceptance, authority or Head. Only the same creator, target/expected Head, bytes, origin/title, profile/options/selection can merge a valid unfrozen attempt. reparse=true forces execution. Partial cache results still require this attempt's acceptance; manual create never merges by digest. + +### T1: confirmation and evidence persistence -### 8.1 Conventions, identities, and operations +One confirmation service owns the database transaction: -Follow [RFC 1437](1437_source_artifact_rest_api.md) for Scope resource identity, naming, errors, pagination, and standard Artifact reads. Collections use lowercase plural nouns, static multiword segments use kebab-case, JSON/query/operationId use snake_case, and URIs have no trailing slash. `select`, `commit`, `retry`, and `cancel` are explicit POST domain actions, not arbitrary state PATCH/PUT. Validate the entire Scope/session/operation/item ownership chain; never look up only the last ID. Do not accept path identities again in request bodies. +1. In a consistent lock order validate current authorization, Scope, private input, expiry, frozen digest, governance, source binding and expected_head_revision; reserve the document. Another nonterminal commit returns 409 publication_pending. A unique source binding assigns the same artifact_id for a new origin. +2. Verify adapter/Family/schema admission and reserve backlog quota; assign stable commit_id. If an explicit update matches both expected Head and its published snapshot, atomically persist the manifest, completed mapping, reused receipt and exact revision, release this reservation/quota, return 200 and create no Sources or Artifact. The manifest may reference that published revision's existing Sources. +3. For a new snapshot, save every ordinary ContentSource, journal entry and existing dirty/pending registration on the same connection. Reuse persistence units, not independently committing HTTP/capture calls. Metadata records commit_id, artifact_id and ordinal, never a future Revision. +4. Save the ordered immutable manifest, accepter/authorization context, target/expected Head, parser provenance, input mapping, publication intent, quotas and acceptance receipt. Any failure rolls everything back; no Artifact/Head is created. +5. After commit, return 202 with C and best-effort wake the framework. Durable intents recover lost notifications. Other processors can now see all accepted Sources, without a promise to consume the whole document in one window. -These are proposed contracts to implement in `openapi/powercontext.yaml`, the single HTTP contract source. Generate Python/JavaScript clients and run contract-test; do not hand-edit generated code. Family is fixed to `importedcontent`; sessions require an existing Scope and cannot create one. API resources do not introduce a separate FileImportJob or ImportedContent master table. +### Complete-document input adapter (required addition) -The following abbreviations are documentation-only; clients send expanded paths, not literal P/S/O/I: +document_commit is a proposed framework input mode. Schedule one commit_id with a generation, Scope, family/schema and code/configuration version; do not also consume that family's journal cursor. The trusted adapter reads exact SourceRefs from the durable manifest and verifies contiguous zero-based ordinals, count, part digests, concatenated full-text digest, ownership and ordinary roles. Missing evidence blocks publication instead of creating a partial document. + +The plugin receives a bounded manifest and controlled `read_part(ordinal)` capability (proposed API): at most 256 KiB per read, 8 MiB per document, 2 MiB for manifest/locations and 128 parts. Body validation can stream. Output excludes body copies and has a separate 4 MiB total draft-JSON budget in document_commit mode. This does not increase the existing journal-window 1 MiB limit. Worst-case input/output/memory acceptance is mandatory; publication cannot be ready before this extension exists. + +The Server adapter registers the complete invocation evidence set. draft.source_refs must exactly equal the ordered manifest, not merely be a subset. Reject client-supplied arbitrary references, forged metadata commit IDs and cross-Scope reads. Only trusted T1 records trigger generation; ordinary content, Working Notes and system commands are not automatically wrapped. The official plugin validates and deterministically assembles one draft for that commit, without network acquisition or reparsing. + +### T2: plugin generation and transactional publication + +Reuse Artifact workers, drafts, repositories, index preparation, leases/fencing and publication transactions. A new trusted completion adapter must share the publication connection; plugin Python must not open a second transaction to record completion. + +Recheck commit generation, enabled plugin generation, document reservation, Scope/governance, the accepter's current scope.read+scope.contribute and Worker write authority inside the transaction. Integrate an explicit Scope-shared ImportedContent profile rather than inheriting generic Worker ownership. Revocation blocks publication; Worker authority cannot substitute for the accepter. Restore that subject's permissions and explicitly retry; administrative action requires separate authorization and audit. + +Create uses expected_revision=0. Updates must read the exact baseline Revision and pass expected_revision validation. Atomically write Revision, Head CAS, ordered lineage, prepared declared indexes, commit_id-to-ArtifactRef mapping, completed status and completion receipt, and release reservation/backlog quota. Failure rolls back T2 without retracting T1 evidence. Recover a lost reply from the completed record, never by creating another Revision. + +A second confirmation for the same document returns publication_pending until the first terminates. External Head/governance changes produce conflicted, release the reservation and retain evidence; never silently read the latest Head and overwrite it. The caller reviews the new Head and confirms a new request. Failed/blocked retains the reservation and retries the same commit. One accepted snapshot produces at most one Revision. Sequential A→B→A produces R1/R2/R3, never restores R1 as Head. Replays do not change Head. + +### Reads, indexes and retention + +Reuse Artifact listing/details/exact history. Generic POST/PUT, rollback and merge cannot bypass document submission and remain unsupported for this family. Sources become visible at T1 and Artifacts at T2. Disable/unload preserves published reads; pending commits wait durably and readiness rejects new confirmations. + +An initial declaration may index title/origin fields stored in Artifact content through shared search. Body text remains only in Sources; the framework does not follow parts automatically for full-text indexing. No initial body/vector-search promise. Undeclared search is unsupported; an unready initial index can return 503, not an empty-document claim. Registered search schema/indexes obey #2246 freezing rules; later body indexing is not an arbitrary hot update. + + +## 8. HTTP, client, and page contracts + +### 8.1 Conventions, identities and operations + +Follow [RFC 1437](1437_source_artifact_rest_api.md): Scope-qualified resources, lowercase plural collections, kebab-case static multiword paths, snake_case JSON/query/operationId and no trailing slash. Domain actions use POST, not arbitrary PUT/PATCH state edits. Path identities cannot be repeated in bodies; validate every parent relationship. All import operations below are proposed, not existing #2210/#2246 APIs. ```text P = /v1/scopes/{scope_id} S = P/import-sessions/{session_id} O = S/operations/{operation_id} I = O/items/{item_id} +C = P/import-commits/{commit_id} ``` -| operationId | Method and path | Success and meaning | +| operationId | Method and path | Success semantics | | --- | --- | --- | -| `get_import_capabilities` | `GET P/import-capabilities` | 200; caller-visible formats, profiles, permissions, effective limits | -| `create_import_session` | `POST P/import-sessions` | 201; fixed Scope and creator | -| `list_import_sessions` | `GET P/import-sessions` | 200; creator's recovery list within this Scope | -| `get_import_session` | `GET S` | 200; session/deadlines, no bodies | -| `import_content_text` | `POST S/texts` | 201 for new Artifact; 200 for update/reuse; request itself confirms | -| `create_import_operation` | `POST S/operations` | 202; one file durably accepted/queued, no published Artifact | -| `list_import_operations` | `GET S/operations` | 200; recovery operations, not a material catalog | -| `get_import_operation` | `GET O` | 200; status, phase, counts, deadlines, actions | -| `list_import_items` | `GET O/items` | 200; members; single documents also have one item | -| `get_import_item` | `GET I` | 200; frozen target, extraction, coverage, diagnostics | -| `get_import_part` | `GET I/parts/{ordinal}` | 200; bounded preview, reading Source after commit | -| `select_import_items` | `POST O/select` | 202; atomically freeze container selection and enqueue | -| `commit_import_item` | `POST I/commit` | 201 for new Artifact; 200 for update/reuse; atomic per document | -| `retry_import_operation` | `POST O/retry` | 202; retry discovery or specified failed documents after readmission | -| `cancel_import_operation` | `POST O/cancel` | 200; cancel uncommitted work or return completed result | - -Published materials reuse existing operations, extending Family registration, schemas, and read projections rather than creating a second detail resource: +| `get_import_capabilities` | `GET P/import-capabilities` | 200; formats, profiles, authorization, parser/publication readiness | +| `create_import_session` | `POST P/import-sessions` | 201; creator-bound session in an existing Scope | +| `list_import_sessions` | `GET P/import-sessions` | 200; creator recovery list | +| `get_import_session` | `GET S` | 200; session and expiry | +| `import_content_text` | `POST S/texts` | 202; T1 saved, publication pending; published-current reuse returns 200 | +| `create_import_operation` | `POST S/operations` | 202; durable single-file acceptance, not confirmation | +| `list_import_operations` | `GET S/operations` | 200; input recovery list | +| `get_import_operation` | `GET O` | 200; discovery, parsing, confirmation and publication summary | +| `list_import_items` | `GET O/items` | 200; member metadata | +| `get_import_item` | `GET I` | 200; frozen target, parsing and commit links | +| `get_import_part` | `GET I/parts/{ordinal}` | 200; bounded preview, Source-backed after T1 | +| `select_import_items` | `POST O/select` | 202; freeze members/targets and queue parsing | +| `commit_import_item` | `POST I/commit` | 202; document T1 succeeded, not Artifact creation; reuse returns 200 | +| `retry_import_operation` | `POST O/retry` | 202; discovery or unfrozen parsing only | +| `cancel_import_operation` | `POST O/cancel` | 200; close unconfirmed input only | +| `get_import_commit` | `GET C` | 200; current publication state and exact outcome of an accepted commit | +| `retry_import_publication` | `POST C/retry` | 202; re-admit publication of this commit without parsing again | + +Reuse standard Artifact operations without another document-details API: | operationId | Method and path | Behavior | | --- | --- | --- | -| `list_artifacts` | `GET P/artifacts/importedcontent` | Visible Heads; exact namespace filter, not body search | -| `get_artifact` | `GET P/artifacts/importedcontent/{artifact_id}` | Standard ArtifactRevision, parts, Head ETag; retain If-None-Match/304 | -| `list_artifact_revisions` | `GET P/artifacts/importedcontent/{artifact_id}/revisions` | Accepted revision history | -| `get_artifact_revision` | `GET P/artifacts/importedcontent/{artifact_id}/revisions/{revision}` | Exact immutable revision; no ETag per existing contract | +| `list_artifacts` | `GET P/artifacts/importedcontent` | Visible published Heads | +| `get_artifact` | `GET P/artifacts/importedcontent/{artifact_id}` | Standard Revision and Head ETag; If-None-Match/304 | +| `list_artifact_revisions` | `GET P/artifacts/importedcontent/{artifact_id}/revisions` | Version history | +| `get_artifact_revision` | `GET P/artifacts/importedcontent/{artifact_id}/revisions/{revision}` | Exact immutable revision | -Generic Artifact POST/PUT reject direct importedcontent writes with 422 unsupported: they cannot bypass parts/lineage/acceptance validation. Other Families retain generic Replace If-Match behavior: 428 if missing, 412 if mismatched. Import commit is a domain command using frozen expected_head_revision and 409 for conflicts; do not treat an Artifact ETag as an operation/item HTTP validator. +Declared search reuses `POST P/artifacts/importedcontent/search`; Artifact discovery supplies supported modes. No import-specific search. Generic Artifact POST/PUT, restore and merge remain unsupported for this family with standard 422 behavior; other families retain ETag/304/412/428. Import expected_head_revision conflicts return 409. Reuse plugin admin management without granting it to ordinary importers. ### 8.2 Common fields, headers, and validation @@ -1027,7 +744,7 @@ Generic Artifact POST/PUT reject direct importedcontent writes with 422 unsuppor | scope_id | Existing Scope contract: 1–256 characters including a non-whitespace character; encode as one URL segment | | session_id, operation_id, item_id | Server-generated opaque strings, 1–128 visible ASCII characters; no prefix parsing or UUID assumption | | artifact_id | Existing 1–128 visible ASCII characters; must belong to the path Scope and importedcontent Family | -| revision / expected_head_revision | Integer, minimum 1 | +| revision / expected_head_revision | Published and update-expected revisions are positive integers; acceptance revision is null before publication | | ordinal | Zero-based integer; the nth part has ordinal n-1 | | Times | RFC 3339 UTC, e.g. `2026-10-19T08:00:00Z` | | Digests | `^sha256:[0-9a-f]{64}$`; input, bundle, snapshot, and text digests have separate meanings | @@ -1037,7 +754,7 @@ Generic Artifact POST/PUT reject direct importedcontent writes with 422 unsuppor When authentication is enabled, send `Authorization: Bearer `. Every import write requires Idempotency-Key (1–128 ASCII letters/digits or `-_.`); missing/malformed headers return 400. All success/error responses include X-PowerContext-Request-ID. JSON is UTF-8, with no extra code/message/data success envelope. Unknown body fields return 422; unknown queries and invalid path/header/cursor values return 400. Duplicate/extra multipart parts return 422. Reject NUL, invalid Unicode, and C0 controls other than TAB/CR/LF in nonblank text. -Location targets S for session creation, O for operation creation, and the exact Artifact Revision for text/commit success. Import receipts are not generic Head representations and do not fabricate their ETag. Admission 429/503 includes Retry-After in seconds. Upload/action 202 returns an acceptance-time ImportOperation snapshot; replay preserves it rather than substituting current state, which is available through GET O. +Session Location points to S; upload Location points to O; text/commit Location points to C. Acceptance receipts do not carry a Head ETag. 429/503 include Retry-After in seconds. Upload/selection/parse-retry 202 returns an immutable ImportOperation acceptance snapshot; text/commit returns ImportAcceptanceReceipt, publication retry returns an ImportCommit acceptance snapshot. Replays preserve the original response; GET obtains current state. ### 8.3 Deployment capabilities and sessions @@ -1052,7 +769,8 @@ Location targets S for session creation, O for operation creation, and the exact | limits | Effective max_text_bytes, max_json_bytes, max_file_bytes, max_metadata_bytes, max_document_text_bytes, max_parts_per_document, max_part_bytes, max_selected_items, session_ttl_seconds, operation_ttl_seconds; also Section 5's format-specific and aggregate limits | | formats[] | format_id, extensions, media_types, level(A/B/C), status(available/unavailable/planned), nullable default_profile_id, profile_ids, extraction_scope, limits, nullable reason | | profiles[] | profile_id, profile_digest, status, defaults, options_schema, extraction_scope, dependencies; options_schema is a bounded JSON Schema 2020-12 subset with additionalProperties=false, never executable code or arbitrary parser classes | -| quotas | Caller-visible quota limits/remaining amounts, without other Principals' usage | +| publication | `{mode:"document_commit",status,reason}`; readiness requires the adapter, plugin schema/permissions and document budgets; otherwise reject new confirmations and retain accepted commits | +| `quotas` | Caller-visible quota limits/remaining amounts, without other Principals' usage | Extensions include the dot, e.g. `.pdf`; media_types are canonical MIME values; reason is `{code,message}`. Profile digests cover pinned configuration/dependencies, not a release clock. This is a current snapshot; every write rechecks eligibility. Initial limits follow Section 5, e.g. max_file_bytes=52428800, max_text_bytes=262144, max_document_text_bytes=8388608, max_parts_per_document=128, max_part_bytes=262144, max_metadata_bytes=65536; clients must not hard-code these as deployed values. @@ -1122,7 +840,7 @@ Idempotency-Key: text-notes-001 | media_type | text/plain | Only text/plain or text/markdown; saved body representation | | text | Required | Nonblank string; UTF-8 bytes ≤ max_text_bytes, total JSON ≤ max_json_bytes | -The request confirms immediately, without a mandatory asynchronous parsing operation; return Section 8.10's ImportCommitReceipt. Pasted HTML/code is stored as source text in the chosen representation, never executed or implicitly extracted. Use file parsing for semantic extraction. No parser, URL, or server-path input is accepted here. +This request confirms the supplied text without requiring a file parsing operation. A new accepted snapshot returns 202 ImportAcceptanceReceipt; reuse of the current published revision returns 200. Poll GET C for publication. HTML/code remains literal text; this endpoint accepts no parser, URL or Server path. ### 8.6 File uploads and parser parameters @@ -1190,7 +908,7 @@ Example single-document 202 response, with Location `/v1/scopes/project-docs/imp "phase": "parsing", "discovery": {"status": "not_required", "retryable": false, "error": null}, "selection_version": 1, - "progress": {"discovered_items": 1, "selected_items": 1, "parsed_items": 0, "failed_items": 0, "committed_items": 0, "excluded_items": 0}, + "progress": {"discovered_items": 1, "selected_items": 1, "parsed_items": 0, "failed_items": 0, "committed_items": 0, "published_items": 0, "excluded_items": 0}, "available_actions": ["cancel"], "created_at": "2026-10-19T08:05:00Z", "expires_at": "2026-10-20T08:00:00Z", @@ -1201,49 +919,47 @@ Example single-document 202 response, with Location `/v1/scopes/project-docs/imp } ``` -### 8.7 Operation queries, items, and previews +### 8.7 Operations, items and previews -ImportOperation uses the same fields for POST acceptance snapshots, GET, and list items: +ImportOperation fields apply to POST acceptance snapshots, GET and list items: | Field | Type and meaning | | --- | --- | | scope_id, session_id, operation_id, family | Path identity and importedcontent | -| status | queued/running/awaiting_selection/awaiting_commit/completed/failed/cancelled/expired; server-derived by Section 6 | -| phase | discovery/parsing/commit/done; errors retain failed phase, success uses done | -| discovery | `{status,retryable,error}`; status not_required/pending/running/completed/failed; nullable ErrorDetail | -| selection_version | Nonnegative integer; container 0 before freezing, 1 after; single document automatically 1 | -| progress | discovered_items (null before complete discovery), selected_items (null before selection), parsed_items, failed_items, committed_items, excluded_items; last four nonnegative. parsed_items counts frozen results including committed items, so counts are not additive categories | -| available_actions | select/retry/cancel hints, revalidated at execution | -| created_at, expires_at, idempotency_expires_at | RFC 3339 UTC; retries do not extend deadlines | -| blocked_reason, retry_after_seconds | Nullable `{code,message}` and nonnegative integer for accepted queue waits | -| error | Nullable ErrorDetail for operation-level errors; item failure is not HTTP query failure | +| status | queued/running/awaiting_selection/awaiting_commit/publishing/completed/failed/cancelled/expired | +| phase | discovery/parsing/commit/publication/done; failure retains its stage | +| discovery | `{status,retryable,error}`; not_required/pending/running/completed/failed | +| selection_version | Nonnegative; containers 0 before selection, 1 after; single documents 1 | +| progress | discovered_items/selected_items nullable when unknown; parsed_items/failed_items/committed_items/published_items/excluded_items nonnegative and overlapping, not an additive total | +| available_actions | select/retry/cancel hints, not authorization | +| created_at, expires_at, idempotency_expires_at | RFC 3339 UTC; input/request-mapping deadlines do not expire accepted publication | +| blocked_reason, retry_after_seconds, error | Nullable safe reason, nonnegative seconds and ErrorDetail | -Internal execution_state, task_slot_held, leases, and fencing generations are not public status fields. Partial success is represented by progress: active work yields queued/running, ready uncommitted results yield awaiting_commit, and only remaining failures yield failed. Cancellation/expiry preserve committed counts and per-item receipts; an operation is not a cross-document transaction. +Priority: preserve cancelled; unconfirmed input past its deadline is expired; otherwise parsing execution/queue is running/queued, selection is awaiting_selection and confirmation is awaiting_commit. Once all selected items reach T1, pending publication is publishing, only blocked/failed/conflicted work is failed, and all published/reused is completed. With mixed input and publication work, display the input stage while progress and commit states expose publication. Accepted commits continue after operation expiry/cancellation through C. committed_items counts T1; published_items counts completed Artifacts. -ImportItem (GET I and bounded GET O/items entries, excluding body text/full content_manifest): +ImportItem fields: | Field | Type and meaning | | --- | --- | -| scope_id, session_id, operation_id, item_id | Complete parent/child identity | -| title, origin, container_chain, input_media_type | Identified title, provenance, member chain, MIME; unidentified MIME is null | -| selectable, selection_suggested, selected | Booleans; structural nodes are not selectable, suggestions do not mean acceptance | -| artifact_target, resolved_target | Intent and frozen resolved target; nullable before selection | -| parse_status | pending/parsing/ready/needs_review/failed | -| commit_status | uncommitted/committed | -| retryable | Whether the current parse error can be retried | -| parser | Nullable; profile_id/profile_digest/router_version/options plus actual backends/conversions; fully returned after freezing | -| statistics | page_count, sheet_count, slide_count, table_count, part_count, text_bytes; nonnegative integers or null, never zero for unknown/inapplicable | -| ocr | requested_mode, nullable boolean used, nullable integer processed_page_count, nullable string engine, languages; incomplete counts are not final totals | -| coverage | complete/partial/unknown; null before a result exists | -| warnings, omissions | Arrays of `{code,message,locator}` with nullable locator; structured issues affect semantic digests, display messages do not. Bound diagnostics; fail if they cannot be represented within budget rather than hiding omissions | -| input_digest, bundle_digest | SHA-256 or null; no invented digests before stage completion | -| reusable_artifact | Nullable standard `{family,artifact_id,revision}` within top-level Scope; possible cached result, neither a successful commit nor a Candidate | -| receipt | Nullable ImportCommitReceipt, present only after this attempt commits | -| available_actions, error | commit hints; nullable ErrorDetail | - -GET I/parts/{ordinal} reads one stable part, not the full body. Return scope_id/session_id/operation_id/item_id/ordinal/text/media_type/text_bytes/text_digest/bundle_digest/locators. Maximum 256 KiB per part, 128 parts per document; return the complete part without truncation. Sections 4–5 define splitting/overflow. Unknown ordinal is 404, unfinished parsing is 409, uncommitted expired previews are 410 and cancelled previews 409. Committed previews read Sources while mappings remain, then standard Artifact/Source APIs provide permanent access. - -Locations use a kind discriminator, e.g. `{kind:"pdf",page_number:3}`, `{kind:"spreadsheet",sheet_name:"Budget",cell_range:"A1:D20"}`, `{kind:"slide",slide_number:5}`, `{kind:"text",line_start:10,line_end:25}`. Pages/slides/lines are one-based; line ranges are inclusive. Full-text character ranges count Unicode code points as `[start,end)`. PDF bbox requires page_width/page_height, unit=pt, origin=top_left, and bbox=[left,top,right,bottom]; never mix pixels and points. Report unavailable locations rather than guessing. +| scope_id, session_id, operation_id, item_id | Complete parent identity | +| title, origin, container_chain, input_media_type | Bounded metadata; unknown MIME null | +| selectable, selection_suggested, selected | Booleans; structural nodes are not selectable | +| artifact_target, resolved_target | Frozen target, nullable before selection | +| parse_status, commit_status | pending/parsing/ready/needs_review/failed; uncommitted/committed, with committed meaning T1 only | +| commit_id, publication_status | Null before T1; then link to C, whose state is authoritative | +| retryable | Parsing retry eligibility, not publication retry eligibility | +| parser | profile_id/profile_digest/router_version/options/actual backends and conversions; null before identification | +| statistics | page_count/sheet_count/slide_count/table_count/part_count/text_bytes; integers or null, unknown is not zero | +| ocr | requested_mode, used, processed_page_count, engine, languages; unknown entries null | +| coverage, warnings, omissions | complete/partial/unknown or null; bounded `{code,message,locator}` arrays | +| input_digest, bundle_digest | SHA-256 or null | +| reusable_artifact | Nullable `{family,artifact_id,revision}`; parse-cache candidate, not this attempt's acceptance | +| receipt | Immutable ImportAcceptanceReceipt after T1, otherwise null | +| available_actions, error | Action hints and safe ErrorDetail | + +`GET I/parts/{ordinal}` returns scope_id/session_id/operation_id/item_id/ordinal/text/media_type/text_bytes/text_digest/bundle_digest/locators. Ordinals start at zero; return one complete part up to 256 KiB without truncation. Missing ordinal 404, unfinished parsing 409, expired unconfirmed preview 410, cancelled preview 409. After T1, read Sources while the mapping remains; thereafter use exact SourceRefs from C or the Artifact. + +Locations discriminate by kind: pdf/page_number, spreadsheet/sheet_name/cell_range, slide/slide_number, text/line_start/line_end. Pages/slides/lines start at one; line ranges are inclusive and character ranges use Unicode code-point `[start,end)`. PDF bbox includes page dimensions, unit=pt, origin=top_left and left/top/right/bottom. Unavailable locations are explicitly unknown. ### 8.8 Selection, retry, and cancellation @@ -1273,11 +989,11 @@ POST O/retry accepts one of two mutually exclusive bodies: stage is required. discovery forbids item_ids and requires failed/retryable discovery, unpublished complete manifest, and valid staging. documents requires a nonempty unique item_ids array of at most 1,000 selected, failed, retryable, unfrozen members of this operation. Reject the whole request if any member is ineligible. Never retry ready/needs_review/committed, change inputs/profiles/targets, or extend deadlines. Manual retries do not reset the per-phase maximum of three automatic retries. Return 202 only after atomic quota readmission, queue, and receipt persistence. 429/503 means not accepted; back off and reuse the key. Accepted background resource waits appear through GET O blocked_reason. -POST O/cancel accepts only `{}`; atomically fence uncommitted writes and return 200 ImportOperation. Repeated cancellation does not repeat effects, completed remains completed, expired returns a tombstone, and published Artifacts survive. select/retry return 409 for cancelled and 410 for expired operations. Every action rechecks creator, current Scope permissions, and ancestry; administrative exceptions follow Section 10. +POST O/cancel accepts only `{}`; atomically fence uncommitted writes and return 200 ImportOperation. Repeated cancellation does not repeat effects, completed remains completed, expired returns a tombstone, and accepted Sources/commits survive and continue publication; cancellation only closes unconfirmed input. select/retry return 409 for cancelled and 410 for expired operations. Every action rechecks creator, current Scope permissions, and ancestry; administrative exceptions follow Section 10. -### 8.9 Commit and concurrency +### 8.9 Confirmation, commit queries and publication retry -POST I/commit does not repeat operation_id/item_id in its body. The example digest illustrates syntax only; use the actual preview digest: +`POST I/commit` accepts only the frozen digest and acceptance decision: ```json { @@ -1286,47 +1002,90 @@ POST I/commit does not repeat operation_id/item_id in its body. The example dige } ``` -bundle_digest is required; accept_partial is boolean, default false. ready accepts only false; needs_review requires explicit true or returns 409 partial_extraction. Target/title/origin/parser/expected Head are frozen and cannot be edited here. Digest mismatch is 409 bundle_conflict; Head mismatch is 409 revision_conflict even when text happens to match. Cancelled/unparsed is 409; expired uncommitted work is 410. +bundle_digest is required; accept_partial defaults to false. Ready accepts false only; needs_review requires true. Safety/format/limit failures cannot be bypassed. Target, title, origin, parser and expected Head are frozen. Digest mismatch is 409 bundle_conflict, Head mismatch 409 revision_conflict and an outstanding reservation 409 publication_pending. Unready/cancelled input returns 409; expired unconfirmed input 410. -One transaction writes all Sources, journal positions, dirty/pending state, Revision/Head CAS/lineage, bindings, quota, and durable receipt; post-commit notification failure remains recoverable. Within mapping retention, recommitting an already committed item with identical acceptance parameters returns its original receipt even after Head advances or operation expiry; changed acceptance parameters conflict. Reauthorize all replay. +A new T1 snapshot returns 202 ImportAcceptanceReceipt with Location C; reuse of an identical published current snapshot returns 200. Never report 201 Artifact creation. Replay always returns the original status/body even after generation; GET C obtains current results. C is independent of session lifetime, with no repeated Scope/commit identity in its body. -For version conflicts, read current Head, show differences, then create a new attempt with new expected Head, original bytes, and new key. Matching parsing cache may help, but never mutate old frozen results or overwrite concurrent content. This domain conflict protocol does not change generic PUT's 428/412 or GET ETag behavior. +GET C requires current scope.read and the creator, or explicit administration with Scope read. Published Artifact/Source content remains Scope-readable. It returns 200 ImportCommit even for generation failures. POST C/retry requires the original accepter's current scope.read+scope.contribute or authorized administration. Its body is: -### 8.10 Shared commit receipt and standard references +```json +{"expected_generation": 2} +``` -Text and file commits return ImportCommitReceipt with flat identity and standard SourceTypeReference/ArtifactReference lineage, not a Source/Artifact union or writable references: +expected_generation is a required positive integer from the latest GET C; mismatch returns 409. Only retryable failed or resolved blocked states can retry. Completed/conflicted/running/pending returns 409 without changing the snapshot, target or expected Head. Reuse commit_id/Sources, CAS-increment generation, retain backlog reservation and register execution intent. Return 202 ImportCommit acceptance snapshot with Location C; replay exactly by key. 503 means plugin/authorization prerequisites remain unavailable, not accepted work. + +### 8.10 Acceptance receipts and publication results + +Text/commit ImportAcceptanceReceipt example (illustrative identifiers/digest): ```json { "scope_id": "project-docs", + "commit_id": "icm_001", "family": "importedcontent", "artifact_id": "ic_001", - "revision": 1, - "outcome": "created", + "revision": null, + "outcome": "accepted", "source_kind": "content", "accepted_partial": false, "snapshot_digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "accepted_at": "2026-10-27T08:00:00Z", + "status_url": "/v1/scopes/project-docs/import-commits/icm_001", "sources": [{"source_type": "content", "source_id": "cs_001"}], - "artifacts": [], "parts": [{"ordinal": 0, "source_type": "content", "source_id": "cs_001", "journal_position": 42}] } ``` -| Field | Type and meaning | +| Field | Format and meaning | +| --- | --- | +| scope_id, commit_id, family, artifact_id | Flat identities; commit_id is an opaque Server-generated 1–128 visible ASCII string; artifact_id may not yet be published | +| revision, outcome | Null/accepted for new snapshots; actual positive revision/reused for current published reuse. Immutable receipt, not a standard ArtifactRevision | +| source_kind, accepted_partial, snapshot_digest | Fixed content, this acceptance decision and semantic digest | +| accepted_at, status_url | T1 acceptance time and same-deployment relative C URL | +| sources, parts | Ordered standard SourceTypeReferences; parts add ordinal/journal_position and exactly match the frozen manifest | + +ImportCommit (GET C and retry acceptance snapshots) includes the identities, digests, acceptance data and sources/parts above, plus: + +| Field | Format and meaning | | --- | --- | -| scope_id, family, artifact_id, revision | Standard Artifact identity; exact accepted revision | -| outcome | created: new Artifact, HTTP 201; updated: new revision of existing Artifact, HTTP 200; reused: exact current revision, HTTP 200 | -| source_kind, accepted_partial | content and this acceptance decision, preserved on successful replay | -| snapshot_digest | Frozen bundle plus acceptance semantics; neither HTTP ETag nor standard content_digest | -| sources | Ordered standard `{source_type,source_id}` direct lineage, all content | -| artifacts | Standard `{family,artifact_id,revision}` direct Artifact lineage, [] if none; Scope from top level | -| parts | Ordered `{ordinal,source_type,source_id,journal_position}` mapping to Source bodies, matching sources/manifest one-to-one; pasted text has one part | +| status | pending/running/blocked/failed/conflicted/completed; not Artifact lifecycle_state | +| generation | Positive integer for retry CAS and Worker fencing; clients cannot blindly increment it | +| revision, outcome | Both null before completion; actual Revision and created/updated/reused when completed | +| completed_at, artifact_url | Null before completion; then UTC time and exact-revision relative URL, not just Head | +| retryable, available_actions | Boolean and array of hints; reauthorize and CAS on invocation | +| error, blocked_reason | Nullable `{code,message,details}` and `{code,message}`; no bodies, secrets or private paths | + +```json +{ + "scope_id": "project-docs", + "commit_id": "icm_001", + "family": "importedcontent", + "artifact_id": "ic_001", + "source_kind": "content", + "accepted_partial": false, + "snapshot_digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "accepted_at": "2026-10-27T08:00:00Z", + "status_url": "/v1/scopes/project-docs/import-commits/icm_001", + "sources": [{"source_type": "content", "source_id": "cs_001"}], + "parts": [{"ordinal": 0, "source_type": "content", "source_id": "cs_001", "journal_position": 42}], + "status": "completed", + "generation": 1, + "revision": 1, + "outcome": "created", + "completed_at": "2026-10-27T08:00:02Z", + "artifact_url": "/v1/scopes/project-docs/artifacts/importedcontent/ic_001/revisions/1", + "retryable": false, + "available_actions": [], + "error": null, + "blocked_reason": null +} +``` -Example digests show syntax only and cannot submit real files. No body text is returned; Location identifies the standard exact Revision followed by Source reads. Source HTTP identity remains source_type, not source_kind/name. Standard ArtifactRevision shape and content_digest calculation are unchanged. +Source references use source_type and ArtifactRef uses family/artifact_id/revision. Standard published Artifact shape, ETag and content_digest remain unchanged. T1 receipts never predict final revisions; use this commit's exact outcome rather than the current Head to determine completion. ### 8.11 Pagination, errors, idempotency, and recovery -New lists use limit (default 50, range 1–100) and opaque cursor. Session lists are Scope-bound by path; operations optionally filter status; items optionally filter selected/parse_status/commit_status. Responses use `{items:[],next_cursor:null}`, null at the end, without fabricated total. Bind cursors to caller, complete ancestry, filters, limit, high-water mark, and TTL; reauthorize every page. High water limits new records, not changing status, so filtered lists are not status snapshots. Invalid/mismatched cursors return 400; expired cursors return 410. Standard Artifact lists retain their shape, adding only importedcontent namespace filtering. +New lists use limit (default 50, range 1–100) and opaque cursor. Session lists are Scope-bound by path; operations optionally filter status; items optionally filter selected/parse_status/commit_status. Responses use `{items:[],next_cursor:null}`, null at the end, without fabricated total. Bind cursors to caller, complete ancestry, filters, limit, high-water mark, and TTL; reauthorize every page. High water limits new records, not changing status, so filtered lists are not status snapshots. Invalid/mismatched cursors return 400; expired cursors return 410. Standard Artifact lists retain their shape, using only framework-provided filters; no new namespace list filter is promised. Use the existing error envelope: @@ -1354,303 +1113,107 @@ Use the existing error envelope: Format/extraction/overflow/parser failures after 202 are operation/item errors; successful queries remain 200, not 500. ErrorDetail is `{code,message,details}` with null details when absent, never credentials/private blob addresses. Keep 412/428 for existing conditional requests; import commit does not borrow them. -Import Idempotency-Key is a new durable domain protocol, not a claim that existing Source/Artifact Create is idempotent. Section 7 defines namespaces. Normalize after defaulting; file requests compare actual byte digest and metadata, not multipart boundaries. Same key/request replays original status and receipt; changed request returns 409. Identical raw replay retains originally frozen defaults despite deployment profile changes. Unaccepted 429/503 writes no success receipt; committed replay returns its original revision without a new write even after Head advances. Retention/tombstones/GC follow Sections 6–7. +Import Idempotency-Key is a new durable domain protocol, not a claim that existing Source/Artifact Create is idempotent. Section 7 defines namespaces. Normalize after defaulting; file requests compare actual byte digest and metadata, not multipart boundaries. Same key/request replays original status and receipt; changed request returns 409. Identical raw replay retains originally frozen defaults despite deployment profile changes. Unaccepted 429/503 writes no success receipt; accepted replay returns its original acceptance snapshot even after publication or Head changes; GET C provides the exact outcome without another Revision. Retention/tombstones/GC follow Sections 6–7. -### 8.12 Clients, UI, and call sequences +### 8.12 Clients, UI and examples -Python clients expose the listed operationIds and pass scope_id to every route. CLI provides text/files/resume/cancel/dry-run; new imports require an existing --scope. Resume --session obtains Scope from a caller-held session receipt or explicit --scope; both must agree if present. With neither stored destination nor explicit Scope, require --scope; do not add an unscoped server lookup. UI recovery stays within the current Scope; switching Scope does not migrate work. +HTTP/OpenAPI is authoritative; generate Python/JavaScript clients. Dashboard uses existing Tabler/Jinja2/HTMX, locales and themes. The write entry defaults off and requires explicit enablement/authentication. CLI/SDK call the same service without independent per-part writes. MCP source add is not file preview/confirmation. Any import MCP tools require separately generated contracts and equivalent recovery coverage. ---artifact-id/--expected-head-revision explicitly update, and parser flags map to parser.options. Containers display the complete manifest before select; unattended use requires explicit selection rules. --accept-complete auto-commits only ready; partial needs separate explicit authorization including cached partial results. dry-run verifies local facts only, never claims complete importability without parsing. Back off on 429/503 using Retry-After rather than presenting unaccepted work as queued. - -The UI sends frozen bundle_digest and the original Idempotency-Key. Refresh/login only resumes reads, never confirms automatically. Resolve uncertain results via status/receipt or same-key replay. Clients measure upload progress; show indeterminate parsing unless the parser reports progress; never expose a half-commit state. - -| Scenario | Calls | +| Scenario | Sequence and outcome | | --- | --- | -| Paste Markdown | GET P/import-capabilities → POST P/import-sessions `{}` → POST S/texts → standard Revision/Source reads | -| Scanned PDF | Create session → POST S/operations with parser.options.ocr_mode=auto → GET O/I/parts → POST I/commit → exact Revision | -| ZIP/mailbox | Upload → GET O until complete discovery → GET O/items → POST O/select → preview/commit per document, separate transactions | -| Update | Explicit artifact_id/expected_head_revision target → preview → commit → updated 200; old Head remains readable until success | -| HTML file | Same file flow, detected text/html; webpage provenance does not fetch a URL | -| Future webpage Connector | Separately validated authorized adapter submits a frozen snapshot; no URL import endpoint in this release | - -### Published bodies, history, and comparison - -Reuse Dashboard history pagination, Revision selection, and comparison pages with a Family-specific read projection, not -another version system. Exact historical parts read their Sources in batches, preserve order, and detect -missing/digest-invalid evidence. Do not depend on expired operations/staging or substitute current Head text for missing -historical parts. - -Compare body text plus meaningful origin, parser/profile, OCR, coverage, omissions, accepted_partial changes. -SourceRefs, positions, digests, timestamps are provenance details, not a substitute JSON diff. Lazily load parts with -bounded comparison characters/output/compute. Over-budget output says comparison is incomplete and offers part-level -reading without silently truncating exact originals. Hide restore/rollback/merge buttons and reject direct API attempts -too. - -No ImportedContent body search in this release. The existing `/v1/scopes/{scope_id}/artifacts/{family}/search` returns -its standard unsupported result for this Family; namespace filtering is metadata only. Future keyword/semantic support -registers an ArtifactSearcher with ArtifactSearchService, defining Scope authorization, version selection, and index -updates, rather than a duplicate search endpoint or implicit vector-ready promise. - -Existing stable Artifact handles, if used, bind one exact immutable Revision. They are not live external-document -identity or a replacement for source bindings. - -## 9. Continuous collection with OpenDAL - -Add `opendal-text-imports` while retaining the existing opendal-text-files/text-file-snapshot evaluation Definition, -naming, permissions, and checkpoint behavior. Use public import APIs, not the old RemoteConnectorWorker path that -registers manifests. Require configured existing scope_id plus scope.read + scope.contribute, not server.admin. Missing -destination/permission fails without Scope creation. Use the same ImportSession protocol. - -Use the same matrix/profiles. Each scan obtains a complete bounded selected-input list, reads actual bytes, and hashes -input_digest. mtime/ETag aid diagnostics but alone cannot prove unchanged content. Identity is Scope + -source_authority_id + relative path/object key. Profile/include-exclude/encoding/OCR changes explicitly create a new -collection configuration binding/checkpoint while retaining document identity under the authority; changed snapshots -create Revisions. New root/bucket means new authority. Display namespace and checkpoint configuration fingerprints do -not define document identity. - -Checkpoint stores completed input identity, raw digest, profile, and exact Scope-bearing ArtifactRef. Stable input_key -derives from binding, input identity/digest, profile, and complete selection/parsing policy; it is not directly the -upload key. - -A durable recovery table separate from successful checkpoint stores `(binding, input_key, session_id, attempt_uuid, -operation_id, expected_head_revision, idempotency_expires_at, retry_after)`. Derive upload key from input_key + -attempt_uuid, persist attempt_uuid with transaction/CAS before sending. Lost responses/crashes resume the same -attempt/key. Replicas racing one input share the durable attempt; different service Principals never share private -recovery tables. - -On expired tombstone, retain committed receipts, then CAS-allocate new attempt_uuid/key and resubmit bytes. Expired -sessions require a persisted new session for the same existing Scope. Also start new attempts when mappings expired and -cannot be recovered. Reuse frozen results through stable mappings but check current/expected Head. Recover committed -receipts without resubmitting those commits; continue unsaved members instead of being permanently stuck at 410. After -discovery, select a stable member set under explicit rules. Input completes only when all expected members have durable -receipts. Recovery-table progress is not checkpoint progress. User/admin-cancelled input is paused; later scans cannot -recreate it until explicit resume with a new attempt key. - -Automatic confirmation must be configured and accept only coverage=complete. Apply this to fresh parses, cached -candidates, and recovered receipts: an existing partial Revision cannot satisfy complete-only or advance checkpoint. -Workers cannot accept historical partial on the user's behalf. Retry discovery with stage=discovery and document -failures with stage=documents; quota rejection preserves state and durable backoff. Missing OCR, damaged attachments, -unsupported selected members, or partial extraction need attention, never silent acceptance. CAS checkpoint only after a -complete round listing and saved/reused receipts for every selected member. operation creation, upload success, or ready -parsing alone is insufficient. - -Other valid files may commit; failures do not roll them back, but retain the old checkpoint. Recover with original keys -while valid, new attempt keys after expiry, and existing receipts. CAS conflicts reload/rescan, never force overwrite. -External deletion affects comparison only, not Artifact/history/Source deletion. Incomplete listings, permission denial, -or connection errors are not deletion evidence. - -Prefer immutable external revisions or conditional reads. Otherwise promise only the bytes actually read, never a -timestamp disguised as exact revision. Credentials stay in the worker, not Sources, checkpoints, logs, or parsers. - -## 10. Security, authorization, and isolation - -Text/file writes require selected Scope scope.read + scope.contribute; lists, history, and Source reads require -scope.read. Ordinary imports do not require server.admin, create Scopes, or initialize access. Sessions/tasks bind -stable creator Principal. Other ordinary callers cannot read uncommitted previews or confirm them. Administrative access -requires explicit server.admin plus corresponding Scope permissions, with audit. Reauthorize each confirmation; old -sessions cannot survive revoked write authority. - -Parser workers receive only assigned operation, staging blob, profile, not arbitrary Scope write permission. -ContentImportApplication commits as the confirming caller. Background parsing may finish accepted input but cannot -create Sources itself. Revocation blocks confirmation; staging expires normally. - -Dashboard write routes validate same-origin Origin and forward public APIs using existing session identity. Preserve -HttpOnly/Secure/SameSite/cookie-path rules, no direct database writes, expanded `/v1` cookie authority, or untrusted -proxy headers. - -Run parsers in restricted processes/containers with CPU, memory, time, file-count, pixel limits. Disable network by -default; prohibit XML external entities, scripts, macros, formula recalculation, link refresh, notebook execution, -dynamic plugin installation. Filenames cannot choose arbitrary disk paths. Raw and extracted content are data, not -execution instructions. - -Archives reject traversal, absolute paths, symlinks/hardlinks, device files, duplicate overwrites. Meter actual -expansion, not just index-declared sizes. Do not accept decryption passwords; encrypted_document for encrypted input, no -DRM bypass. - -Default OCR is local with installed language packs. Cloud OCR/vision needs explicit deployment enablement and visible -external-transfer disclosure, not unknown-provider fallback. Ordinary extraction does not implicitly invoke generative -models. Meter OCR, vision, and downstream Artifact usage separately. +| Pasted Markdown | Capabilities → create session → POST S/texts 202 → GET C pending/completed → exact Artifact revision; no LLM | +| 80-page PDF with 40 parts | Upload → inspect text/OCR/omissions → POST I/commit 202 → T1 atomically saves all 40 parts and manifest → document-mode plugin → GET C completed; no assumption that a 20-entry journal window contains the document | +| Scanned PDF without OCR | needs_review or ocr_required; no confirmation for empty text, explicit acceptance for usable partial text; collect complete-only rejects it | +| Update | Specify artifact_id/expected_head_revision → confirmation 202 → old Head remains readable → T2 CAS; conflicts retain evidence and expose conflicted | +| ZIP/mailbox | Complete discovery → selection → per-member preview/confirmation → individual C; independent transactions, no rollback of accepted members | +| Generation failure | GET C failed → correct dependency → POST C/retry with generation → GET C completed; no reupload or duplicate Sources | +| Source collect / OpenDAL | Same parsing/confirmation APIs → durably record commit_id → recover publication independently → CAS checkpoint only after exact completion receipts | -## 11. Errors and observability +Document lists show published Artifacts only; recovery views also show evidence saved with pending publication. Empty Artifact lists do not mean upload failed. Cancelled operations retain links to accepted commits. Reuse existing history/comparison pages, reading ordered Sources through exact Revision.parts without substituting Head evidence. Bound comparison computation/display and disclose incomplete display. Do not show rollback/restore/merge actions initially. -Retain the error envelope and distinguish unsupported_format, parser_unavailable, capability_insufficient, -encoding_required, invalid_encoding, encrypted_document, damaged_document, ocr_required, no_extractable_text, -parse_timeout, input_limit_exceeded, output_limit_exceeded, archive_limit_exceeded, partial_extraction, -idempotency_conflict, revision_conflict, source_binding_conflict, session_expired, bundle_conflict, selection_conflict, -operation_cancelled, operation_expired, quota_exceeded. -401/403/404 follow existing visibility. Invalid fields 422; hard request limits 413; unsupported media 415; -idempotency/state/cancel conflicts 409; expired uncommitted operations 410; throttling 429; dependencies unavailable -503. Failures after 202 belong to operation state returned by a 200 query, not a retroactive upload HTTP failure. -Network timeouts may mean outcome unknown; verify with the same key. +## 9. Source collect and OpenDAL -Logs exclude body, tokens, full local paths, Prompts, OCR images. Metrics distinguish uploaded bytes, detected formats, -parse outcomes, OCR pages, confirmation-needed items, committed documents/parts, reuse hits, parse/commit latency, -checkpoint conflicts, last successful completion. High-cardinality IDs are not metric labels. +Deploy OpenDAL acquisition as a Source collect plugin, reusing #2210 instances, targets, execution processes, lifecycle, credential separation and checkpoint foundations. Loading requires administration; runtime identities need only their Scope permissions. Targets must already exist. Do not silently change old opendal-text-files/text-file-snapshot evaluation types, identities, authorization or checkpoints; configure the new mode explicitly. -“Saved” counts durable document receipts only. Container summaries separate saved, failed, excluded, and pending. -Metadata-only/empty bodies are not success. Downstream cost/progress must come from processor evidence, not parsing -success. +Use the proposed public import adapter with the same capabilities/session/upload/confirmation/commit-status APIs. Do not access private PluginContext properties or use current per-Source run_batch submission for T1. Parsing may run near acquisition, but Server must validate normalized input budgets/digests/permissions. Public uploads accept neither arbitrary remote URLs nor Server paths. -## 12. Compatibility, implementation, and acceptance +Scope+source_authority_id+relative path/object key defines identity. mtime/ETag assists detection, while actual bytes define the digest; prefer immutable external revisions or conditional reads. Credential rotation preserves authority; changing root/bucket creates another authority. Profile/filter changes update the collection fingerprint without changing document identity. -### Implementation boundaries +Persist attempts with binding, input_key, attempt_uuid, session/operation/item, commit_id, idempotency expiry and backoff. CAS the attempt before sending; recover lost responses with the same key. After T1, retain commit_id and recover publication through C even after session expiry, never reupload accepted evidence. Only pre-T1 expiry needs new bytes and a new attempt key. -Implement Section 1's fixed parser families/routes, not ad hoc runtime library selection. Stage work by responsibility: -text persistence/unified manifest, Family operation creation/commit plus durable ledger/standard parsing package, then -OCR/long-tail/OpenDAL. Both A and B remain formal targets; A alone does not complete this RFC. +A successful checkpoint means every selected member in a complete listing has a completed/reused exact ArtifactRef. T1 is evidence_saved, not a successful checkpoint; the plugin's latest single-Source report is not whole-document completion. The adapter reuses checkpoint CAS and durably associates attempts through a transaction or equivalent protocol. Other files may proceed while failures remain outstanding; crash recovery must not create duplicate Revisions. -Register importedcontent, operation storage, indexes without changing existing Source/Artifact references. Historical -Sources are not automatically wrapped as ImportedContent. Update allowlists, exact reads, history, lineage, Dashboard -readers without generation/Candidate inclusion. Do not switch old OpenDAL bindings in place; explicit reimport may -create new content Sources, without claiming old type identity preservation. Core Python support remains unchanged; -heavy dependencies belong to separate workers. +Automatic confirmation accepts coverage=complete only, including caches and recovery; it does not inherit earlier partial acceptance. User-cancelled unconfirmed input becomes paused, not automatically recreated by scanning. Stopping a target does not retract an accepted Server publication; show that saved commits may still complete. External deletion never deletes Sources/Artifacts/history; incomplete listing and permission failures cannot imply deletion. -### Database initialization, upgrades, and recovery +## 10. Authorization and isolation -All persistent structures must support SQLite, embedded seekdb, and OceanBase. Version-manage Section 3's three core -tables plus chosen bindings, receipts, quota ledger, and indexes; do not lazily create auxiliary tables on first -request. Published material remains in Artifact/Source tables, without new ImportedContent master/version tables. +Private sessions/previews require their creator or explicit administration. T1 rechecks scope.read+scope.contribute; T2 checks both original accepter and Worker current authority. Revocation blocks publication. Plugin administration alone cannot silently accept on someone's behalf. Integrate Scope-shared reads across Family authorization, HTTP/runtime/SDK/UI; avoid lists that are readable while details require Worker ownership. -`server db-migrate status/plan/apply/verify` supplies infrastructure, but current registered bundles manage limited -tables, not all business schemas. Register import schema bundles/revisions, frozen table/index manifests, postcondition -verification, dependencies, compatible versions. Do not modify already released revisions. +Parser workers receive only assigned staging/profile access, not arbitrary Scope writes. Artifact workers receive controlled committed evidence and exact part reads, not temporary input URLs. Subprocess isolation is not an untrusted-code sandbox; deploy trusted plugins and enforce resource/time/network defaults, archive-path and decompression defenses. -| Area | Required constraint | -| --- | --- | -| Empty/existing databases | Initialization and upgrade yield the same schema; inspect managed tables/indexes/history/conflicting residues, never automatically adopt arbitrary existing tables as migrated | -| Uniqueness/indexes | Session/operation keys, `(operation_id,item_id)`, stable source identity, partitioned idempotency uniqueness; bounded recovery/GC indexes by creator, Scope, state/lease expiry, TTL. Long keys use collision-checked digest indexes where backend limits require, never silent truncation | -| Maintenance gate | API writes, parser dispatch, Connector commits, GC obey migration maintenance; no import before schema ready or mixed-version replicas writing incompatible structures | -| Interrupted DDL | SQLite uses transactions as supported. seekdb/OceanBase MySQL-compatible DDL may implicitly commit, so whole-upgrade rollback is not guaranteed. Record steps/results; inspect/plan before resume/manual recovery, no blind repeat of nonidempotent DDL | -| TTL/runtime data | Session/operation/receipt/staging cleanup follows Sections 6–7 and preserves tombstones/receipts. Stable bindings, published Artifacts/Sources, and referenced mappings do not expire with operations | -| Downgrade/release | Disable new imports without deleting materials; no unverified schema downgrade. Bundle verification proves only managed postconditions, not whole-database/authorization/end-to-end readiness | +Never execute code, macros, notebook cells or HTML scripts. Disable XML DTD/external entities. Reject archive absolute/traversal paths, symbolic/hard links, devices and duplicate overwrites. Reject encrypted/DRM input explicitly. OCR defaults local; cloud services require explicit deployment and visible disclosure, without runtime model downloads. Logs/errors/metrics exclude text, signed URLs, credentials and private paths. -Deliver empty-database, existing-data upgrade, repeated execution, interrupted recovery validation on all three -backends; existing Sources/Artifacts/exact refs must remain intact. Ship scripts/manifests/indexes/initialization with -the feature; SQLite development auto-creation alone is not production support. +## 11. Errors and observability -### Contract and acceptance +Retain 401/403/404 visibility rules. Invalid fields 422, hard limits 413, unsupported media 415, state/idempotency/version conflicts 409, unconfirmed expiry 410, quota 429 and unavailable dependencies 503. After 202, parsing/publication failures appear in state objects returned by successful GET 200. Network timeout can mean unknown outcome: replay the same key or query C. 429/503 includes Retry-After. -HTTP acceptance covers Scope paths and ancestry, repeated body identity rejection, standard source_type references and flat Artifact receipts, created=201/updated or reused=200, distinct 400/409/410/422 behavior, original 202 replay, asynchronous failures returned by successful GET 200, parse_status/commit_status mapping, and unchanged generic Artifact ETag/304/412/428 behavior. Both locale operation tables, defaults, examples, and OpenAPI must agree. The long infographic remains a separate explanatory artifact, not an RFC embed. +Measure upload, complete/partial parsing, T1 acceptance, pending/blocked/failed/conflicted/completed publication, index readiness and downstream processing separately. Never count 202 as completed import. Observe generations, recovery delay, backlog age, parts and budgets. Logs may correlate commit_id; metrics avoid high-cardinality identities. Reuse repository operation/unit/outcome/error conventions; logging failure cannot alter transactions. -Use openapi/powercontext.yaml as source of truth, run make api-generate and make contract-test, never hand-edit -generated code. Implementation validation includes make check, relevant Source/Server/Access/Dashboard/Connector tests, -make docs-test, and tox when Python support changes. Cross-component acceptance belongs in tests/e2e/. RFC acceptance -does not mean these feature tests have passed. +## 12. Implementation, migrations and acceptance -| Scenario | Acceptance requirement | -| --- | --- | -| Complete Scope-authorized read/write path | scope.read + scope.contribute suffice for import/list/detail/history/body without Artifact owner/Prompt administration; corresponding paths stop after revocation | -| Generic restore/merge, POST/PUT, retired targets | HTTP/Runtime both reject without orphan Revision/lost lineage.sources; no UI actions or retired revival | -| Ordinary content roles/eligibility | working_note=False, internal=None, handoff_receipt=False; forged roles rejected; eligible parts reach configured processors | -| Atomicity, replay, lost notification | Parts, dirty/pending, Artifact, Head, receipt commit/rollback together; replay does not retrigger; lost notify remains recoverable | -| Three-backend schema lifecycle | Equivalent constraints, existing data readable, no false ready after DDL interruption; replicas obey maintenance | -| Static/runtime/Scope discovery | Contract is not enablement; ordinary users need no server.observe; profile failures isolated; disabling imports preserves history | -| Body comparison/search boundary | Exact Source body/semantic comparison; explicit budget warning; history independent of staging; absent searcher unsupported, namespace not full text | -| Required existing Scope/missing-invalid target | UI blocks; malformed/missing 422, invisible/missing/unauthorized 404/403; no accepted input/operation | -| Fixed target/navigation/cross-Scope | Immutable nonempty scope_id; child operations cannot override; targets/bindings check ownership; changing destination needs new session | -| Ordinary permissions/revocation/invalid destination | No server.admin needed; confirmation rechecks and never repairs/creates target | -| Multiple sessions/quotas/concurrent commits | Charge fixed Scope from acceptance; atomic reservation/release, no quota migration/bypass | -| CLI/OpenDAL target setup | Missing existing Scope rejected before upload; resumed target fixed; conflicting --scope fails | -| Equal names, renames, equal independent content | Names/digests are not identity; explicit updates retain ID, independent creates stay distinct | -| A→B→A, old replay, Head races | R3 for returning content; old receipt does not roll back Head; 409 conflicts never overwrite | -| Parser/filter changes, renames, member reordering | Configuration does not misidentify documents; no guessed correspondence without stable IDs | -| Metadata/parts/lineage/reverse lookup | Exact versions/parts agree; no duplicated body, invented full consumption, or future relationships | -| Type/wire name | ImportedContent/importedcontent consistent across persistence, registration, API, clients, Dashboard; four standard reads work | -| Unified content across inputs | Paste, HTML/PDF, members, OpenDAL persist content; SourceRef types agree; webpage acquisition remains a Connector | -| Constant source_kind/default normalization | Omitted/explicit content share fingerprints/digests/cache semantics; others 422; no constant-field filter index | -| Cross-input updates/history | File→paste still checks Head/snapshot; each content version keeps origin/format/parser | -| New preview/existing update | No new public Artifact/Source before confirmation; old Head survives parse/failure/retry/cancel | -| Generic writes/internal-state forgery | No direct forged Artifact or arbitrary operation PUT/PATCH; reject cross-session handles | -| Private operations/ordinary Scope readers | Other readers cannot preview/confirm private input; standard reads expose accepted content only | -| Operation cleanup/governance | Exact history survives cleanup; progress creates no Revision/governance parsing state | -| Default routing/backend loss/one fallback | Pinned profile/router; no implicit OCR/external transfer; shared budgets; frozen output immutable | -| LiteParse limits/page_errors/complexity/OCR | Probes are not execution; missing pages not complete; fixed Docling complex-layout branch | -| Full/values-only XLSB | Missing formula semantics keeps planned; values-only explicit needs_review; complete-only rejects | -| OOXML template/macros, ODF/flat XML/repeats | Correct original-container relationships; no suffix hacks/macros; enforce node/expansion budgets before loading | -| Office audit/converted locations | Selected revisions/footnotes/hidden/notes retained; no invented native coordinates from converted pages | -| Service tasks/signed resources/temporary images | Versions/transfers traceable; saved text independent of expiring URLs; no permanent-image promise | -| Every claimed format/generation | Real fixtures with verifiable body/locations/omissions; detection alone fails acceptance | -| TXT/code/config and UTF-8/UTF-16/GB18030/Big5 | Strict decoding, Chinese and lines correct, no silent replacement | -| Native/scanned/mixed/multicolumn/table PDF | Page order, OCR deduplication, omissions correct; missing OCR never full success | -| Word tables/footnotes/revisions/comments | Declared view/selection respected; no silent selected-content loss | -| XLS/XLSX/XLSB, multisheet/hidden/missing caches | All sheets, no recalculation/invented values; verifiable locations | -| PPT/PPTX/templates/notes/hidden slides | Correct order/inclusion; no macro execution | -| HTML/notebook/EPUB/EML/MSG/MBOX | No external fetch/execution; correct alternative body/child boundaries | -| OCR images/multipage TIFF/pixel limits/blank pages | Enforced limits, no metadata-as-body, traceable outputs | -| Office/EPUB versus ordinary ZIP/TAR | Correct distinction; traversal/nesting/bombs bounded | -| Missing library/metadata-only result | No available declaration or empty Source | -| Long text/many parts/overflow/huge paragraph | Concatenated digest matches complete text; overflow fails whole document | -| Disconnect/crash/stale lease | Recover same operation; old generation cannot overwrite frozen output | -| Retry after OCR preview/concurrent confirms/profile upgrade | Confirm frozen bundle; identical semantics reuse, changed output preserves old evidence | -| Pre/post-save digest and differing refs/times | Receipts do not affect content digest; same target/expected Head has at most one successful advance | -| Two Principals/equal content in one Scope | No shared private operation; independent creates differ; updates check permission/Head | -| Expiry/idempotency cleanup/worker recovery | New attempt keys progress without duplicate committed materials/early checkpoint | -| Discovery/selection/partial commit/cancel/GC races | Consistent states/aggregation; committed work remains, uncommitted cannot pass cancellation/expiry | -| as_text/reparse/selection/retry/cancel | API/CLI/UI agree; invalid combinations reject; same current semantics do not duplicate Sources | -| Replicas/clients/quota/crash | Atomic cumulative reservation; release disk only on deletion; no OCR refund; durable backoff | -| A accepted partial; B/new attempt hits snapshot | needs_review/uncommitted with omissions and explicit acceptance; reuse only same current document snapshot; independent create remains new; successful same-key replay needs no new confirmation | -| OpenDAL cached/recovered partial | complete-only rejects; no checkpoint advance merely because Revision exists | -| New key with old expired/cancelled/failed/completed task | New operation, coverage-aware persistent reuse; old key retains old result | -| Transient discovery failure/retries exhausted | stage=discovery works without item_ids; publish complete manifest atomically, retain frozen IDs | -| Retry slotless failed work, including partial commits | 429/503 leaves state/counters unchanged; later same-key admission holds one slot and enqueues once | -| Awaiting partial confirmation/only failed members/one failure | Waiting confirmation holds slot; inactive failure may release; do not release other active resources or extant bytes | -| Any part/record/pending write failure | Whole document rollback; recovery does not duplicate Sources | -| Failed container members/explicit partial | Accurate per-document outcomes and durable omissions, no false completeness | -| Revocation/cross-Scope/cross-site/malicious file | No unauthorized confirmation/reads/execution | -| Checkpoint/profile/filter changes | Advance only complete committed results; no silent configuration-based skips | -| SQLite/seekdb/OceanBase, shared staging/expiry GC | Consistent recovery/commit; GC cannot destroy confirming bundles | -| Chinese/English, themes, small screens, reopen | Capability lists, previews, omissions, per-item retry, full reading usable | +### Implementation order and release gates -# Drawbacks +1. Pin both plugin dependencies and compatibility with the original master baseline; unmerged dependencies are not completed capabilities. +2. Deliver document_commit input/controlled part reads/evidence validation, Scope-shared Family access and transactional T2 completion, covering maximum document budgets. +3. Implement nine import persistence responsibilities, T1, C queries/retries, idempotency and quotas; generate from HTTP/OpenAPI without editing generated code. +4. Implement the official ImportedContent plugin and shared parsers, preserving all section 1 A/B targets and optional C scope; unvalidated profiles remain planned. +5. Integrate Dashboard/CLI/SDK and the public Source collect/OpenDAL adapter; run recovery and complete-document acceptance. -Broad support increases dependency, fixture, and security-maintenance costs. Legacy Office, complex spreadsheets, and -proprietary generations differ; maintain capability matrices and failures instead of promising arbitrary files. +No second Artifact version system or Source process manager. Parsing workers are distinct from Artifact generation workers. Installing plugins is not completed text import. Keep the infographic separate. This PR proposes design only, without runtime/database changes. -Parser workers, durable staging, leases, and segmented transactions are more complex than text capture. OCR needs -independent speed/accuracy/language budgets. Text/digests cannot reconstruct all binary layout, imagery, or interaction. +### Database and compatibility -Per-document atomic commits append many Sources, increasing transaction and downstream cost. Validate 8 MiB/128-part -ceilings across all three backends rather than raising them without bounds. Containers permit partial success, which -users must distinguish from document completion. +Register import and dependency schemas, initialization, upgrades, indexes, uniqueness and postconditions for SQLite, embedded seekdb and OceanBase. No first-request table creation. Reuse db-migrate status/plan/apply/verify with new revisions and frozen manifests, never rewriting released migrations. Recover interrupted DDL according to backend capabilities; MySQL-compatible DDL does not promise whole-upgrade rollback. -History retains old and new text; Head is latest accepted content, not external real-time truth or deletion propagation. -Successful parsing does not establish complete model understanding. +Backups and maintenance gates cover every writer. Cross-process checkpoints/reservations/receipts use database CAS and uniqueness, not process locks alone. Preserve Source/Artifact history and exact references; existing content is not automatically wrapped. TTL must retain pending evidence and durable result mappings. Report backend execution separately: local SQLite can be validated first; unavailable seekdb/OceanBase instances remain untested rather than inferred from SQLite. -# Rationale and alternatives +### Required behavioral acceptance -**ImportedContent Artifact + Revision.** Existing immutable versions, Head, lineage, and reads simplify -history/relationship queries versus ImportRecord, at the cost of Family registration, dedicated transactional -writes/readers, bindings, indexes. Constant content reuses evidence; origin/format/parser remain separate. Reclaimable -ledgers hold execution without FileImportJob or progress Revisions. +| Scenario | Required result | +| --- | --- | +| Ordinary scope.read+scope.contribute user | Preview, confirm, query C and read publication without admin; plugin deployment authority separate | +| Any T1 Source/manifest/receipt failure | No evidence/intent commits and no Artifact | +| Lost wakeup or restart after T1 | Durable recovery, original 202 replay, no duplicate Sources | +| T2 failure or lost reply | Artifact, Head, indexes and completion record commit/rollback together; retry produces at most one Revision | +| 80 pages/40 parts crossing journal windows | Commit-scoped complete assembly; missing/out-of-order/digest-invalid evidence blocks, not partial publication | +| 8 MiB/128 parts and 2 MiB manifest | Test separate input budgets and 4 MiB draft limit; other window budgets unchanged | +| Forged commit metadata/references or cross-Scope input | Reject; no wrapping ordinary content or expanding evidence authority | +| Concurrent updates and A→B→A | One outstanding commit per document, correct R1/R2/R3, no stale Worker overwrites | +| Revocation, retirement or Head change after T1 | Blocked/conflicted; preserve old revisions/evidence without silently retargeting | +| Plugin disable/unload/reload | Published reads survive; pending commits durable; stale generations cannot commit; recover after readiness | +| 24-hour expiry/cancel and 30-day mapping expiry | Clean private input only; C still verifies accepted/published outcomes | +| collect/checkpoint crash boundary | T1 is not completion; exact C result restores CAS without duplicate publication | +| Format matrix | Real structure/encoding/formula/cache/revision/comment/OCR/container/location/omission samples; metadata-only fails | +| HTTP and locales | Same 17 import operations; acceptance 202/reuse 200/state GET 200; standard conditional requests unchanged | +| Databases and UI | Independent backend evidence; distinguish preview/evidence/publication/index readiness | + +Implementation runs make api-generate, make contract-test, make check, focused Source/Artifact/Access/Parser/SDK/Dashboard/Connector tests and make docs-test. Cross-component tests belong in tests/e2e; use tox when supported Python versions are affected. Document checks are not runtime acceptance. -**Explicit identity and fixed existing Scope.** Distinguish creation/update, use authorized stable keys for collection, -avoid filename/content merging. Sessions own target/private preview/recovery, not Scope creation, permissions, quota -migration. Upload, confirmation, and accounting share one authorization boundary. +# Drawbacks -**Native structure separately from PDF layout.** Sheets/slides/email/archives have native object/member identities and -are not universally printed to PDF. LiteParse handles simple PDF, Docling layout/OCR, independent profiles complex -alternatives. Multiple dependencies buy explainable structure/locations/omissions; RAGFlow/LlamaIndex jobs, -deduplication, indexes, and default Readers do not own persistence here. +Two stages permit visible evidence without a published Artifact. Publication failure cannot roll back evidence already consumed downstream. Durable handoff and recovery UI are necessary. One outstanding commit blocks further updates to that document but avoids undefined publication ordering. -**Pluggable parsers with fixed routing.** UTF-8-only APIs cannot handle Office/PDF, and one generic engine cannot -guarantee all semantics. Validated adapters are more diagnosable than implicit fallback and allow future extensions. +Complete-document input and transactional completion require framework extensions, not just two plugin declarations. Large-document budgets, parser dependencies and OCR cost still require real samples; plugin packaging does not remove these costs. -**Isolated parsing, reused ContentSource.** Keep dependencies and file risk outside Core requests; normalized evidence -reuses Source/Scope/lineage/processing without a Source family per suffix. +# Rationale and alternatives -**Asynchronous parsing, synchronous per-document commit.** OCR/large inputs exceed normal HTTP timeouts. Operation creation returns -durable recovery handles; confirmation atomically commits frozen output instead of exposing partially parsed Sources. +Choose two stages to reuse Artifact plugin execution/publication. T1 atomically accepts document evidence and intent; T2 atomically publishes the Artifact and completion record. Holding an HTTP request open does not combine transactions. -**Segmentation instead of truncation/manual splitting.** Natural boundaries/locations remain usable, exact -concatenation/digest preserve completeness. Atomic saving avoids half-published documents but does not force every -consumer to process the entire body in one window. +Source-only capture can use content.inline but lacks document identity, revisions and import-result reading. Synchronous Source+Artifact atomicity is a different contract needing a controlled publication API, not the current generate workflow. Per-window aggregation needs durable accumulation and historical evidence access and risks premature cursor advancement. This proposal uses a complete commit as its input unit. -**Extraction is not universal understanding.** Body/table/OCR/metadata coverage differs. Report omissions, never infer -missing text or count metadata as body; retain explicit partial acceptance. +Use native structure and PDF layout routing with mature parsers; RAGFlow/LlamaIndex do not own versioning, authorization, jobs or transactions. Body text is stored once in Sources and precisely referenced by ImportedContent. Future body indexing needs an explicit derived projection, not body duplication to evade input budgets. -**Library support is not product support.** Generations, dependencies, options, quality, budgets, and fixtures all -determine availability; discovery advertises validated deployed adapters only. # Prior art @@ -1680,22 +1243,10 @@ Upstream references inform adapters, not proof of completed PowerContext support # Unresolved questions -1. Default parser families/candidate boundaries are fixed in Section 1. Freeze release dependency versions, PDF thresholds, and performance budgets after fixture validation. XLSB formulas, legacy Word revision views, and long-tail gaps remain planned until implemented; partial profiles cannot imply full delivery. -2. Validate 50 MiB input, 8 MiB output, 500 pages, 128 parts on SQLite/embedded seekdb/OceanBase and actual deployment resources, including transaction/downstream costs. -3. Validate initial OCR language packs and proprietary generations with real materials. Apple/WPS availability requires body fixtures, not detection alone. -4. Permanent binaries/downloads need separate quotas, retention, exact-evidence, and deletion authorization; this proposal retains parser staging only. -5. If downstream consumers must use only Sources from document Head, separately define version selection, old pending work, and historical references. Head does not change current Source consumption rules automatically. +1. Finalize document_commit SDK names, physical mappings and adapter composition with framework implementation; complete evidence validation, two-stage transactions and budgets are mandatory acceptance constraints. +2. Validate 50 MiB files, 8 MiB text, 128 parts and OCR quotas with real samples and each backend before advertising availability. +3. Decide whether title/origin search is a default deployment declaration before registering the first searchable schema. Body indexing requires a separate contract, not assumed hot-update support. # Future possibilities -Extend registered parsers/capabilities to more languages, domain formats, and proprietary documents. Define -transcription, chart reasoning, formula semantics, and handwriting separately; their cost/accuracy cannot hide inside -ordinary parsing. - -Yuque, webpages, and knowledge platforms use separate Connectors with explicit external identities, versions, -authorization, and exact reads. Databases/very large datasets need selective acquisition protocols rather than unlimited -document expansion. - -Future work may include permanent binary archives, original-text search, and exact-location navigation. Document-level -generation progress must use real per-Family Source coverage/output evidence. Preserve Scope boundaries, snapshot -identity, omissions, and durable receipts in every extension. +Add other trusted Source connectors, permanent original-file archives, explicit body-search projections and cross-document reference browsing. Containers still publish per document; downstream Memory reports its own coverage/completion evidence. Extensions preserve existing Scopes, stable document identity, exact revisions, explicit omissions and recoverable commits. diff --git a/docs/zh/rfcs/2073-text-material-import.md b/docs/zh/rfcs/2073-text-material-import.md index 7e31ca3789..60f9a18418 100644 --- a/docs/zh/rfcs/2073-text-material-import.md +++ b/docs/zh/rfcs/2073-text-material-import.md @@ -9,28 +9,27 @@ title: 文本资料导入 - Design Baseline: [PowerContext master / cab054db（2026-10-19）](https://github.com/oceanbase/powercontext/tree/cab054dbf6d86e2311b079388e5158178257def2) - Related RFCs: [Source Definition 与 Observation](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1400_source_definition_and_observation_model.md)、[Source 与 Artifact REST API](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1437_source_artifact_rest_api.md)、[Source 访问与生成准入](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1458_artifact_generation_source_access.md)、[Scope 与 Source 发现](https://github.com/oceanbase/powercontext/blob/cab054dbf6d86e2311b079388e5158178257def2/docs/zh/rfcs/1502_scope_source_discovery.md) +- Plugin Baselines: [Source #2210](https://github.com/oceanbase/powercontext/pull/2210) (`6ef5c494d701c355686806b6b1042d5729267690`), [Artifact #2246](https://github.com/oceanbase/powercontext/pull/2246) (`9fe2e68220a6e0aa8ebe4b78900cd12421db5cc3`). + # Summary -为 PowerContext 提供覆盖常见文本、办公文档、网页文件、表格、邮件、电子书、代码、图片与归档包的资料导入能力。用户必须选择已有且具有 scope.read + scope.contribute 权限的 Scope 后才能导入；粘贴文本、上传文件，或通过独立 OpenDAL worker 持续采集本地目录和对象存储。输入经过格式识别、编码处理、文档解析、可选 OCR 和有界分段，保存为不可变的 `content` Source，并保存为 `ImportedContent` Artifact（Family 标识 `importedcontent`），返回精确 Revision 回执。 +为 PowerContext 提供文本、办公文档、网页文件、表格、邮件、电子书、代码、扫描件与归档包导入。用户必须选择已有且具有 `scope.read` + `scope.contribute` 权限的 Scope；粘贴或上传后，先解析和私有预览，再明确确认。持续采集使用 Source collect 插件和相同解析组件。 -采用“广泛格式支持、明确能力分级”的产品契约。轻量文本可直接保存；复杂文件通过隔离的解析 worker 处理；扫描件及图片需要显式启用 OCR。标准文档能力包覆盖 PDF、现代与传统 Office、OpenDocument、HTML、邮件和电子书等常见格式。部署必须公开实际可用的格式、解析器与限制，不能把文件类型识别成功或仅提取到元数据当作正文导入成功。 +确认采用两个持久阶段：T1 原子保存一份文档的全部 `content` Sources、不可变提交清单、发布意图和接收回执；T2 由官方 ImportedContent Artifact 插件确定性组装制品，框架原子发布 Revision、Head、lineage、已声明索引与完成回执。T1 不生成 Artifact；两阶段之间允许“证据已保存，制品待生成”。`202` 不是导入完成，只有 T2 成功或明确复用已发布版本才是完成。 -一份逻辑资料对应一个 ImportedContent Artifact，source_kind 固定为 content，统一使用现有文本证据类型，每次接受的新快照对应一个不可变 Revision，正文保存在其精确引用的 ContentSource 中。导入成功包含导入资料制品及证据保存，不承诺 Memory 等下游制品已经生成、原始二进制已归档或原文已建立检索索引。Head 表示 PowerContext 最近接受的版本，不代表外部文件的实时最新版；持续采集不传播删除。本文是拟新增契约，不代表功能已经发布。 +一份逻辑资料对应一个 `ImportedContent`，Family 为 `importedcontent`，`source_kind` 固定为 `content`。正文只长期保存在 Source，Artifact 记录文档信息、解析 provenance、遗漏与有序精确引用，不需要 LLM 或 Candidate 审批。不新增 ImportedContent 主表、版本表或 FileImportJob 业务资源。 -已保存资料复用 Artifact 表、标准读取、Revision 与 lineage。导入写入由 ImportedContent Family 的专用操作负责；运行进度和私有预览只进入可清理的内部操作账本，不作为制品版本，也不新增 FileImportJob 业务资源。 +本 RFC 区分插件已有能力、本文需要的框架扩展与导入专属能力。两项插件 PR 均是依赖基线，不代表已经合并或已支持完整文本导入；完整文档输入装配、持久交接和事务完成回调是交付前置条件。解析 profile 按部署真实状态公开 available/unavailable/planned，不把识别成功当作正文提取成功。长图作为独立说明材料，不嵌入 RFC。 # Motivation -项目资料不只存在于 TXT 和 Markdown，也广泛存在于 Word、PDF、Excel、PowerPoint、网页导出、邮件与扫描件。只扩展扩展名白名单无法完成导入：二进制格式需要解析，扫描页需要 OCR，表格需要保留行列关系，长文需要分段，解析不完整时还需要用户可见的错误与恢复方式。 - -仓库已有 ContentSource、Scope 权限、Source journal、文本投影、Source 集合与精确读取，以及 OpenDAL evaluation Connector 的 durable receipt 和 checkpoint CAS。这些能力构成保存与恢复基础，但不等于已经实现通用文件解析。 +Source 插件统一采集入口、目标与运行管理；Artifact 插件统一制品生成、版本、读取及声明式检索。导入不重复建设这些框架，但仍需处理文件上传、私有预览、用户确认、完整文档边界和失败恢复。普通 Source journal 窗口不能代替一份文档的完整输入，插件 checkpoint 也不能代替用户确认回执。 -现有 OpenDAL evaluation 使用 `text-file-snapshot` 类型，worker 会注册 Definition manifest。它的 UTF-8 文件采集、权限要求和公开读取范围不直接构成本提案的文件支持合同。本提案复用内置 `content` Source，新增独立的资料导入应用服务与可选解析部署，不改变既有 Source Create/Capture 接口。 - -目标是让用户知道：当前部署能读取哪些格式、提取了哪些内容、遗漏了什么、材料保存到哪个 Scope，以及重复提交和失败时如何继续。 +文件解析在证据保存前完成，保证预览正文与最终接受正文一致。制品生成只消费已接受的精确证据，不重新下载原文件，不重跑 OCR。Memory 等下游处理仍由各自配置控制，证据保存或 ImportedContent 发布均不保证下游生成完成。 # Guide-level explanation + ## 支持范围与能力分级 以下是本 RFC 的目标支持矩阵。A 为基础能力，B 为标准文档能力包，C 为可选 OCR 或扩展适配器。发布时每一项都需要实际样本验收；未安装依赖显示“未启用”，未完成适配显示“尚不支持”，不得统一标为支持。 @@ -65,36 +64,28 @@ A、B 是正式文件导入的交付范围；C 是同一扩展机制下按部署 ## 用户流程 -用户先选择已有且可贡献的 Scope，再进入文本或文件导入。全局入口必须明确选择目标；从某个 Scope 内进入时可预选当前 Scope，但仍须显示目标并校验权限。未选择有效目标时不能创建 ImportSession、提交粘贴文本或上传文件；没有可用 Scope 时提示先在现有 Scope 管理流程中准备目标。导入页面不提供创建 Scope 的表单或自动创建行为。 +从 Scope 内进入可以预选目标，全局入口必须选择已有 Scope；没有可用目标时引导到既有 Scope 管理流程。导入页不创建 Scope。会话固定 Scope 与创建者，切换目标需新会话。 -选定 Scope 后，页面展示当前部署支持的格式、实际限制和调用方权限。创建 ImportSession 时固定非空 scope_id 与创建者；同一会话的全部输入都属于该 Scope。改变目标必须创建新的会话，不能迁移正在解析或待确认的输入。 - -纯文本校验后可直接确认保存；文件先上传并解析，展示正文预览、页/表/幻灯片数量、分段数、OCR 使用情况及遗漏，再确认保存。确认时重新校验目标 Scope 及写入权限，在同一事务内保存单份资料的 Source、ImportedContent Revision、Head、lineage 和回执。失败、取消和只预览均不产生已提交资料。 +页面先展示可用格式、限制、解析与发布就绪状态。文本请求本身即确认；文件展示正文、页/表/幻灯片数量、分段数、OCR 与遗漏后再确认。部分提取必须显式接受；取消、到期和未确认预览不产生正式 Source。 ```mermaid flowchart TD - A["选择已有 Scope"] --> P["校验 scope.read + scope.contribute；创建固定 scope_id 的 ImportSession"] - P --> B{"输入方式"} - B -->|粘贴文本| C["校验与规范化"] - B -->|上传文件| D["文件操作创建：耐久暂存与隔离解析"] - D --> E["预览正文、分段、OCR 与遗漏"] - E -->|无可用正文| X["失败；不保存资料"] - E -->|完整确认或显式接受 partial| F["确认冻结 bundle_digest"] - C --> G["重新鉴权；在会话固定 Scope 内单份资料原子保存"] - F --> G - G --> I["ContentSource 分段 + ImportedContent Revision + lineage + 回执"] - I --> J["后续文件使用同一 Scope；下游处理按既有配置运行"] + A["选择已有 Scope"] --> B["粘贴 / 上传"] + B --> C["共享解析；文件私有预览"] + C --> D["明确确认冻结结果"] + E["Source collect 插件"] --> F["采集；相同解析；complete-only 策略"] + F --> D + D --> T1["T1：Sources + 提交清单 + 发布意图 + 接收回执"] + T1 --> Q["证据已保存；制品待生成"] + Q --> P["ImportedContent Artifact 插件"] + P --> T2["T2：Revision + Head + lineage + 完成回执"] + T2 --> V["导入完成；标准 Artifact 阅读"] ``` -普通导入只要求目标 Scope 的 `scope.read` + `scope.contribute`，不要求 `server.admin`。会话不能替代当前授权；上传后被撤权、目标不存在或不可写时，确认不得继续保存或自动改换目标。管理代操作遵循第 10 节。 - -第一次手动上传或粘贴默认“新建文档”；系统分配新的 artifact_id。更新现有文档必须从该文档进入“更新文件/文本”，明确指定 artifact_id 和预期 Head Revision。文件名相同、正文相似或字节摘要相同都不能自动合并文档；只可提示可能重复并让用户选择。重命名上传文件不影响明确指定的文档身份。 - -多份文件分别呈现待上传、解析中、待确认、已保存、已存在、需处理、版本冲突和结果待核实。文件名、已选择的目标 Scope、新建/更新文档意图始终可见。不用文件名推断既有文档身份；并发保存仍按同一 Scope 下的幂等、配额及 Head 并发规则处理。 +状态明确区分“解析中 / 待确认 / 证据已保存、制品待生成 / 生成阻塞或失败 / 导入完成”。T1 后证据已在 Scope 可见，其他获准处理器可以消费；新资料在 T2 前不出现在 Artifact 列表。更新期间旧 Head 始终可读，只有 T2 成功才推进。 -例如，Word 提取成功、PDF 中含扫描页但未启用 OCR、Excel 的公式缺少缓存值时：Word 可以确认保存；PDF 必须说明受影响页，启用 OCR 后新尝试或显式接受部分文本；Excel 显示公式和“未保存计算结果”，不编造数值。不能把三份都显示为完整提取。 +同名文件或相同摘要不自动合并。首次手动导入创建新的文档身份，更新必须明确指定 artifact_id 与 expected_head_revision。一个文档同时最多存在一个未终结的发布提交；失败后恢复同一提交，不再上传同一文件。已保存证据不会因后续制品失败或取消剩余输入而回滚。 -标准解析按照已声明范围保存，不能承诺完全还原二进制。界面使用“查看提取文本”；纯文本显示“查看原文”。图片、布局、图表和嵌入对象的处理范围与已知遗漏必须可见。解析完成与预览可见都不等于已经保存 Source 或 ImportedContent Revision。 ## 格式特有行为 @@ -106,31 +97,31 @@ flowchart TD - **PDF/OCR**：按页选择文本层或 OCR，避免同一区域重复叠加。空白页、图片页、解析失败页分别报告；低质量 OCR 输出是机器识别结果，不能称为精确转录。布局/页数信息不可得时报告未知，不能伪造完整性。 - **邮件与归档**：邮件正文中的引用历史保留，纯文本与 HTML 二选一并记录规则，避免重复。附件默认列出但不导入；用户选择后逐附件处理。归档默认展开第一层，嵌套包需显式开启且受深度限制；失败或未选成员在清单中保留。 -## 保存与后续处理 - -一份逻辑文档对应一个 `ImportedContent` Artifact，接受的快照对应 Revision；长文按结构分成多个不可变 ContentSource。单份文档的全部分段、Revision、Head、lineage、待处理登记和回执在同一事务提交。事务仅写入会话固定的已有 Scope，不创建 Scope 或初始化权限。跨文件、归档成员和邮件间允许部分成功。 - -成功表示导入资料制品和正文证据已可靠保存，不需要 LLM 或 Candidate 审批。Memory 等既有处理器是否消费这些 Source、处理多少、产生什么候选或制品，继续由原有配置和领域契约决定。importedcontent Family 不默认注册自动生成处理器，不能因称为 Artifact 就触发递归生成。 - -历史 Revision 与其 Source 保留，不因 Head 变化而覆盖。资料详情可以显示版本历史，以及按精确 SourceRef 或真实 ArtifactRef 反查的下游引用；不在导入时预填未来“关联制品列表”。一次 consumer window 不保证处理全部分段，不虚构全文理解完成状态。 - -原文件仅用于解析暂存，默认不是长期文件档案。保存后保留提取文本、文件摘要、解析版本、定位及遗漏；下载原始二进制和长期存档不属于本期承诺。 - -## 持续采集与 Dashboard +# Reference-level explanation -OpenDAL worker 在存储附近运行，持有读取凭据并把文件字节提交给导入服务；Server 不通过用户提供的路径或 URL 读取任意文件。文本与复杂文件使用同一份能力矩阵。周期执行仍由外部调度器负责。 +## 1. 架构与插件复用边界 -Dashboard 保持既有 Tabler、Jinja2、HTMX、中英文及明暗主题。写入口由 `POWERCONTEXT_SERVER_DASHBOARD_TEXT_IMPORT_ENABLED` 显式开启，默认关闭，继续满足现有静态 Bearer 和 enforced access 前置条件。文件解析另要求解析服务可用，关闭该服务不影响已保存资料阅读。 +| 能力 | 依据与状态 | 本功能如何使用 | +| --- | --- | --- | +| Source 能力发现、插件加载、Scope 权限、write/collect 分流 | 复用 #2210 | 管理员部署，普通用户在已有 Scope 操作；发现能力不等于格式解析已实现 | +| collect 实例、目标、独立 Runtime、启停与 checkpoint | 复用 #2210 | OpenDAL 作为采集插件；不再自建平行实例管理服务 | +| `content.inline` 同步写入 | #2210 已有，完整导入不直接调用 | 仅适合单 Source 捕获；不能替代完整文档确认事务 | +| Artifact Family/schema、ArtifactDraft、Worker、生命周期 | 复用 #2246 | 官方 importedcontent 插件确定性生成，不调用 LLM | +| Revision、Head CAS、lineage、标准 get/list/历史读取 | 复用 #2246 和现有 Artifact 存储 | 不建独立资料版本系统；Scope 共享读取策略需显式适配 | +| 声明式索引、构建与查询 | 复用 #2246，按部署声明启用 | 首期仅可声明标题/出处等 content 内字段；不自动解引用 Source 做全文检索 | +| 上传、受控暂存、私有预览、解析与确认 | 本 RFC 新增 | 文件 worker 与 Server 同步 write 隔离；保留导入专属接口 | +| `document_commit` 完整文档输入、精确分段读取与证据校验 | 本 RFC 要求的框架扩展 | 替代本 Family 的 journal-window 触发；不是已有 InputContract 字段 | +| T1 持久交接、T2 同事务完成回调、提交回执与恢复 | 本 RFC 新增集成 | 连接 Source 和 Artifact 两个事务；不能仅发送内存通知 | +| collect 调用导入提交并确认 checkpoint 的公开适配器 | 本 RFC 要求的 Source 集成扩展 | 不访问 PluginContext 私有 client，不把普通 run_batch 当整文档提交 | -页面结构为“选择已有 Scope → 资料列表 → 选择文本/文件 → 解析预览与问题清单 → 确认保存 → 精确阅读”。切换 Scope 不改变正在运行任务的归属，也不自动提交未完成输入。关闭浏览器后可从任务记录恢复已接受的文件解析；未上传完的本地文件需重新选择。 +固定依据：[Source write](https://github.com/oceanbase/powercontext/blob/6ef5c494d701c355686806b6b1042d5729267690/src/powercontext/source_management/write.py)、[Source Context](https://github.com/oceanbase/powercontext/blob/6ef5c494d701c355686806b6b1042d5729267690/src/powercontext/source_management/context.py)、[Artifact SDK](https://github.com/oceanbase/powercontext/blob/9fe2e68220a6e0aa8ebe4b78900cd12421db5cc3/src/powercontext/artifact_plugins.py)、[Artifact Worker](https://github.com/oceanbase/powercontext/blob/9fe2e68220a6e0aa8ebe4b78900cd12421db5cc3/src/powercontext/builtin/runtime/artifact_plugins.py)。这些链接说明可复用基线，本文新增契约必须另行实现并验收。 -# Reference-level explanation +Source write 当前在 Server 内同步执行，一次提交一个 observation，add JSON 上限 1 MiB、observation 上限 4 MiB；不承载大文件或 OCR。自定义 write 禁止覆盖内置 Definition，不能靠重新注册 content 接管本功能。collect Runtime 可独立部署；当前本地 collect 进程管理要求 Linux，不能宣称 macOS/Windows 上已完成本地托管。 -## 1. 架构 +Artifact 当前按 journal 窗口提供证据，输入预算 1 MiB，draft 直接 SourceRef 必须来自该次调用；GenerationContext 的 Artifact 读取不能代替任意历史 Source 读取。ImportedContent 的 8 MiB/128 段文档范围需要专门的 document_commit 输入模式，不能直接调大 max_sources 或跳过校验。此模式通过受信服务端适配器装配，不改变其他 Family 的窗口与预算。 -入口、解析和保存之间的责任边界见“用户流程”图。文件通过 ImportedContent Family 的文件操作创建接口产生冻结的 ImportedContentBundle，执行过程由内部操作账本和隔离 worker 承担；用户确认或 worker 按显式策略接受后，ContentImportApplication 才执行单文档原子提交。制品处理仍沿用已有配置与进度机制。 +同一 importedcontent binding 仅启用 document_commit 模式，禁止同时消费全部 content journal。新插件默认 disabled；部署必须加载并显式启用，还须确认 schema、权限、文档预算与适配器就绪。解析器依赖在部署时安装，复杂解析在独立 worker，Artifact 插件只消费已保存的精确证据。 -文件获取、解析、文本保存与制品生成是四个边界。Core 不导入任意用户指定 parser，不运行 Office 宏，不执行代码。正式部署提供独立 parser worker；轻量文本路径不要求文档解析依赖。 ### 1.1 解析器职责与固定选型 @@ -329,53 +320,45 @@ B 级表示目标交付范围而非无条件可用承诺。未达到必需语义 同一后缀可有一个标准完整 profile 和显式选择的部分 profile；默认路由不得自动从完整切到部分。上传时校验所选选项是否在能力范围内：未知或矛盾组合为 422，识别后发现必需语义不受支持则解析失败 `capability_insufficient`。不能以 accept_partial 绕过格式识别、安全、大小、解密及用户选择的必需语义。仅已声明允许部分结果的 profile，可在有正文且遗漏可解释时生成 needs_review。 -## 3. 数据模型 +制品发布就绪状态单独报告为 publication；Source 插件能力发现不等于文件格式支持。document_commit 输入适配、ImportedContent 插件或声明 schema 不可用时，允许读取和私有预览，但确认返回 503 publication_unavailable。确认与插件状态变化竞争时，以事务门禁判断；已接受提交等待恢复，不能丢弃。 + +## 3. 数据模型与持久结构 -新增 `ImportedContent` 类型及 `importedcontent` Family（显示名“导入资料”），由 ContentImportApplication 管理写入。一份逻辑文档对应一个 Artifact，各次接受的快照对应不可变 Revision。不另建独立 ImportRecord 作为第二份资料身份与内容权威。ImportedContent 表示已经接受的内容，不表示上传任务、解析中间态或模型生成的 Memory。 +ImportedContent 是资料身份；ImportSession/operation/item 是输入尝试；commit_id 是一次已经确认的不可变快照及其发布交接。提交状态不是第二份资料目录，不提供任意 CRUD，也不是 Artifact Revision。 ```mermaid -flowchart TD - I["ImportSession：创建者、固定且非空的已有 scope_id"] --> J["内部 operation / item_id：临时解析与确认状态"] - J --> B["ImportedContentBundle：冻结内容"] - B -->|原子提交| R["ImportedContent Artifact Revision"] - S["Scope"] --> H["Artifact Head：最近接受的版本"] - H --> R - R -->|有序 lineage.sources| C["ContentSource 1..N：正文与定位"] - D["下游 Artifact Revision"] -->|实际使用的精确 SourceRef| C - D -.->|确实以文档版本为输入时才记录 ArtifactRef| R +flowchart LR + S["已有 Scope"] --> I["私有 Session / Operation / Item"] + I -->|确认 T1| C["不可变文档提交清单"] + C --> P["有序 Content SourceRef"] + C --> Q["持久发布状态与意图"] + Q -->|插件 T2| R["ImportedContent Revision"] + R -->|精确 lineage| P + H["Artifact Head"] --> R ``` -| 对象 | 职责与持久化 | -| --- | --- | -| ImportSession | 创建者 Principal、固定且非空的已有 scope_id、期限；承载同一目标下的导入恢复、隔离及并发控制，不授予额外 Scope 权限 | -| 内部操作记录 / item_id | 创建文件操作返回 operation_id；记录临时输入、发现清单、租约、配额、解析与确认状态；item_id 仅在 operation 内稳定，不是资料永久身份，也不是 Artifact | -| ImportedContentBundle | 解析后冻结的正文、语义 manifest、coverage 与 digest；确认针对它，不重新计算预览内容 | -| ImportedContent Artifact | 稳定身份 `(scope_id, family=importedcontent, artifact_id)`；首次手动导入或稳定来源首次出现时分配 | -| ImportedContent Revision | 精确快照 `(scope_id, family, artifact_id, revision)`；含结构化元信息和有序 Source 映射 | -| Artifact Head | 最近成功接受的 Revision；更新通过 expected_head_revision 乐观并发控制，不等同外部实时版本 | -| 来源绑定 / 请求映射 | 稳定来源键到 Artifact 身份、输入解析缓存、幂等键到精确回执；属于索引与操作账本，不复制一份 ImportRecord | - -### 类型命名与来源语义 - -领域/Python 类型为 `ImportedContent`，Family 的持久化及 HTTP wire name 为 `importedcontent`。该 Family 统一承载外部资料的已接受快照，`source_kind` 固定为 `content`，表示正文使用现有 ContentSource，不区分粘贴、文件、网页等输入方式。 +以下为逻辑表职责；实现时冻结实际 SQL 名、约束和迁移清单。没有按文件后缀新增的业务表。 -| 字段或概念 | 含义与约束 | -| --- | --- | -| `family=importedcontent` | 管理整份导入资料的身份、版本与分段关系 | -| `source_kind=content` | ImportedContent 元数据中的固定值；所有正文分段均使用现有 content Source | -| `input_media_type` | 原始输入格式，例如 text/markdown、application/pdf、text/html | -| `origin` | 具体出处的 namespace、key 和可选 display_name，不作为授权依据 | -| `parser` | 实际解析器、配置、版本、OCR 和转换链 | - -ImportedContent Revision、冻结 bundle 和读取响应必须包含 `source_kind: "content"`。texts 与文件上传 metadata 可省略该字段，由服务端补齐；显式提供时只能为 content，其他值返回 422。先补齐固定值再计算请求指纹、冻结摘要和缓存键，因此省略与显式 content 在其他字段相同时语义一致。OpenAPI 使用单值约束，Python 模型使用 Literal["content"]；默认值不意味着允许调用方选择其他 Source 类型。 - -此字段不重命名或替代已有 `SourceRef.source_type`。保存时校验所有 parts/lineage.sources 的 `source_type` 均为 content，且与 ImportedContent 的 source_kind 一致；不能把其他 Source 类型或系统保留角色混入正文。标准列表不新增 source_kind 筛选或索引，因为该 Family 的取值恒定；出处仍可按 namespace 筛选。 +| 持久结构 | 归属 | 职责与约束 | +| --- | --- | --- | +| `pc_sources`、Source journal、既有 processing intents | 复用 Core | 不可变正文、精确引用和下游待处理登记 | +| `pc_artifacts`、`pc_artifact_heads`、Source/Artifact lineage 表 | 复用 Artifact | 发布后的唯一资料与版本权威 | +| Family/schema、binding/lease/cursor、索引定义/状态/投影 | 复用 #2246 | 不复制插件注册、调度和检索表；其八张逻辑框架表不算导入专属表 | +| collect 实例、目标、checkpoint 状态 | 复用 #2210 | 不复制实例生命周期；不能作为私有预览或完整文档回执 | +| `pc_import_sessions` | 导入新增 | 创建者、固定 Scope、期限 | +| `pc_import_operations` | 导入新增 | 耐久输入引用、发现、选择、租约、配额与解析状态 | +| `pc_import_operation_items` | 导入新增 | `(operation_id,item_id)` 唯一；冻结解析结果、目标和 commit_id，不复制永久正文 | +| `pc_import_source_authorities` | 导入新增/连接配置适配 | 可信连接身份、固定根/桶、获准 Principal 与 Scope；不能由 origin 自授 | +| `pc_import_source_bindings` | 导入新增 | `(scope_id,authority_id,source_key)` 唯一到 artifact_id；key 可用带原值碰撞检查的摘要索引 | +| `pc_import_request_receipts` | 导入新增 | 调用方、动作、规范化请求指纹、原始 HTTP 状态与响应；精确重放 | +| `pc_import_quota_ledger` | 导入新增 | Principal/Scope/部署共享预留与实际用量；CAS 和对账 | +| `pc_import_commits` | 导入新增 | commit_id、Scope、artifact_id、冻结目标/预期 Head、确认者、完整清单、接受语义与 SourceRef；T1 后不可变 | +| `pc_import_publications` | 导入新增/框架持久交接适配 | commit_id 唯一；状态、generation、租约、重试、完成 ArtifactRef；文档占位保证每个目标最多一个非终结提交 | -输入方式由文本入口、文件入口或获准 Connector 决定，格式和解析 profile 决定如何提取。上传 HTML、PDF、TXT，粘贴文本及将来的网页快照在形成文本证据后都使用 source_kind=content。统一 Source 类型不表示无需解析二进制，也不表示已经支持网页抓取；网页链接获取仍属独立 Connector，本期不提供任意 URL 采集入口。 +导入新增九项逻辑持久职责；是否复用已有配额/连接配置物理表须在实现迁移清单中明确，不能因此省略其约束。提交清单不复制正文，仅保存恢复与精确证据装配必需的信息；待发布记录不受 24 小时暂存 TTL 清除。成功后的轻量 commit_id→ArtifactRef 结果映射随版本保留，保证 collect 与客户端在短期请求回执过期后仍能核实。 -`origin` 使用 namespace、key 及可选 display_name；连接身份、source_authority_id 和稳定来源绑定由获准 Connector 路径生成/校验，客户端填写出处不能证明授权。未来网页采集需由 Connector 提供页面身份、抓取时间和经脱敏的显示 URL；URL 不是服务器任意取数指令，也不默认等同稳定身份。完整凭据、签名 URL、敏感查询参数不得进入长期元数据。归档和邮件成员通过 container_chain 保留父容器及成员身份；不能把无正文的容器伪造为 ImportedContent。 +`source_kind` 固定 `content`，不区分粘贴、文件、网页；具体入口、MIME、origin、parser 各司其职。省略与显式 content 在请求归一后等价，其他值返回 422。SourceRef 的字段仍是 `source_type`，不是 source_kind。同一 Scope 的已保存证据与已发布资料随 Scope 可读，私有操作只向创建者及显式授权管理员开放。 -固定的 source_kind 作为 schema 字段进入 bundle_digest，并间接进入 snapshot_digest；它不决定解析路由、artifact_id、来源绑定或缓存分区。显式更新同一 Artifact 时仍校验 expected_head_revision，并按完整快照语义决定 reused 或新增 Revision；即使从文件更新为粘贴文本，各版本的 source_kind 都为 content，具体出处、格式与解析信息分别随版本保留。 ### ImportedContent Revision 的 content @@ -392,10 +375,12 @@ ImportedContent Revision、冻结 bundle 和读取响应必须包含 `source_kin | `coverage`、`warnings`、`omissions`、`accepted_partial` | 声明范围内的完整性与显式接受决定；已知遗漏长期保留 | | `container_chain` | 逐层外部文件摘要、成员路径/序号及出处；与跨快照稳定成员身份分开 | -正文仅由 Source 长期保存，不在 content 中重复保存全文。Source metadata 保存 ImportedContent artifact_id、revision、part ordinal、source_kind、受控来源与定位；其中保存后分配的 artifact_id/revision/SourceRef 不参与内容摘要，source_kind、来源及定位已在冻结 bundle 中参与摘要。普通导入生成普通 captured ContentSource，显式保证 `working_note=False`、`internal=None`、`handoff_receipt=False`；不能设置 lineage_only、handoff receipt、working-note metadata 或其他系统保留角色。`source_kind=content` 只约束类型，不等于生成准入已满足；保存前同时通过既有 Source 类型及角色校验，拒绝客户端借 metadata 伪装内部 Source，不静默接受后再改变其语义。 +正文仅由 Source 长期保存，不在 content 中重复保存全文。Source metadata 保存 ImportedContent artifact_id、commit_id、part ordinal、source_kind、受控来源与定位；其中保存后分配的 artifact_id/commit_id/SourceRef 不参与内容摘要，source_kind、来源及定位已在冻结 bundle 中参与摘要。普通导入生成普通 captured ContentSource，显式保证 `working_note=False`、`internal=None`、`handoff_receipt=False`；不能设置 lineage_only、handoff receipt、working-note metadata 或其他系统保留角色。`source_kind=content` 只约束类型，不等于生成准入已满足；保存前同时通过既有 Source 类型及角色校验，拒绝客户端借 metadata 伪装内部 Source，不静默接受后再改变其语义。 namespace 最长 128 字符、key 最长 1024 字符，均非空且无首尾空白、控制字符。文件 key 为相对 POSIX 路径，拒绝绝对路径、反斜线、`.`、`..` 和空片段，不做大小写折叠。来源不保存凭据、签名 URL 或本机绝对路径。origin 为必填对象，namespace/key 由调用方提供；CLI/页面可根据输入生成并展示出处标签，但不按文件名匹配既有 Artifact；粘贴接口中的 key 是出处标签，不是稳定来源绑定。 +Source 保存时尚无最终 Artifact Revision，禁止预填 revision 或发布后改写不可变 Source。content 另外保存 commit_id，发布成功后由提交状态记录关联精确 ArtifactRef。 + ### 如何判断同一份文档 | 输入方式 | 稳定身份规则 | @@ -413,54 +398,6 @@ source_authority_id 是持久配置身份，指向固定存储根/桶和连接 同一 Scope 内稳定来源绑定使用唯一约束，防止并发首次采集创建两个 Artifact。显式手动新建不受 filename 或 digest 唯一约束。同一内容可以属于两份不同文档；内容缓存去重不能改变文档身份、权限或来源。 -### 复用现有表、注册与读取能力 - -ImportedContent 的唯一持久资料身份使用 `(scope_id, family=importedcontent, artifact_id)`。复用 `pc_artifacts` 保存 content 与不可变 Revision、`pc_artifact_heads` 保存已接受 Head、`pc_artifact_lineage_sources` 保存有序 SourceRef，正文使用 `pc_sources`。确有上游 Artifact 输入时才使用 `pc_artifact_lineage_artifacts` 的精确版本引用。不新增 ImportedContent 主表、ImportedContent 版本表或独立 ImportRecord。 - -Family 注册覆盖类型、序列化、读取、授权与允许的操作，复用对应的既有 registry、授权 profile 和路由适配点；不能只增加一个 Family 枚举就认为所有通用接口已经适配，也不要求为本期另建全局插件框架。更新 OpenAPI family schema、服务端能力与 Dashboard 映射，明确以下边界： - -| 能力 | 本期行为 | -| --- | --- | -| list/get、精确 Revision、历史、版本比较 | 支持；统一以所在 Scope 的 `scope.read` 授权 | -| texts/operations/commit 专用写入 | 支持；要求 `scope.read` + `scope.contribute`，并执行会话创建者隔离和事务内校验 | -| 通用 Artifact POST/PUT 创建或替换 | 不支持；不能绕过确认提交 parts、lineage 或成功状态 | -| generic restore/rollback、merge/undo_merge | 不支持；不注册相应适配器、不加入允许列表，HTTP 与 Runtime 均拒绝，页面不展示动作 | -| 模型生成、Dream/Candidate 发布 | 不支持；不注册 ImportedContent 生成或候选处理器 | -| 独立 Artifact 分享与 owner 授权 | 不支持；已发布资料随 Scope 可见，创建者信息只用于审计 | -| Artifact 全文/语义搜索 | 本期不支持；不注册 searcher，不把 namespace 列表过滤称为正文检索 | - -ImportedContent 是 Scope 内共享资料。现有通用 Artifact 读路由对多数 Family 使用 `artifact.read`，写路由及部分 Family 又有独立策略；接入必须同时适配 HTTP 访问声明/解析、运行时授权 profile、标准列表/精确读取、Source 读取与页面动作判断，不能让列表可见而详情仍要求 Artifact owner，也不能照搬 Prompt 的管理员写入规则。会话创建者约束仅适用于未发布输入和操作恢复，发布后不成为永久 owner；移除 Scope 读权限后，创建者也不能依靠旧回执访问正文。治理动作保持既有独立授权，不由 `scope.contribute` 隐式获得。 - -通用恢复路径根据历史 content 构建新 Artifact lineage，不能保证复制本 Family 的有序 `lineage.sources`，因此不可直接启用。即使只需要恢复旧正文，也须未来另行定义专用动作：校验完整 parts 与精确 Source 可读性，明确复用或新建证据、版本 provenance、Head CAS 及下游 pending/dirty 行为；本期不将通用 PUT 或历史版本页面按钮当作恢复入口。 - -已提交资料列表、当前版本、历史与精确版本读取复用通用 Artifact API；正文按 parts 中的精确 SourceRef 使用已有 Source 阅读能力组合。来源筛选可增加可重建投影/索引；反向 lineage 查询只表示实际引用到的分段或 Artifact 版本，不推断消费了全文。 - -### 操作存储的最小职责 - -异步执行需要耐久状态，但它不构成第二套资料模型。以下表名是拟新增的存储契约，不声称仓库已有通用任务表可以直接复用： - -| 存储 | 必要职责与约束 | -| --- | --- | -| `pc_import_sessions` | 会话创建者、固定非空 scope_id 与期限；创建时校验目标已存在且可贡献，目标不可修改 | -| `pc_import_operations` | operation_id、session_id、family、操作种类、原始输入暂存引用/摘要、冻结配置、执行状态、租约/generation、阶段重试与截止时间；仅服务执行恢复 | -| `pc_import_operation_items` | `(operation_id, item_id)` 唯一；成员清单、冻结目标、选择/解析/确认状态、bundle 暂存引用与摘要、最终精确 ArtifactRef；分成员原子提交 | -| 来源绑定、请求回执与输入索引 | 稳定来源键到 Artifact 的唯一映射、幂等请求到接收/提交结果、可复用解析结果索引；不复制已保存正文或版本权威 | -| 配额账本与可重建查询投影 | 跨进程准入/释放、来源过滤和反查；不能以无锁内存计数替代共享账本 | - -操作/成员记录支持有界状态更新、租约查询和清理；原始文件与待确认正文位于受控 staging store，不塞入 Artifact content 或操作表大 JSON。确认后长期正文、provenance 和来源关系以 Source + ImportedContent Revision 为准，操作记录仅保留恢复所需回执并按 TTL 清理。 - -不提供独立 FileImportJob 的业务 CRUD、永久详情页或与 ImportedContent 重复的资料目录。前端可以恢复“尚未完成的导入操作”，但操作编号不是 ArtifactRef；操作句柄查询是异步协议的一部分，不授予底层状态的任意写入。少量操作表不能消除，但不按输入格式新增任务领域类型。 - -### 写入与可见性边界 - -ContentImportApplication 作为该 Family 的专用写入适配器，持有一个数据库事务并复用 ArtifactRepository 的 create/revise。现有通用制品创建路径生成的命令 Source 不能替代真实正文分段；该适配器必须保证 parts 与 lineage.sources 一一对应。所有入口共用同一提交服务，不能直接调用通用 PUT 绕过确认。 - -新资料在确认前不创建 Artifact/Head/正文 Source；可在操作账本预分配 artifact_id，但普通 Artifact list/get 不可见。已有资料更新期间 Head 始终指向已接受版本，解析失败、重试、取消、进度变化不会新增 Revision 或改变 Head。仅确认成功后原子追加 Revision 并切换 Head，心跳和阶段变化只更新操作记录。 - -`pc_artifact_heads.lifecycle_state` 的 active/deprecated/retired 继续表示制品治理状态，不能塞入 uploading/parsing/failed。预览按会话创建者鉴权；其他具有 scope.read 的普通用户只能读已发布内容。确认时在同一事务中锁定/校验当前 Scope 权限、目标 Head/治理状态和操作有效性，已 retired 或已合并到其他目标的制品按既有不可写规则拒绝；导入不自动复活、不自动跟随合并目标，也不提供恢复已退役制品的入口。 - -ImportSession 从创建起即固定已有 scope_id。会话与内部操作记录用于私有预览、恢复、租约和 TTL；这些可变状态不写成 Artifact Revision。会话目标、权限与幂等规则遵循第 7 节。 - ## 4. 规范化、定位与分段 纯文本严格按确定的编码解码，去除一个开头 BOM。保留换行、缩进及 Unicode 字符，不将非法字符替换成 U+FFFD，不做全局 NFKC 或代码格式化。拒绝空白正文、NUL、surrogate 和除 TAB/CR/LF 外的 C0 控制字符；原始控制字符密集的文件不按普通文本强行导入。 @@ -471,7 +408,7 @@ ImportSession 从创建起即固定已有 scope_id。会话与内部操作记录 分段按章节、页、幻灯片、表格行组和段落等自然边界进行，单段最多 256 KiB UTF-8；过长单一段落再按行和 Unicode 字符边界切分。分段算法固定版本，无重叠、不截断。按 ordinal 拼接分段正文必须精确恢复冻结的完整提取文本，text_digest 一致。用于展示的标题和重复表头放 metadata，不能偷偷插入每段正文导致重建不一致。 -每文档最多 8 MiB 提取文本、128 段，定位与 manifest 最多 2 MiB；超限拒绝该文档而非静默保留前半部分。每段内部 Source serialization 还须满足现有 4 MiB observation 上限。处理器单次只消费部分 Source 时继续遵守自己的进度，不把文档级原子保存误解为原子生成。 +每文档最多 8 MiB 提取文本、128 段，定位与 manifest 最多 2 MiB；超限拒绝该文档而非静默保留前半部分。每段内部 Source serialization 还须满足现有 4 MiB observation 上限。其他处理器继续遵守各自消费窗口；ImportedContent 专用 document_commit 模式必须校验完整文档，不把 T1 原子保存误解为 T2 同步完成。 ## 5. 大小、时间与解析限制 @@ -524,164 +461,134 @@ ImportSession 从创建起即固定已有 scope_id。会话与内部操作记录 Principal/Scope 配额不足返回 429，部署容量或依赖不足返回 503，均携带 Retry-After；单个输入硬上限仍为 413。已接受任务的等待原因通过查询返回，worker 持久化下次重试时间并带抖动退避，不能热循环重复上传。 -## 6. 内部执行、私有预览与耐久暂存 +所有选中文档达到 T1 且没有解析/预览待处理工作时释放解析任务槽，即使 operation 仍为 publishing；发布积压槽独立遵循第 6 节。 -复杂文件由 ImportedContent Family 的文件操作创建接口接受，返回可恢复的 operation_id；执行状态保存在第 3 节的内部操作账本，由独立解析 worker 处理。ImportedContent Revision 仅管理已接受快照。本文下称“任务”均指该操作的内部执行工作，不引入 FileImportJob 领域实体或另一类 Artifact。 +## 6. 解析执行、状态与保留 -可以复用经过验证的数据库事务、租约、fencing、背压与 worker 基础组件；不把现有 Artifact Processing Supervisor 的生成任务直接改作导入任务。现有生成调度不自然具备上传暂存、私有预览与人工确认语义；如抽取公共执行基础设施，必须保留本文状态、权限和恢复约束。解析成功只冻结 bundle，不直接触发 Source/Artifact 写入。 +原文件耐久写入受控 staging、校验摘要、登记配额与队列后才返回上传 202。多副本使用共享耐久存储，不能依赖 API 进程 /tmp。上传中断不执行半文件；孤立 blob 对账清理前仍占字节配额。独立 parser worker 使用固定 profile、输入摘要、选择版本、租约 generation 和截止时间；过期 worker 不得覆盖新结果。 -执行状态、成员解析状态和成员确认状态分别存储，不能共用一个 status 字段；“ready”不是 Artifact 的发布或治理状态。 +普通单文档自动选中。容器先完整发现有界成员清单，再原子发布发现完成和稳定 item_id，等待一次 select 冻结成员与各自目标。发现未完成不能选择或推进 checkpoint；附件默认不选。改变成员、解析选项或目标需新尝试。已发布清单不得被 retry 重新编号。 -| 层级 | 字段与取值 | -| --- | --- | -| 任务执行 | execution_state：uploading / queued / running / idle / failed / cancelled / expired；idle 表示当前无待执行解析 | -| 内部任务阶段 | internal_phase：upload / discovery / selection / parsing / confirmation / done；discovery_state：not_required / pending / running / complete / failed | -| 文档选择 | selected：是否纳入本次导入；容器发现完成后通过 select API 冻结选择 | -| 文档解析 | parse_state：pending / parsing / ready / needs_review / failed；ready 和 needs_review 都对应不可变 bundle | -| 文档保存 | commit_state：uncommitted / committed；提交事务进行中不公开半提交状态 | +解析状态为 pending/parsing/ready/needs_review/failed。ready 表示声明提取范围完整且必需语义满足；已知遗漏为 needs_review，只有显式 accept_partial 才能确认。空正文、加密、安全违规、超限和必需语义不满足不能靠 accept_partial 绕过。冻结 bundle_digest 后不能预览 A、保存重算 B。 -```mermaid -flowchart TD - U["任务：uploading"] --> Q["任务：queued"] - Q --> R["任务：running；发现文档清单"] - R --> I["任务：idle；等待选择或确认"] - I -->|冻结选择，有待解析文档| Q - R -->|暂时故障，最多自动重试三次| Q - R -->|任务级失败或自动重试耗尽| F["任务：failed"] - F -->|发现阶段可重试，原输入有效且配额重新准入| Q - Q -->|取消或到期| X["任务：cancelled / expired"] - R -->|取消或到期| X - I -->|取消或到期| X -``` +临时解析错误最多自动重试三次；永久错误不重试。显式解析 retry 仅处理尚未冻结且 retryable 的发现或文档，重新准入配额，不重置自动重试计数。解析和制品发布的重试互不混用：T1 后只能恢复同一个 commit 的生成，不能重新解析覆盖证据。 ```mermaid -flowchart LR - P["文档：pending"] --> R["文档：parsing"] - R --> A["文档：ready"] - R --> B["文档：needs_review"] - R --> F["文档：failed"] - F -->|可重试、输入有效且配额准入| P - A -->|确认冻结 digest| C["保存：committed"] - B -->|确认 digest 并接受 partial| C +stateDiagram-v2 + [*] --> parsing + parsing --> ready + parsing --> needs_review + parsing --> parse_failed + parse_failed --> parsing: 显式重试且输入有效 + ready --> pending: 确认 T1 + needs_review --> pending: 接受 partial 并确认 T1 + pending --> running: Artifact Worker 领取 + running --> completed: T2 原子发布 + running --> failed: 可恢复生成失败 + running --> blocked: 插件或授权不可用 + running --> conflicted: Head 或治理状态冲突 + failed --> pending: 发布重试 + blocked --> pending: 条件恢复后显式重试 ``` -以上为内部耐久状态，不直接作为 HTTP 字段。公开 ImportOperation.status 由服务端派生，优先级为：已完成或已取消的终态保持；到期的未完成工作为 expired；存在执行工作为 running，否则存在已排队工作为 queued；完整发现后未选择为 awaiting_selection；选中非空且全部 committed 为 completed；存在未提交 ready/needs_review 为 awaiting_commit；任务级失败或仅剩失败成员为 failed。自动重试已安排的失败仍属于 queued。部分已提交只通过 progress 计数表达，不引入覆盖工作状态的 partially_committed。公开 phase 为 discovery/parsing/commit/done：发现及等待选择映射 discovery，正文处理映射 parsing，等待提交映射 commit，全部成功映射 done；失败/取消/过期保留最后阶段。内部 discovery_state=complete 映射公开 discovery.status=completed，parse_state/commit_state 分别映射 parse_status/commit_status。202 之前的 uploading 不公开为可恢复已接受操作。未选成员计入 excluded_items，取消/过期仍返回已提交计数与回执。 - -普通单文档输入自动固定 selected=true。归档、邮箱或带附件文件先有界发现完整成员清单，返回稳定 item_id、container_chain、类型及默认选择建议；附件默认不选，普通归档成员默认选。发现后暂停，调用方通过 select API 一次冻结明确的成员 ID 集合，再开始正文解析。超限或无法完成发现不能冒充完整清单；返回 discovery 错误。容器结构节点不可选择，未选附件记 excluded。选择不再改变；改变成员集合或解析选项须创建新尝试。临时 item_id 由本 operation 的持久成员映射产生，自动重试不重新分配。discovery_state=complete 与完整成员清单在同一事务中发布，只有此时才允许 selection。发现期间的中间清单不允许选择，不据其推进 checkpoint。临时故障令 discovery_state=failed 并返回阶段错误和 discovery_retryable；自动重试耗尽仍可通过显式 discovery retry 恢复。若完整清单尚未发布，重试可从原始暂存重新发现并复用内部成员身份；已发布清单不可重新发现、改写或重新分配 ID。损坏、加密、超限等永久错误标记不可重试。 - -原始字节耐久写入受控 staging store 并校验摘要后才返回 202 和排队任务。上传中断不留下可执行的半文件，上传占位及孤立 blob 由 GC 对账回收。单机可用受控状态目录，多副本使用共享耐久存储，不能依赖 API 进程的 /tmp。任务保存固定 profile、输入摘要、租约 owner/generation、attempt、deadline、选择版本及冻结 manifest。领取、结果发布、重试、取消和过期均以事务校验状态及 generation；过期 worker 不得覆盖新结果。 +会话/未确认 operation 最长 24 小时，重试不延长；到期或取消与确认事务串行化，先成功的确认保留，先到期的未确认输入返回 410，取消返回 409。取消仅封闭未确认输入，不停止已接受提交或删除 Source。解析终态、tombstone 和请求映射保留到 expires_at 后 30 天；相同请求重放返回原 HTTP 状态和响应。 -临时失败最多自动重试 3 次，永久格式/加密/损坏/超限错误不自动重试。任务内重试仅处理尚未冻结的文档，不覆盖 ready、needs_review 或 committed。显式 retry API 区分成员发现与文档解析两个阶段，只恢复尚未冻结且标记可重试的失败工作；两个阶段的自动重试计数分别记录，手动重试不重置计数，仍受截止时间、配额和次数记录约束。冻结 ImportedContentBundle 返回 bundle_digest；确认必须针对该值，不能预览 A 后保存重算 B。ready 只表示声明提取范围内完整且必需语义满足，已知遗漏或无法证明声明范围完整时为 needs_review；必需语义不足且 profile 不允许对应部分结果则 failed；accept_partial=true 的接受决定永久保存。 +已确认提交的恢复不依赖 session、item 或 staging：T1 持久化生成所需清单和 provenance，正文来自 Source。所有选中文档接受后、取消或到期后，原始 blob/缓存最迟一小时内清理；未完成容器保留到期限。pending/running/blocked/failed 提交和必要来源映射不得随 TTL 清理；conflicted 记录保留核实结果及证据，不自动撤回 Sources。轻量完成映射与精确 Artifact 版本长期保留。原文件不是长期档案。 -任务 TTL 从上传占位创建起最长 24 小时，且不晚于 session.expires_at；重试和选择不延长。到期或取消先封闭未提交文档写入、撤销租约，再清理临时数据；与正在确认的事务串行化：先成功的确认保留回执，先取消/到期则未提交确认失败。取消不删除已保存资料。任务状态及幂等映射保留至 expires_at 后 30 天：查询返回 200 的 tombstone（状态、时间与耐久回执引用）；过期后的未提交文档预览/确认返回 410，取消后返回 409；同上传键重放返回同一任务或 tombstone，不会隐式新建。超过 idempotency_expires_at 仅承诺通过永久 ImportedContent Revision 查询，客户端必须使用新尝试键。 +解析槽与发布槽独立：已有解析配额遵循第 5 节；T1 额外预留持久发布积压额度（每 Principal/Scope/部署默认 20/100/500）。pending/running/blocked/failed 均占用积压额度直到 completed/conflicted，防止插件停用期间无界接受。失败不释放持久发布额度；重试不重复计费，Worker 运行并发由 Artifact 框架控制。所有限额通过能力接口公布。 -全部选中成员保存完成后的原始文件与解析缓存最迟 1 小时内清理；取消/到期数据也在 1 小时清理窗口内回收，期间继续计入配额。未全部处理的容器保留到 expires_at。已提交预览从 Source 读取，ImportedContent Revision 的回执、定位和遗漏长期保留。staging blob 不公开为任意读取地址，也不承诺原始二进制长期下载。 -## 7. 身份、幂等与原子保存 +## 7. 身份、幂等与两阶段发布 -### 会话与 Scope +### 会话、摘要与请求身份 -ImportSession 创建必须在路径提供已有 scope_id，body 只能为 {}；服务端校验 Scope 存在且调用方具备 scope.read + scope.contribute 后，保存创建者和不可变 scope_id。该字段非空，不接受延迟解析的目标或创建参数。创建会话本身不创建 Scope、Artifact 或 Source，也不授予新权限。 +ImportSession 创建要求已有 Scope、scope.read + scope.contribute，body 仅 `{}`。会话、operation、item、commit、Source 与 Artifact 固定同一 Scope；不接受 body 覆盖路径身份或跨 Scope 引用。每次读写重新授权，未确认输入还校验创建者；Source/Artifact 的已保存内容按 Scope 授权。 -会话、operation、item、暂存配额及最终 Artifact/Source 始终归属同一 Scope。子操作同时校验路径 scope_id 与 session_id 的固定目标一致，不允许请求正文覆盖 scope_id；其他 Scope 的 artifact_id、SourceRef 或来源绑定不能作为当前目标。切换目标需重新创建会话并重新提供输入；已有操作不转移。 +Idempotency-Key 必填，1–128 个 ASCII 字母、数字或 `-_.`。创建会话键空间为 Principal+动作+key，请求指纹含 scope_id；其他 session 操作为 Principal+session_id+动作+key，publication retry 为 Principal+scope_id+commit_id+动作+key。指纹包含完整路径、规范化 body、实际文件摘要、冻结目标及接受语义；相同 raw 请求沿用第一次默认值/profile。同键同请求返回原状态码和响应，不同请求 409；每次重放重新鉴权。429/503 未接收不形成成功回执。 -创建会话、提交文本、接收上传、选择成员、重试与确认均重新校验该 Scope 的当前权限；操作查询/预览要求会话创建者及当前 Scope 访问权限，管理代操作须显式授权并审计。确认事务还须验证 Scope 仍存在且可写、会话/操作未过期或取消，以及目标 Head/治理状态；撤权或目标失效后不保存资料，不自动补建 Scope 或选择其他目标。取消与清理按第 6、10 节执行。 +artifact_id 是逻辑身份，commit_id 是接受快照，operation/item 是输入尝试，ArtifactRef 是已发布精确版本。bundle_digest 对 RFC 8785 canonical JSON 的 schema_version、source_kind、origin/container_chain、title、输入摘要/MIME/长度、parser/profile/编码/OCR、规范化/分段版本、全文摘要、content_manifest、coverage 与结构化遗漏计算 SHA-256。排除运行诊断、展示消息、时间、Principal、操作 ID、commit_id、artifact_id、Revision、SourceRef 与 journal position。snapshot_digest 对 `{schema_version,bundle_digest,accepted_partial}` 计算；不同文档和 A→B→A 历史允许重复摘要，禁止全局唯一摘要替代资料身份。 -会话上传有效期为创建起 24 小时；新 operation 及子操作不延长它，operation.expires_at 不晚于会话期限。会话与操作回执至少保留至最后一个 operation.expires_at 后 30 天，便于结果未知时恢复。会话到期禁止新上传与未提交保存，保留已提交回执；不删除目标 Scope、Artifact 或 Source。 +缓存只能复用解析结果，不能继承确认、权限或 Head。相同创建者、目标/预期 Head、原始输入、origin/title、profile/options/选择才可合并有效未冻结尝试；reparse=true 强制新执行。partial 缓存仍需本次显式接受，手动新建永远不按摘要跨文档归并。 -### 内容、身份与请求分别建模 +### T1：确认与证据保存 -Idempotency-Key 必填，长度 1–128，允许 ASCII 字母、数字及 -_.。创建会话映射键为稳定 Principal + operation + key，规范化后的 scope_id 必须计入创建请求指纹；同键改换 Scope 返回 409。其余导入写操作为 Principal + session_id + operation + key，session_id 唯一确定不可变目标 Scope。相同键、相同请求重放原结果，不同请求返回 409；每次重放重新授权。重放沿用首次冻结的默认值/profile。文件指纹包含实际输入摘要、规范化 metadata 与目标文档意图；所有指纹包含动作名、路径/请求中的 operation_id 与 item_id（适用时），避免同键操作不同句柄被错误重放；提交指纹另外包含 bundle_digest、accept_partial 及冻结目标/预期 Head。 +统一确认服务拥有一个数据库事务： -区分四种身份：artifact_id 是逻辑文档；operation_id/item_id/attempt key 是一次尝试；bundle_digest 是冻结解析结果；ArtifactRef 是一次已接受版本。snapshot_digest 表示内容及接受语义，可在不同文档甚至同一文档的多个历史 Revision 中重复，不能加 `(scope_id, snapshot_digest)` 全局唯一约束。 +1. 按固定锁序验证当前授权、Scope、私有输入、期限、冻结摘要、治理状态、来源绑定和 expected_head_revision；保留文档级发布占位。同文档已有非终结提交返回 409 publication_pending，新来源绑定通过唯一约束分配同一 artifact_id。 +2. 校验发布适配器/Family/schema 已准入，预留积压配额；生成稳定 commit_id。若显式更新的预期 Head 匹配且快照等于当前已发布版本，同事务保存提交清单与 completed 结果映射、reused 回执及精确版本，释放本次占位/预留后返回 200，不新增 Source 或 Artifact；清单可以引用原版本的 Source。 +3. 对新快照，在同一 connection 保存全部普通 ContentSource、journal、既有 dirty/pending；复用既有保存单元，不循环调用各自开启事务的 HTTP/capture 接口。metadata 记录 commit_id、artifact_id、ordinal，不预填最终 Revision。 +4. 保存有序不可变提交清单、确认者及授权上下文、目标与预期 Head、解析 provenance、输入映射、发布意图、配额和幂等接收回执。任何一步失败全部回滚；不创建 Artifact/Head。 +5. 提交后返回 202 和 C 状态地址，再尽力唤醒框架。通知丢失由持久发布意图恢复。其他处理器此时可见全部已接受 Source，但不保证同一消费窗口处理全文。 -bundle_digest = SHA-256(RFC 8785 canonical JSON)，输入为带 schema_version 的不可变对象：source_kind、受控来源、container_chain、title、input_media_type/input_bytes/input_digest、parser/profile/encoding/OCR 与规范化/分段版本、text_media_type/text_bytes/text_digest、有序 content_manifest、coverage、结构化 warnings/omissions。manifest 仅含 ordinal、字节数、分段摘要、全文字符区间及定位，不引用临时 blob。诊断消息、耗时、时间戳、Principal、session/operation/run/item ID、artifact_id/revision、SourceRef、journal position 全部排除。snapshot_digest 对 `{schema_version, bundle_digest, accepted_partial}` 规范化后计算，仅 needs_review 可令 accepted_partial=true。保存事务附加 parts 回执，不改写冻结摘要。 - -### Revision、Head 与重复输入 - -| 情况 | 行为 | -| --- | --- | -| 首次手动新建 | 本次创建意图分配 artifact_id，保存 Revision 1；同名或同内容的另一新建意图仍是另一文档 | -| 更新指定文档，预期 Head 匹配且 snapshot_digest 等于当前 Head | 返回 reused 与当前精确 ArtifactRef，不新增 Source 或 Revision | -| 更新指定文档，预期 Head 匹配但快照不同 | 新增 Revision 与分段，原子推进 Head；旧证据不变 | -| 预期 Head 与实际不符 | 返回 409 revision_conflict；展示当前版本，由调用方重新确认，不静默覆盖，即使内容碰巧相同也不跳过冲突 | -| A → B → A | 创建 R1(A)、R2(B)、R3(A)；不能用全局摘要命中把 Head 偷换回 R1。本期 R3 创建自己的 Source 分段 | -| 已成功请求同键重放 | 返回第一次的精确回执，即使 Head 已推进；不新增版本，也不回退 Head | -| 同输入重新解析 | reparse=true 新建尝试；若当前 Head 与冻结语义相同则 reused，实际输出或影响输出的 parser/profile 变化则新 Revision | +### 完整文档输入适配（必须新增） -手动更新在创建 item 时冻结 artifact_id 和 expected_head_revision。持续采集在接收输入时从来源绑定读出目标和预期 Head；新来源冻结“预期不存在”，绑定唯一约束冲突返回可恢复冲突，不能把旧解析结果自动覆盖新版本。会话中明确的新建意图在重试时保持同一分配身份，跨独立新建请求不按文件名或摘要合并。 +`document_commit` 是本文提出的框架输入模式。调度以 commit_id 领取一份文档并绑定 generation、Scope、family/schema 与代码配置版本；不能同时走该 Family 的 journal cursor。受信适配器从持久清单逐段读取精确 SourceRef，核对 0 起连续 ordinal、段数、摘要、全文拼接摘要、归属及普通角色；缺失任何分段即阻塞，不发布部分文档。 -### 解析复用与部分提取 +插件获得有界的文档清单及受控 `read_part(ordinal)` 能力（拟新增 API），每次最多 256 KiB，整个文档最多 8 MiB，清单/定位最多 2 MiB、128 段。正文可流式校验；插件输出不复制全文，document_commit 模式的 draft JSON 总预算 4 MiB。这些是独立模式预算，不放宽当前窗口 1 MiB 限额。部署必须通过最坏样本的输入/输出及内存验收，未实现扩展时 publication 不得 ready。 -任务跨键复用仅允许同创建者、同 session、同明确目标文档与预期 Head、相同原始输入/origin/title、固定 profile/选项和冻结选择。独立手动新建意图不跨键合并。任务须未到期、暂存可用、execution_state 为 queued/running/idle、有未完成工作且没有已提交 partial；failed/cancelled/expired/completed 不参与。合并检查与请求映射写入以事务/锁串行化。同上传键始终恢复原任务或 tombstone；reparse=true 禁止合并。不同 Principal 不返回对方私有任务或预览。 +适配器在服务端登记本次完整证据集合，draft.source_refs 必须与清单按顺序完全相等；不接受客户端任意 SourceRef、metadata 伪造的 commit_id 或越 Scope 读。只处理 T1 服务端提交记录，不把普通 content、Working Note 或系统命令包装为资料。官方插件仅校验、组装确定性 ImportedContent 内容，且返回这一 commit 的一个 draft；禁止网络获取和重跑解析。 -耐久输入映射以明确逻辑文档目标、输入摘要、受控来源/title、profile 和完整选项为键；容器包含稳定成员映射与选择。它索引已冻结解析结果及精确历史 ArtifactRef，而不是授予覆盖 Head 的权限。缓存命中可以省略 OCR，但必须按本次目标与预期 Head 决定 reused 或创建新 Revision；不能直接把历史版本当当前版本。手动新建仍创建新文档，跨 Scope 不暴露命中。 +### T2：插件生成与事务发布 -缓存 coverage=complete 可以复用解析结果，但交互文件仍需本次确认；明确配置 accept-complete 的 worker 才可自动确认。缓存 coverage=partial 时，本次 item 必须为 needs_review/uncommitted，展示 reusable_artifact、coverage 与遗漏，不计入 completed；本次显式 accept_partial=true 后才可保存。不能继承其他 Principal 或上次尝试的部分提取接受决定。同一次已成功提交的同键重放保留原决定。 +复用 Artifact Worker、ArtifactDraft、Repository、索引准备、租约/fencing 与发布事务。新增受信完成适配器必须与发布共用 connection；不能在插件 Python 回调里另开事务写完成状态。 -reparse=true 必须重新提供原文件并记录实际执行；运行诊断属于临时任务历史，Revision 保存影响内容的 provenance。解析缓存可以指向历史版本，但缓存指针不改变 Head；最近完整解析结果可更新缓存索引，旧版本继续精确读取。 +事务内重新验证提交 generation、插件代次/启用状态、文档占位、当前 Scope/治理状态、提交者 scope.read+scope.contribute 及 Worker 后台写权限。ImportedContent 采用明确的 Scope 共享 profile，不能直接继承通用插件“Worker owner”规则。撤权后标为 blocked，不将 Worker 权限视作用户授权替代；恢复相同主体权限后显式 retry，管理员代操作另行授权与审计。 -### 原子保存边界 +新文档 expected_revision=0；更新必须读取精确基准 Revision 并由框架校验 expected_revision。发布事务写 Revision、Head CAS、有序 lineage、已准备的声明式索引、commit_id→ArtifactRef、publication=completed、完成回执并释放占位/积压额度。任何失败全部回滚，T1 证据不回滚。提交成功但响应丢失时，以完成记录恢复，不能再创建 Revision。 -一份文档的事务包含：锁定会话、操作/成员及必要的来源绑定/Head、当前授权与状态校验、幂等校验、固定 Scope 的配额结算、全部 ContentSource 与 journal position、处理器 durable pending/dirty、ImportedContent Revision、Head CAS、精确有序 lineage、输入映射、item 提交结果及请求回执。reused 分支不新增分段，但仍原子记录本次接受及回执。任何故障全部回滚。 +同文档第二个确认在前一个提交终结前返回 publication_pending。外部治理或写入导致 Head/目标状态变化时，发布标为 conflicted，释放占位但保留证据，不自动读最新 Head 重放覆盖；调用者检查新 Head 后用新请求重新确认。failed/blocked 保留占位，通过同一 commit 重试；相同快照重试至多一个 Revision。A→B→A 依次完成生成 R1/R2/R3，不回退到 R1。已完成请求重放不修改 Head。 -Source metadata 中的目标 Revision 在事务内确定，但不进入摘要；验证 parts 顺序、manifest 与 lineage 一致。不可先逐段调用独立 Source HTTP API 再声称整文档原子性。SQLite、embedded seekdb 与 OceanBase 均采用数据库事务和唯一约束/CAS，不能仅依赖进程锁。本期不自动清理持久文档版本及其证据。 +### 读取、索引与保留 -现有 record capture 会自行开启事务；导入不能循环调用它，再单独保存 Artifact。实现应抽取或复用接受同一 connection 的内部保存单元，由 ContentImportApplication 唯一持有外层事务，保持以下顺序与职责： +正式列表/详情/精确历史复用 Artifact API；通用 POST/PUT、回滚、合并不能绕过文档提交，首期不开放。Source 可见性从 T1 开始，Artifact 可见性从 T2 开始。插件停用/卸载不影响已发布读取；已有未完成提交持久等待恢复，新的确认被就绪门禁拒绝。 -1. 在事务内验证授权、会话/item 状态、请求回执、目标治理状态、来源绑定和 expected_head_revision；按全局一致的锁顺序获取提交所需锁，确定将写入的 Revision。 -2. 校验冻结 bundle、manifest 及全部普通 ContentSource；逐段调用 `SourceRepository.add_with_status(connection, ...)`，取得精确 SourceRef 与 journal position。该 Repository 已在新增 Source 的同一事务内执行处理意图 `mark_dirty`，导入沿用此唯一职责，不另外重复维护一套 dirty 状态。 -3. 对 `created=True` 的分段，沿用现有 capture 的配置规则，为配置的 Source processing bindings 登记 durable pending；对已存在且身份/内容相同的 Source 不重复触发。共享保存单元必须保留 record capture 的既有行为，不能只调用 Repository 后遗漏 pending，也不能给 ImportedContent 本身注册生成消费者。 -4. 以实际写入结果构造有序 parts 和 `lineage.sources`，调用接受该 connection 的 `ArtifactRepository.create/revise`；在同一事务内完成 Head CAS、配额、来源映射、item 结果和幂等回执。任一分段、pending、Artifact 或回执写入失败，全部回滚。 -5. 提交成功后才发送唤醒通知、返回已保存结果；通知丢失仍由已提交的 pending/dirty 恢复。`reused` 和请求重放不再新增 Source、推进 Head 或重复登记消费。 +可在首次注册时声明标题/出处等 Artifact content 字段的文本索引，查询使用通用 search。正文仍仅在 Source，框架不会自动跟随 parts 建立全文索引；首期不承诺全文或向量检索。无声明则返回框架“不支持”，首次索引未就绪可返回 503，不能当作资料不存在。检索 schema/索引声明遵循 #2246 的冻结限制，不能把后续补全文索引描述成任意热更新。 -锁与 CAS 失败按第 7 节的冲突和重放语义处理，不以第二个事务补写前一事务遗漏的正文或待处理状态。后台生成只能观察到整份文档已提交的分段集合；其具体消费窗口仍由各处理器决定，不承诺一轮生成消费全文。 ## 8. HTTP、客户端与页面契约 ### 8.1 规范、身份与接口清单 -遵循 [RFC 1437](1437_source_artifact_rest_api.md) 的 Scope 资源身份、命名、错误结构、分页与标准 Artifact 读取契约。集合使用小写复数名词，静态多单词路径使用 kebab-case，JSON/query 与 operationId 使用 snake_case，URI 不带末尾斜线。`select`、`commit`、`retry`、`cancel` 是显式 POST 领域动作，不提供任意 PATCH/PUT 状态修改。所有下级路径校验 Scope、session、operation、item 的真实所属关系，不能仅凭最后一个 ID 查询。路径确定的身份不得在 body 重复传入。 - -本节描述待实施的公开契约；实现时以 `openapi/powercontext.yaml` 为唯一契约源，生成 Python/JavaScript 客户端并执行 contract-test。不得手工修改生成代码。Family 固定为 `importedcontent`，会话只接受已有 Scope，不创建 Scope。接口层级不引入独立 FileImportJob 或 ImportedContent 主表。 - -以下缩写只用于阅读，实际 HTTP 路径须完整展开，不能发送字面量 P/S/O/I： +遵循 [RFC 1437](1437_source_artifact_rest_api.md)：Scope 定位资源，集合使用小写复数、静态多词路径使用 kebab-case，JSON/query/operationId 使用 snake_case，不带末尾斜线。领域动作使用 POST，不提供任意状态 PUT/PATCH。路径身份不得重复出现在 body；每级校验真实父子归属。以下都是拟新增导入契约，不能宣称 #2210/#2246 已提供。 ```text P = /v1/scopes/{scope_id} S = P/import-sessions/{session_id} O = S/operations/{operation_id} I = O/items/{item_id} +C = P/import-commits/{commit_id} ``` | operationId | 方法与路径 | 成功状态与含义 | | --- | --- | --- | -| `get_import_capabilities` | `GET P/import-capabilities` | 200；当前调用者可见的格式、profile、权限与实际限额 | -| `create_import_session` | `POST P/import-sessions` | 201；创建固定 Scope、固定创建者的会话 | -| `list_import_sessions` | `GET P/import-sessions` | 200；当前 Scope 内创建者自己的恢复列表 | -| `get_import_session` | `GET S` | 200；会话及期限，不返回原文 | -| `import_content_text` | `POST S/texts` | 201 新建资料；200 更新或复用；请求本身即确认 | -| `create_import_operation` | `POST S/operations` | 202；单文件耐久接收并排队，尚未发布 Artifact | -| `list_import_operations` | `GET S/operations` | 200；操作恢复列表，不是资料列表 | -| `get_import_operation` | `GET O` | 200；状态、阶段、计数、期限及动作 | -| `list_import_items` | `GET O/items` | 200；成员清单，单文档也有一个 item | -| `get_import_item` | `GET I` | 200；冻结目标、提取结果、覆盖与问题 | -| `get_import_part` | `GET I/parts/{ordinal}` | 200；有界正文预览，已提交时读取 Source | -| `select_import_items` | `POST O/select` | 202；原子冻结容器成员选择并排队 | -| `commit_import_item` | `POST I/commit` | 201 新建资料；200 更新或复用；单文档原子提交 | -| `retry_import_operation` | `POST O/retry` | 202；重新准入后重试发现或指定失败文档 | -| `cancel_import_operation` | `POST O/cancel` | 200；取消未提交部分或返回已完成结果 | - -正式资料复用以下既有接口及 operationId；不另外建立“导入资料详情”资源。仅扩展 Family 注册、schema 和读取投影： +| `get_import_capabilities` | `GET P/import-capabilities` | 200；格式、profile、权限、解析及发布就绪状态 | +| `create_import_session` | `POST P/import-sessions` | 201；已有 Scope 的固定创建者会话 | +| `list_import_sessions` | `GET P/import-sessions` | 200；创建者恢复列表 | +| `get_import_session` | `GET S` | 200；会话及期限 | +| `import_content_text` | `POST S/texts` | 202；T1 已保存、待发布；当前已发布版本复用为 200 | +| `create_import_operation` | `POST S/operations` | 202；单文件耐久接收，尚未确认 | +| `list_import_operations` | `GET S/operations` | 200；输入操作恢复列表 | +| `get_import_operation` | `GET O` | 200；发现、解析、确认与发布汇总 | +| `list_import_items` | `GET O/items` | 200；成员元数据 | +| `get_import_item` | `GET I` | 200；冻结目标、解析结果与提交链接 | +| `get_import_part` | `GET I/parts/{ordinal}` | 200；有界预览，T1 后读取 Source | +| `select_import_items` | `POST O/select` | 202；冻结成员与目标并排队解析 | +| `commit_import_item` | `POST I/commit` | 202；单文档 T1 成功，非 Artifact 创建成功；复用为 200 | +| `retry_import_operation` | `POST O/retry` | 202；只重试发现或未冻结解析 | +| `cancel_import_operation` | `POST O/cancel` | 200；仅封闭未确认输入 | +| `get_import_commit` | `GET C` | 200；T1 已保存提交的当前发布状态与精确结果 | +| `retry_import_publication` | `POST C/retry` | 202；重新准入同一提交发布，不重跑解析 | + +复用标准 Artifact 接口，不新增资料详情 API： | operationId | 方法与路径 | 行为 | | --- | --- | --- | -| `list_artifacts` | `GET P/artifacts/importedcontent` | 当前可见 Head；新增 namespace 精确过滤，不是正文搜索 | -| `get_artifact` | `GET P/artifacts/importedcontent/{artifact_id}` | 标准 ArtifactRevision、正文 parts 与 Head ETag；保留 If-None-Match/304 | -| `list_artifact_revisions` | `GET P/artifacts/importedcontent/{artifact_id}/revisions` | 已接受的历史版本 | -| `get_artifact_revision` | `GET P/artifacts/importedcontent/{artifact_id}/revisions/{revision}` | 精确不可变版本；按既有契约不使用 ETag | +| `list_artifacts` | `GET P/artifacts/importedcontent` | 当前可见已发布 Head | +| `get_artifact` | `GET P/artifacts/importedcontent/{artifact_id}` | 标准 Revision 和 Head ETag；If-None-Match/304 | +| `list_artifact_revisions` | `GET P/artifacts/importedcontent/{artifact_id}/revisions` | 历史版本 | +| `get_artifact_revision` | `GET P/artifacts/importedcontent/{artifact_id}/revisions/{revision}` | 精确不可变版本 | -通用 Artifact POST/PUT 不允许直接写入 importedcontent，返回 422 的不支持错误；不能绕过专用写服务校验 parts、lineage 和接受语义。其他 Family 的通用 Replace 仍使用 If-Match，缺失返回 428、不匹配返回 412。本 Family 导入 commit 是领域命令，使用冻结的 expected_head_revision，冲突返回 409；不把 Artifact ETag 当成 operation/item 的 HTTP 校验值。 +声明检索时复用 `POST P/artifacts/importedcontent/search`；可用模式由 Artifact 能力发现给出，不新增导入 search。通用 Artifact POST/PUT、恢复和合并不开放给本 Family，返回标准 422 不支持；其他 Family 的 ETag/304/412/428 不变。导入冻结 expected_head_revision 冲突为 409。插件管理复用其管理员接口，不赋给普通导入者。 ### 8.2 公共字段、headers 与请求校验 @@ -690,7 +597,7 @@ I = O/items/{item_id} | `scope_id` | 沿用现有 Scope ID：1–256 字符且含非空白字符；作为单个 URL segment 编码 | | `session_id`、`operation_id`、`item_id` | 服务端生成的不透明字符串，1–128 个可见 ASCII 字符；客户端不得解析前缀或假定 UUID | | `artifact_id` | 沿用 Artifact ID 约束：1–128 个可见 ASCII 字符；必须属于路径 Scope 与 importedcontent Family | -| `revision` / `expected_head_revision` | 整数，最小 1 | +| `revision` / `expected_head_revision` | 已发布版本和更新预期值为正整数；接收回执的 revision 在未发布时为 null | | `ordinal` | 整数，从 0 开始；第 n 段的 ordinal=n-1 | | 时间 | RFC 3339 UTC，例如 `2026-10-19T08:00:00Z` | | 摘要 | `^sha256:[0-9a-f]{64}$`；input、bundle、snapshot、text 摘要含义独立 | @@ -700,7 +607,7 @@ I = O/items/{item_id} 启用认证时携带 `Authorization: Bearer `。全部导入写请求必须有 `Idempotency-Key`（1–128 字符，ASCII 字母、数字及 `-_.`）；缺失或非法 header 返回 400。所有成功/失败响应返回 `X-PowerContext-Request-ID`。JSON 使用 UTF-8；不使用额外 code/message/data 成功信封。未知 body 字段返回 422，未知 query、非法路径/header/cursor 返回 400；重复 multipart part 或额外 part 返回 422。服务端拒绝非空白正文中的 NUL、非法 Unicode 和除 TAB/CR/LF 外的 C0 控制字符。 -创建会话的 Location 指向 S；创建操作的 Location 指向 O；文本/commit 成功的 Location 指向精确 Artifact Revision。导入回执不是通用 Head 表示，不伪造其 ETag。返回 429/503 时包含秒数形式的 Retry-After。上传和动作的 202 返回接收时的 ImportOperation 快照，重放不替换成最新状态，调用者通过 GET O 查询变化。 +创建会话的 Location 指向 S；创建操作的 Location 指向 O；文本/commit 成功的 Location 指向提交状态 C，精确 ArtifactRef 在 C 发布完成后返回。导入回执不是通用 Head 表示，不伪造其 ETag。返回 429/503 时包含秒数形式的 Retry-After。上传、选择和解析重试的 202 返回接收时的 ImportOperation 快照；文本/commit 返回 ImportAcceptanceReceipt，发布重试返回 ImportCommit 接收快照，重放不替换成最新状态，调用者通过 GET O 查询变化。 ### 8.3 当前部署能力与会话 @@ -715,6 +622,7 @@ I = O/items/{item_id} | `limits` | 实际限额，包含 max_text_bytes、max_json_bytes、max_file_bytes、max_metadata_bytes、max_document_text_bytes、max_parts_per_document、max_part_bytes、max_selected_items、session_ttl_seconds、operation_ttl_seconds；还包含第 5 节按格式及累计配额的限制 | | `formats[]` | format_id、extensions、media_types、level(A/B/C)、status(available/unavailable/planned)、default_profile_id（可空）、profile_ids、extraction_scope、limits、reason（可空） | | `profiles[]` | profile_id、profile_digest、status、defaults、options_schema、extraction_scope、dependencies；options_schema 为有界的 JSON Schema 2020-12 子集，additionalProperties=false，不接受执行代码或任意 parser 类名 | +| `publication` | `{mode:"document_commit",status,reason}`；适配器、插件 schema/权限及文档预算都就绪才 ready；否则禁止新确认，已有提交持久保留 | | `quotas` | 当前调用者允许查看的配额上限和剩余量；不泄漏其他 Principal 的使用情况 | extensions 含点，例如 `.pdf`；media_types 使用规范 MIME；reason 为 `{code,message}`。profile 摘要覆盖固定配置和依赖版本，不是发布时钟。能力响应是当前快照，每个写动作仍重新校验。初始 limits 来自第 5 节，例如 max_file_bytes=52428800、max_text_bytes=262144、max_document_text_bytes=8388608、max_parts_per_document=128、max_part_bytes=262144、max_metadata_bytes=65536；客户端不得硬编码为部署实际值。 @@ -785,7 +693,7 @@ Idempotency-Key: text-notes-001 | media_type | 默认 text/plain | 只接受 text/plain、text/markdown，描述保存的正文表示 | | text | 必填 | 非空白字符串；UTF-8 字节数不超过 max_text_bytes，整个 JSON 不超过 max_json_bytes | -该请求本身即确认，不创建必需的异步解析操作；成功返回第 8.10 节 ImportCommitReceipt。粘贴 HTML/代码按所选正文表示保存原文，不执行或隐式提取；需要格式解析时使用文件入口。此接口不接受 parser、URL 或服务器文件路径。 +该请求本身即确认，不要求文件解析 operation；证据事务成功返回 202 ImportAcceptanceReceipt，复用当前已发布版本返回 200。随后通过 GET C 查询制品发布；不会同步保证 Artifact 已生成。粘贴 HTML/代码按所选正文表示保存原文，不执行或隐式提取；需要格式解析时使用文件入口。此接口不接受 parser、URL 或服务器文件路径。 ### 8.6 文件上传与解析参数 @@ -853,7 +761,7 @@ curl --request POST \ "phase": "parsing", "discovery": {"status": "not_required", "retryable": false, "error": null}, "selection_version": 1, - "progress": {"discovered_items": 1, "selected_items": 1, "parsed_items": 0, "failed_items": 0, "committed_items": 0, "excluded_items": 0}, + "progress": {"discovered_items": 1, "selected_items": 1, "parsed_items": 0, "failed_items": 0, "committed_items": 0, "published_items": 0, "excluded_items": 0}, "available_actions": ["cancel"], "created_at": "2026-10-19T08:05:00Z", "expires_at": "2026-10-20T08:00:00Z", @@ -864,49 +772,47 @@ curl --request POST \ } ``` -### 8.7 操作查询、成员与预览 +### 8.7 操作、成员与预览 -ImportOperation（POST 接收快照、GET 及列表项使用同一字段定义）： +ImportOperation 字段用于 POST 接收快照、GET 和列表项： -| 字段 | 类型与含义 | +| 字段 | 类型、含义 | | --- | --- | -| scope_id、session_id、operation_id、family | 路径身份与固定 importedcontent | -| status | queued / running / awaiting_selection / awaiting_commit / completed / failed / cancelled / expired；由服务端按第 6 节派生 | -| phase | discovery / parsing / commit / done；失败保留失败阶段，成功为 done | -| discovery | `{status,retryable,error}`；status 为 not_required/pending/running/completed/failed；error 可空，结构为 ErrorDetail | -| selection_version | 非负整数；容器未冻结为 0，选择后为 1；单文档自动固定为 1 | -| progress | discovered_items（发现未完成为 null）、selected_items（未选择为 null）、parsed_items、failed_items、committed_items、excluded_items；后四项为非负整数，parsed_items 指已冻结结果，包含已提交项，不得把各计数相加当总数 | -| available_actions | 当前允许尝试的 select/retry/cancel 数组；仅提示，执行时重新校验 | -| created_at、expires_at、idempotency_expires_at | RFC 3339 UTC；期限不因重试延长 | -| blocked_reason、retry_after_seconds | 可空的 `{code,message}` 与非负整数；说明已接受任务的排队阻塞 | -| error | 可空 ErrorDetail；任务级错误，不把单个成员失败伪装成 HTTP 查询失败 | +| scope_id、session_id、operation_id、family | 路径身份与 importedcontent | +| status | queued/running/awaiting_selection/awaiting_commit/publishing/completed/failed/cancelled/expired | +| phase | discovery/parsing/commit/publication/done；失败保留出错阶段 | +| discovery | `{status,retryable,error}`；not_required/pending/running/completed/failed | +| selection_version | 非负整数；容器选择前 0、冻结后 1；单文档 1 | +| progress | discovered_items、selected_items 未知时 null；parsed_items、failed_items、committed_items、published_items、excluded_items 非负整数，计数重叠不能相加为总数 | +| available_actions | select/retry/cancel 提示数组，不授予权限 | +| created_at、expires_at、idempotency_expires_at | RFC 3339 UTC；只约束输入与请求映射，不使已接受发布过期 | +| blocked_reason、retry_after_seconds、error | 可空安全原因、非负秒数、ErrorDetail | -内部 execution_state、task_slot_held、租约和 fencing generation 不直接成为公开状态。部分已提交通过 progress 表达；仍有活动工作时 status 为 queued/running，有待确认结果时为 awaiting_commit，仅剩失败成员时为 failed。取消或过期仍返回已提交数量，成员详情保留提交回执；整个操作不是跨文档事务。 +优先级：cancelled 保留；尚有未确认输入且到期为 expired；否则有解析执行/队列分别为 running/queued，待选择为 awaiting_selection，待确认为 awaiting_commit；全部选中成员 T1 后，有待发布为 publishing，只有阻塞/失败/冲突为 failed，全部已发布或 reused 为 completed。发布与输入并存时按输入阶段显示，progress 和各提交状态仍展示发布进展。已接受提交不因 operation=expired/cancelled 停止，独立查询 C。committed_items 仅指 T1，published_items 才指制品完成。 -ImportItem（GET I；GET O/items 的列表项包含同样的有界元数据，不含正文或完整 content_manifest）： +ImportItem 字段： -| 字段 | 类型与含义 | +| 字段 | 类型、含义 | | --- | --- | | scope_id、session_id、operation_id、item_id | 完整父子身份 | -| title、origin、container_chain、input_media_type | 识别后的标题、出处、成员链与 MIME；尚未识别的 MIME 为 null | -| selectable、selection_suggested、selected | boolean；结构节点不可选，建议不等于已经选择 | -| artifact_target、resolved_target | 选定意图与解析后的冻结目标；选择前可空 | -| parse_status | pending / parsing / ready / needs_review / failed | -| commit_status | uncommitted / committed | -| retryable | boolean；仅表示当前解析错误是否可重试 | -| parser | 可空；profile_id、profile_digest、router_version、options 及实际 backends/conversion 信息；已冻结时完整回显 | -| statistics | page_count、sheet_count、slide_count、table_count、part_count、text_bytes；非负整数或 null，未知/不适用不能当 0 | -| ocr | requested_mode、used（boolean/null）、processed_page_count（integer/null）、engine（string/null）、languages；未执行完的统计不冒充最终值 | -| coverage | complete / partial / unknown；尚无结果为 null | -| warnings、omissions | `{code,message,locator}` 数组；locator 可空。结构化问题参与语义摘要，展示消息不参与；禁止无界诊断堆积，无法有界表达时失败而非隐藏遗漏 | -| input_digest、bundle_digest | SHA-256 或 null；未完成阶段不伪造摘要 | -| reusable_artifact | 可空的标准 `{family,artifact_id,revision}`，Scope 来自顶层；候选缓存命中，不表示本次已保存，也不是 Candidate | -| receipt | 可空 ImportCommitReceipt；只在本次成功确认后存在 | -| available_actions、error | commit 动作提示数组、可空 ErrorDetail | - -`GET I/parts/{ordinal}` 按稳定 ordinal 逐段读，不返回所有正文。响应含 scope_id、session_id、operation_id、item_id、ordinal、text、media_type、text_bytes、text_digest、bundle_digest、locators。段最大 256 KiB，返回完整段，不静默截断；单文档最多 128 段。超限处理遵循第 4、5 节。ordinal 不存在返回 404，尚未解析完成返回 409；未提交预览过期为 410、取消为 409；已提交内容在操作映射保留期间从 Source 读取，此后使用标准 Artifact/Source 接口。 - -定位使用 kind 判别对象，例如 `{kind:"pdf",page_number:3}`、`{kind:"spreadsheet",sheet_name:"预算",cell_range:"A1:D20"}`、`{kind:"slide",slide_number:5}`、`{kind:"text",line_start:10,line_end:25}`。页、幻灯片和行从 1 开始，行范围含两端；全文字符区间使用 Unicode code point 的 `[start,end)`。若有 PDF bbox，必须同时提供 page_width/page_height、unit=pt、origin=top_left，bbox=[left,top,right,bottom]，不能混用像素与点。无法可靠定位时明确说明，不猜坐标。 +| title、origin、container_chain、input_media_type | 有界元数据；未知 MIME 为 null | +| selectable、selection_suggested、selected | boolean；结构节点不可选 | +| artifact_target、resolved_target | 冻结目标；选择前可空 | +| parse_status、commit_status | pending/parsing/ready/needs_review/failed；uncommitted/committed，committed 仅指 T1 | +| commit_id、publication_status | T1 前为 null；之后关联 C，不能代替其权威状态 | +| retryable | 当前解析错误能否重试，不表示发布重试资格 | +| parser | profile_id/profile_digest/router_version/options/实际 backend 与转换信息，未识别为 null | +| statistics | page_count/sheet_count/slide_count/table_count/part_count/text_bytes，整数或 null；未知不等于 0 | +| ocr | requested_mode、used、processed_page_count、engine、languages；未确定项为 null | +| coverage、warnings、omissions | complete/partial/unknown 或 null；问题为 `{code,message,locator}` 有界数组 | +| input_digest、bundle_digest | SHA-256 或 null | +| reusable_artifact | 可空 `{family,artifact_id,revision}`；只表示解析缓存候选，不代表本次接受 | +| receipt | T1 后的不可变 ImportAcceptanceReceipt，之前 null | +| available_actions、error | 动作提示与安全 ErrorDetail | + +`GET I/parts/{ordinal}` 返回 scope_id/session_id/operation_id/item_id/ordinal/text/media_type/text_bytes/text_digest/bundle_digest/locators。ordinal 从 0 开始，每次完整读取最多 256 KiB，不截断；缺段 404、未完成解析 409、未确认预览过期 410、取消 409。T1 后映射有效期内从 Source 读取，之后经 C 的精确 SourceRef 或 Artifact 读取。 + +locator 为 kind 判别对象：pdf/page_number；spreadsheet/sheet_name/cell_range；slide/slide_number；text/line_start/line_end。页、幻灯片、行从 1 开始，行区间含两端，字符采用 Unicode code point `[start,end)`。PDF bbox 同时携带页尺寸、unit=pt、origin=top_left，数组为 left/top/right/bottom；无法定位时明确 unknown。 ### 8.8 容器选择、重试与取消 @@ -936,11 +842,11 @@ expected_selection_version 必填且只能为 0；items 必填、非空、去重 stage 必填。discovery 禁止 item_ids，仅允许发现失败且 retryable、完整清单未发布、原始暂存有效的操作。documents 的 item_ids 必填、非空、唯一、最多 1,000 项，必须全部是本操作已选、失败、可重试且尚未冻结的文档；一个不满足即整请求拒绝，不部分重试。不能重试 ready/needs_review/committed，不改变输入/profile/目标或期限。手动重试不重置每阶段最多三次的自动重试计数。只有重新准入配额、队列和幂等回执原子写入后返回 202；429/503 表示本次未接收，可用原 key 退避重试。已接收的后台资源等待通过 GET O 的 blocked_reason 表达。 -`POST O/cancel` 请求只能为 `{}`，原子封闭未提交写入并返回 200 ImportOperation。重复取消不重复执行，已完成返回 completed，已过期返回 expired tombstone，已发布 Artifact 不删除。select/retry 对 cancelled 返回 409，对 expired 返回 410。所有动作重新检查创建者、当前 Scope 权限和父子归属；管理员代操作遵循第 10 节。 +`POST O/cancel` 请求只能为 `{}`，原子封闭未提交写入并返回 200 ImportOperation。重复取消不重复执行，已完成返回 completed，已过期返回 expired tombstone，已保存 Source 和待发布提交继续保留并生成；取消仅影响未确认输入，不撤销已接受提交。select/retry 对 cancelled 返回 409，对 expired 返回 410。所有动作重新检查创建者、当前 Scope 权限和父子归属；管理员代操作遵循第 10 节。 -### 8.9 确认保存与并发 +### 8.9 确认、提交查询与发布重试 -`POST I/commit` 不再在 body 重复 operation_id/item_id。示例摘要仅演示格式，实际必须使用预览返回的值： +`POST I/commit` 只接受冻结摘要与接受决定： ```json { @@ -949,47 +855,90 @@ stage 必填。discovery 禁止 item_ids，仅允许发现失败且 retryable、 } ``` -bundle_digest 必填；accept_partial 为 boolean，默认 false。ready 只接受 false；needs_review 必须显式 true，否则 409 partial_extraction。目标、标题、origin、parser 和预期 Head 已冻结，commit 不接受修改。摘要不符为 409 bundle_conflict，目标 Head 不符为 409 revision_conflict；内容碰巧相同也不能跳过版本检查。已取消/未解析完成为 409，未提交结果已过期为 410。 +bundle_digest 必填；accept_partial 默认 false。ready 只接受 false，needs_review 必须 true；安全/格式/超限错误不可绕过。目标、title、origin、parser、预期 Head 均已冻结，不能在 commit 修改。摘要不符 409 bundle_conflict，Head 不符 409 revision_conflict，未终结发布占位 409 publication_pending；尚未完成/取消为 409，未确认输入到期 410。 -一个提交事务内写全部 Source、journal positions、dirty/pending、Revision/Head CAS/lineage、来源绑定、配额和耐久回执；事务后通知失败仍能恢复。已提交 item 在映射保留期内以相同接受参数再次提交返回原回执，即使 Head 已推进或操作后来到期；不同接受参数返回冲突。每次重放重新授权。 +T1 新快照返回 202 ImportAcceptanceReceipt 与 `Location: C`，当前已发布相同快照复用返回 200；不返回 201 Artifact 创建成功。同一请求重放永远返回第一次状态码及响应，即使已生成；通过 GET C 取得最新结果。C 独立于 session 生命周期，不在 body 重复 scope_id/commit_id。 + +`GET C` 需要当前 scope.read + 创建者身份，或显式管理员与 Scope 读权限；已公开 Artifact/Source 仍按 Scope 读取。返回 200 ImportCommit，生成失败不会变成 HTTP 500。`POST C/retry` 要求原确认者当前 scope.read+scope.contribute，或授权管理代操作；body 为： + +```json +{"expected_generation": 2} +``` -发生版本冲突时读取当前 Head、展示差异，以新 expected_head_revision、原文件和新 key 创建操作；可以复用匹配的解析缓存，但不能修改旧冻结结果或覆盖当前版本。该冲突协议仅适用于导入动作，不改变通用 Artifact PUT 的 428/412 和 GET 的 ETag 规则。 +expected_generation 必填、正整数，取最近 GET C；不匹配 409。仅 failed 且 retryable，或阻塞原因已消除的 blocked 可以重试；completed/conflicted/running/pending 为 409，不改变快照、目标或预期 Head。重试沿用 commit_id 和 Source，CAS 增加 generation、保留积压预留并登记执行意图，返回 202 ImportCommit 接收快照和 Location C；同 key 精确重放。503 表示插件或授权前置仍不可用，不伪造已接收。 -### 8.10 统一提交回执与标准引用 +### 8.10 接收回执与发布结果字段 -文本保存和文件 commit 返回 ImportCommitReceipt；身份平铺，lineage 复用标准 SourceTypeReference/ArtifactReference，不新增 Source/Artifact union 或可写引用： +文本/commit 的 ImportAcceptanceReceipt 示例（标识与摘要仅演示格式）： ```json { "scope_id": "project-docs", + "commit_id": "icm_001", "family": "importedcontent", "artifact_id": "ic_001", - "revision": 1, - "outcome": "created", + "revision": null, + "outcome": "accepted", "source_kind": "content", "accepted_partial": false, "snapshot_digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "accepted_at": "2026-10-27T08:00:00Z", + "status_url": "/v1/scopes/project-docs/import-commits/icm_001", "sources": [{"source_type": "content", "source_id": "cs_001"}], - "artifacts": [], "parts": [{"ordinal": 0, "source_type": "content", "source_id": "cs_001", "journal_position": 42}] } ``` -| 字段 | 类型与含义 | +| 字段 | 格式与含义 | +| --- | --- | +| scope_id、commit_id、family、artifact_id | 平铺身份；commit_id 为服务端不透明 1–128 可见 ASCII 字符，artifact_id 可能尚未发布 | +| revision、outcome | 新快照 null/accepted；当前版本复用时实际正整数/reused；回执不可变，不冒充标准 ArtifactRevision | +| source_kind、accepted_partial、snapshot_digest | 固定 content、本次接受决定与语义摘要 | +| accepted_at、status_url | T1 接受时间与同部署 C 的相对 URL | +| sources、parts | 有序标准 SourceTypeReference；parts 含 ordinal/journal_position，与冻结清单完全相同 | + +ImportCommit（GET C 及 retry 接收快照）包含上述身份、摘要、接受信息、sources/parts，并包含： + +| 字段 | 格式与含义 | | --- | --- | -| scope_id、family、artifact_id、revision | 与标准 Artifact 身份相同；revision 是本次实际接受的精确版本 | -| outcome | created：首次创建 Artifact，HTTP 201；updated：已有 Artifact 新 Revision，HTTP 200；reused：复用当前精确版本，HTTP 200 | -| source_kind、accepted_partial | 固定 content、此次接受语义；相同成功请求重放保持原值 | -| snapshot_digest | 冻结 bundle 与接受语义的摘要；不是 HTTP ETag，也不是标准 Artifact content_digest | -| sources | 标准 `{source_type,source_id}` 有序直接 lineage；全部为 content | -| artifacts | 标准 `{family,artifact_id,revision}` 直接 Artifact lineage；没有则 []；Scope 来自顶层 | -| parts | `{ordinal,source_type,source_id,journal_position}` 有序数组，正文在 Source；与 sources 顺序及 manifest 一一对应；粘贴为一段 | +| status | pending/running/blocked/failed/conflicted/completed；不是 Artifact lifecycle_state | +| generation | 正整数；发布重试与 worker fencing 依据，客户端不得自增后盲写 | +| revision、outcome | 未完成均 null；completed 为实际 Revision 与 created/updated/reused | +| completed_at、artifact_url | 未完成 null；完成为 UTC 时间与精确版本相对 URL，不能只返回 Head URL | +| retryable、available_actions | boolean、数组；只提示，调用时重新鉴权与 CAS | +| error、blocked_reason | 可空 `{code,message,details}`、`{code,message}`，无正文、凭据或私有路径 | + +```json +{ + "scope_id": "project-docs", + "commit_id": "icm_001", + "family": "importedcontent", + "artifact_id": "ic_001", + "source_kind": "content", + "accepted_partial": false, + "snapshot_digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "accepted_at": "2026-10-27T08:00:00Z", + "status_url": "/v1/scopes/project-docs/import-commits/icm_001", + "sources": [{"source_type": "content", "source_id": "cs_001"}], + "parts": [{"ordinal": 0, "source_type": "content", "source_id": "cs_001", "journal_position": 42}], + "status": "completed", + "generation": 1, + "revision": 1, + "outcome": "created", + "completed_at": "2026-10-27T08:00:02Z", + "artifact_url": "/v1/scopes/project-docs/artifacts/importedcontent/ic_001/revisions/1", + "retryable": false, + "available_actions": [], + "error": null, + "blocked_reason": null +} +``` -示例 digest 只表示语法，不能用于提交真实文件。回执不返回全文；精确内容由 Location 指向标准 Revision 再读取 Source。已有 Source 接口仍使用 source_type，不改成 source_kind 或 name；标准 ArtifactRevision 外形与 content_digest 计算规则不变。 +来源引用始终使用 source_type，ArtifactRef 始终是 family/artifact_id/revision，SourceRef 不是 ArtifactRef。已发布 Artifact 的标准外形、ETag 和 content_digest 不变。T1 回执不预测最终版本；用户读取当前 Head 后必须按本次 C 的精确结果判断导入完成。 ### 8.11 分页、错误、幂等与恢复 -新增列表 query 使用 limit（默认 50，1–100）与 cursor（不透明字符串）；会话列表已由路径限定 Scope，操作列表可按 status 过滤，成员列表可按 selected/parse_status/commit_status 过滤。响应为 `{items:[],next_cursor:null}`，末页 next_cursor=null，不返回虚假 total。cursor 绑定调用者、完整父路径、过滤、limit、高水位及 TTL；每页重新授权。高水位限制新增记录，不冻结可变状态，因此过滤结果不承诺状态快照。非法/不匹配 cursor 为 400，过期为 410。标准 Artifact 列表沿用现有结构，仅为 importedcontent 增加 namespace 精确过滤。 +新增列表 query 使用 limit（默认 50，1–100）与 cursor（不透明字符串）；会话列表已由路径限定 Scope，操作列表可按 status 过滤，成员列表可按 selected/parse_status/commit_status 过滤。响应为 `{items:[],next_cursor:null}`，末页 next_cursor=null，不返回虚假 total。cursor 绑定调用者、完整父路径、过滤、limit、高水位及 TTL；每页重新授权。高水位限制新增记录，不冻结可变状态，因此过滤结果不承诺状态快照。非法/不匹配 cursor 为 400，过期为 410。标准 Artifact 列表沿用现有结构，沿用插件框架已提供的过滤项；不承诺新增 namespace 列表过滤。 错误使用现有 envelope： @@ -1017,201 +966,107 @@ bundle_digest 必填；accept_partial 为 boolean，默认 false。ready 只接 202 之后发生的格式、提取、超限或解析错误写入 operation/item.error；成功查询仍为 200，不伪装成 HTTP 500。ErrorDetail 复用 `{code,message,details}`，details 无内容时为 null，不包含凭据或私有 blob 地址。412/428 保留给既有条件请求，本期导入 commit 不借用它们。 -导入的 Idempotency-Key 是此领域新增的耐久协议，不声称既有 Source/Artifact Create 已幂等。第 7 节定义键空间；补齐默认值后规范化请求，文件按实际字节摘要及 metadata 比较，不按 multipart boundary 比较。同 key 同请求返回原状态码和接收/提交回执，不同请求返回 409。相同 raw 请求重放沿用首次冻结默认值，不受部署 profile 更新影响。未接收的 429/503 不写成功回执；已提交重放即使 Head 改变也返回原版本，不产生新 Revision。期限、tombstone 与 GC 见第 6、7 节。 - -### 8.12 客户端、页面与调用顺序 +导入的 Idempotency-Key 是此领域新增的耐久协议，不声称既有 Source/Artifact Create 已幂等。第 7 节定义键空间；补齐默认值后规范化请求，文件按实际字节摘要及 metadata 比较，不按 multipart boundary 比较。同 key 同请求返回原状态码和接收/提交回执，不同请求返回 409。相同 raw 请求重放沿用首次冻结默认值，不受部署 profile 更新影响。未接收的 429/503 不写成功回执；已确认重放即使发布状态或 Head 改变也返回原接收快照；GET C 返回本次精确结果，不产生新 Revision。期限、tombstone 与 GC 见第 6、7 节。 -Python client 公开与表格一致的 operationId，并将 scope_id 显式传递到所有路由。CLI 提供 text、files、resume、cancel、dry-run；新导入必须先提供已有 --scope。恢复 --session 时从调用方保存的会话回执读取 Scope 或显式提供 --scope，二者存在时必须相同；没有本地目标信息必须要求 --scope，不增加绕过 Scope 的服务器查询。页面当前 Scope 内恢复会话，切换 Scope 不迁移旧任务。 +### 8.12 客户端、页面与使用举例 -CLI 的 --artifact-id 与 --expected-head-revision 明确更新，解析参数映射到 parser.options；容器先展示完整清单再 select，无人值守必须显式配置选择规则。--accept-complete 仅自动提交 ready，partial 还需单独明确授权，包括缓存命中的 partial。dry-run 只验证本地事实，未运行解析器不能宣称文件可完整导入。收到 429/503 按 Retry-After 退避，不显示为已接收。 +HTTP/OpenAPI 是契约源，生成 Python/JavaScript 调用；Dashboard 采用既有 Tabler/Jinja2/HTMX、中英文及主题。写入口默认关闭，显式启用并满足鉴权要求。CLI/SDK 使用同一服务，不自行串联逐段写入。MCP 的 source add 不能假装支持文件预览和提交；若暴露导入 MCP 工具，必须另行生成契约并覆盖同样的恢复测试。 -页面提交按钮发送冻结 bundle_digest 与原 Idempotency-Key；刷新/重新登录只恢复查询，不自动确认。结果不明时读取状态/receipt 或同键重放。上传进度由客户端计量；解析器未报告页进度时显示处理中；保存不公开半提交状态。 - -| 场景 | 调用顺序 | +| 场景 | 调用顺序与结果 | | --- | --- | -| 粘贴 Markdown | GET P/import-capabilities → POST P/import-sessions `{}` → POST S/texts → 标准 Revision/Source 阅读 | -| 扫描 PDF | 创建会话 → POST S/operations（parser.options.ocr_mode=auto）→ GET O/I/parts → POST I/commit → 精确 Revision | -| ZIP/邮箱 | 上传 → GET O 等完整发现 → GET O/items → POST O/select → 逐文档预览与 commit；各文档独立事务 | -| 更新已有资料 | artifact_target 指定 artifact_id/expected_head_revision → 解析预览 → commit → updated 200；旧 Head 在成功前始终可读 | -| HTML 文件 | 同文件流程，服务端识别 text/html；origin 中的网页说明不触发抓取 | -| 未来网页 Connector | 经独立验收的授权适配器提交冻结快照；本期没有 URL 导入接口 | - -### 已发布正文、历史与比较 - -复用现有 Dashboard 的历史分页、Revision 选择及比较页面；为 ImportedContent 增加读取投影，不另建一套版本系统。历史读取不依赖已经过期的 import operation 或 staging 文件，而是按精确 Revision 的 parts 批量读取对应 ContentSource，保持 ordinal 顺序并验证缺失/摘要异常；不能用当前 Head 正文填补旧版本缺段。 - -比较默认呈现正文差异，以及 origin、parser/profile、OCR、coverage、omissions、accepted_partial 等有意义的元信息变化；SourceRef、journal position、摘要和保存时间只作溯源详情，不能用其 JSON 变化代替正文变化。正文按分段延迟加载并设置比较字符数、输出与计算预算；超出预算显示“比较未完整展示”及分段入口，保留完整精确原文，不在后台截断后声称无差异。历史页面不显示恢复、回滚或合并按钮，直接调用相应 API 也不能成功。 - -本期不提供 ImportedContent 正文搜索，既有 `/v1/scopes/{scope_id}/artifacts/{family}/search` 对此 Family 返回统一的不支持结果；列表的 namespace 精确过滤仅查询元数据。后续若增加关键词或语义检索，应在现有 ArtifactSearchService 注册符合契约的 `ArtifactSearcher`，补齐 Scope 权限、版本选择和索引更新规则，不另建重叠搜索 API，也不因导入成功就承诺向量索引已就绪。 +| 粘贴 Markdown | 能力 → 创建会话 → POST S/texts 202 → GET C pending/completed → 精确 Artifact Revision；不调用 LLM | +| 80 页 PDF、40 个分段 | 上传 → 查看预览/OCR/遗漏 → POST I/commit 202 → T1 全部 40 段与清单一起保存 → 文档模式插件组装 → GET C completed；不依赖 20 条 journal 窗口凑齐 | +| 扫描 PDF 未启用 OCR | 显示 needs_review 或 ocr_required；无正文不得确认，有部分正文须用户明确接受；collect complete-only 不接受 | +| 更新已有资料 | 指定 artifact_id/expected_head_revision → 确认 202 → 旧 Head 可读 → T2 CAS 推进；冲突保留证据并显示 conflicted | +| ZIP/邮箱 | 完整发现 → 选择 → 各成员预览与确认 → 各自 C；独立事务，不因某成员失败撤回已保存资料 | +| 生成失败 | GET C failed → 修正依赖 → POST C/retry 带 generation → GET C completed；不重新上传或重复生成 Source | +| Source collect / OpenDAL | 相同解析与确认 API → 持久记录 commit_id → 独立恢复发布 → 收齐精确完成回执后 CAS checkpoint | -若使用既有稳定 Artifact handle，只能绑定某个精确不可变 Revision；它不代表随 Head 更新的外部文档身份，不替代第 3 节的稳定来源绑定。 +正式列表仅展示已发布 Artifact；恢复页同时展示已保存证据但尚未发布的提交，不能把空 Artifact 列表解释为文件未上传。操作取消后仍提供已接受 C 链接。历史与比较复用现有页面，通过精确 Revision.parts 按序读取 Source；不以当前 Head 补旧版本缺段。大文对比有明确计算/展示预算，不完整展示须提示。首期不显示回滚、恢复或合并按钮。 -## 9. OpenDAL 持续采集 -新增 `opendal-text-imports` 模式，保留原 `opendal-text-files` / `text-file-snapshot` evaluation 的 Definition、名称、权限和 checkpoint 行为。新模式通过公开导入 API 提交，不调用每次会注册 manifest 的旧 RemoteConnectorWorker 路径，向已有 Scope 导入无需 `server.admin`；worker 必须配置已有 scope_id，并具备 scope.read + scope.contribute；缺失目标或权限时拒绝运行，不自动创建 Scope。worker 使用同样的 ImportSession 协议。 +## 9. Source collect 与 OpenDAL -worker 使用相同能力矩阵和 parser profile。一次扫描获取完整且有界的选中文件清单，实际读取原始字节并计算 input_digest。mtime/ETag 可辅助诊断，不能单独证明未变。逻辑文档身份是 Scope + source_authority_id + 相对 path/object key。profile、include/exclude、编码或 OCR 策略变化需显式创建新的采集配置 binding/checkpoint，但沿用同一 authority 下的文档身份，变化内容提交为新 Revision；root/桶变化则是新 authority。显示 namespace 不能代替持久来源身份，checkpoint 配置 fingerprint 也不能成为文档身份。 +OpenDAL 采集实现作为 Source collect 插件部署，复用 #2210 的实例、目标、运行进程、启停、凭据隔离与 checkpoint 基础。插件加载要求管理员，运行身份只获得所需 Scope 权限。目标必须已有且不可自动创建。旧 opendal-text-files/text-file-snapshot evaluation 不原地改变类型、身份、权限或进度；新模式显式配置。 -checkpoint 只保存已完成输入身份、原始摘要、解析 profile 与对应含 Scope 的精确 ImportedContent ArtifactRef。稳定 input_key 由 binding、输入身份、原始摘要、profile 及完整选择/解析策略计算，用于匹配输入结果，不直接作为每次上传键。 +collect 插件通过拟新增公开导入适配器使用相同 capabilities/session/upload/confirm/commit-status API，不能访问 PluginContext 私有属性，也不能把当前 run_batch 的逐 Source 提交用作整文档 T1。解析组件可在采集机器附近运行，但 Server 必须重新验证规范化输入的预算、摘要与权限；本期公开上传不接受任意远程 URL/Server 路径。 -worker 在独立于成功 checkpoint 的耐久恢复表中保存 `(binding, input_key, session_id, attempt_uuid, operation_id, expected_head_revision, idempotency_expires_at, retry_after)`；上传键由 input_key 与 attempt_uuid 派生，先以事务/CAS 写入 attempt_uuid 再发送请求。响应丢失或 worker 崩溃时恢复同一尝试和键。多个副本竞争同一输入需复用同一持久尝试；不同服务 Principal 不共享私有任务恢复表。 +Scope+source_authority_id+相对 path/object key 确定文档，mtime/ETag 仅辅助检测，实际读取字节摘要确定输入；优先固定外部不可变版本/conditional read。凭据轮换不改变 authority，根/桶变化是新 authority。profile/filter 策略变化需更新采集配置 fingerprint，但不改变资料身份。 -查询得到 expired tombstone 时，先保留其中已提交回执，再以 CAS 分配新的 attempt_uuid 与上传键重新提供文件；旧 session 到期时为原有 scope_id 创建新会话并持久记录 session_id；超过幂等保留窗口且无法恢复旧任务时同样新建尝试。新尝试仍使用稳定输入映射复用已冻结解析结果，并核对当前 Head 与预期版本；已成功提交且待恢复的原始回执不重复提交，未保存成员继续解析，不因固定键永久卡在 410。容器发现后 worker 依据明确的选择规则提交稳定成员集合；只有本轮期望成员均有耐久回执才完成输入。恢复表推进不代表成功 checkpoint 推进。管理员或用户主动取消的输入标记为 paused，后续扫描不自动重建任务；只有明确恢复该输入后才以新 attempt key 继续。 +持久尝试记录包含 binding、input_key、attempt_uuid、session/operation/item、commit_id、幂等期限和退避时间；先 CAS 记录尝试再发请求，响应丢失恢复同 key。T1 回执到达后持久记录 commit_id，session 到期仍通过 C 恢复制品发布，不重新上传已接受证据。T1 前输入到期才可重新提供字节并新建 attempt key。 -worker 自动确认的范围必须在部署配置中明确，只接受 coverage=complete 的文档。此校验覆盖新解析结果、已有快照候选及恢复回执：即使存在 ImportedContent Revision，coverage=partial 也不能满足 complete-only，必须标为需处理且禁止据此推进 checkpoint。worker 不代表用户接受历史 partial。发现阶段可重试失败通过 stage=discovery 恢复，文档失败通过 stage=documents 恢复；配额拒绝保留旧状态并持久退避。扫描页缺 OCR、损坏附件、选中成员不支持或部分提取时标为需处理，不自动接受 partial。每个被配置选中的容器成员都获得已保存/已存在回执且整轮 listing 完整，才 CAS 提交 checkpoint。仅创建 operation、解析 ready 或上传成功都不够。 +checkpoint 的“完成”定义为本轮完整 listing 中全部选中成员均得到 completed/reused 的精确 ArtifactRef。T1 仅代表 evidence_saved，不推进成功 checkpoint；插件的最近单条 SourceRef 报告也不代表整个文档完成。适配器须复用 checkpoint CAS，并以同事务或等价耐久协议记录提交尝试关联。其他文件可以继续，但失败成员保留未完成状态；崩溃恢复不能重复创建 Revision。 -其他正常文件可继续提交；失败项不回滚它们，但旧 checkpoint 保留。恢复时在有效尝试内同键查询与重试，过期后更换尝试键并复用已保存结果。并发 CAS 冲突重新加载并扫描，不强行覆盖。删除外部路径只影响后续比较，不删除 ImportedContent、历史 Revision 或 Source；不完整 listing、permission denied 或临时连接失败不能作为删除证据。 +自动确认只接受 coverage=complete，涵盖新解析、缓存与恢复结果；不继承历史 partial 接受。用户取消未确认输入后标记 paused，扫描不得自动重建。停止 target 不撤销已接受的服务端发布；界面明确显示已保存提交仍可完成。删除外部文件不删除 Source、Artifact 或历史，不完整 listing 和权限错误不能推断删除。 -尽可能固定外部 immutable revision 或使用 conditional read；无法提供这一能力时，仅承诺处理本次实际读取字节，不把时间戳包装成精确外部 revision。存储凭据只存在于 worker，不进入 Source、checkpoint、日志或 parser。 +## 10. 权限与隔离 -## 10. 安全、权限与隔离 +会话/私有预览只给创建者及显式授权管理员；T1 重新校验 scope.read+scope.contribute；T2 再检查原确认者与 Worker 当前权限。原确认者撤权后发布 blocked，管理员不能仅靠插件管理员权限静默替其接受资料。正式 ImportedContent 与 Source 采用 Scope 共享读取；必须适配 Family 授权、HTTP、runtime、SDK 和页面，不能使列表可见但详情受 Worker owner 限制。 -所有文本/文件导入都要求已选择 Scope 的 `scope.read` + `scope.contribute`；文档列表、历史 Revision 与 Source 阅读要求 `scope.read`。普通导入不需要 `server.admin`，导入服务不执行 Scope 创建或权限初始化。会话和文件任务绑定创建者稳定 Principal；其他普通调用方不能读取未提交预览或代其确认。管理访问须显式 server.admin 且具备目标 Scope 的对应权限，所有代操作审计。每次确认重新鉴权；创建者撤权后不得凭旧会话写入。 +Parser worker 仅得到指定 staging/profile，不获得任意 Scope 写权限。Artifact worker 得到受控提交证据，只读精确分段，不读取输入临时 URL。两种子进程隔离都不等于不可信代码安全沙箱，部署仅允许可信插件；文件仍须实施内存/CPU/超时、无网络默认、路径穿越和解压炸弹防护。 -Parser worker 只获得调度授予的指定 operation、staging blob 和 profile，不获得任意 Scope 写入权限；最终保存由 ContentImportApplication 以确认调用方权限执行。后台解析可以完成已接受输入，但不会自行生成 Source。撤销权限后禁止确认提交，暂存按到期规则清理。 - -Dashboard 所有写路由做同源 Origin 校验，使用现有会话身份转发公开 API；保持 HttpOnly/Secure/SameSite 和 cookie 路径规则，不直接写库、不向 `/v1` 扩展 cookie 权限，不信任未经部署认可的代理头。 - -解析器运行在受限独立进程/容器，限制 CPU、内存、时间、文件数和像素；默认禁用外网，禁止 XML 外部实体、脚本、Office 宏、公式重算、链接更新、Notebook 执行与动态插件安装。用户文件名不用于任意磁盘路径。原始内容与解析文本都视为资料，不作为执行指令。 - -归档拒绝路径穿越、绝对路径、符号/硬链接、设备文件及重复覆盖，实际计量展开预算；不得仅依赖压缩包索引所写尺寸。密码文件不接收解密口令，本期返回 encrypted_document；DRM 不绕过。 - -OCR 默认采用本地引擎和已安装语言包。云 OCR 或视觉模型需要部署显式启用及用户可见的外发说明，不自动将资料传给未知 provider。普通文档提取不应隐式调用生成式模型；OCR、视觉模型与下游 Artifact 用量分开统计。 +不执行代码、宏、Notebook 单元或 HTML 脚本；XML 禁用 DTD/外部实体；归档拒绝绝对路径、遍历、符号/硬链接、设备文件及重复覆盖。密码/DRM 输入明确拒绝。OCR 默认本地；云服务需显式部署和用户可见外发说明，不在线下载模型。日志、错误、指标不携带正文、签名 URL、凭据或私有路径。 ## 11. 错误与可观测性 -保留既有 error envelope，并区分 unsupported_format、parser_unavailable、capability_insufficient、encoding_required、invalid_encoding、encrypted_document、damaged_document、ocr_required、no_extractable_text、parse_timeout、input_limit_exceeded、output_limit_exceeded、archive_limit_exceeded、partial_extraction、profile_conflict、idempotency_conflict、revision_conflict、source_binding_conflict、session_expired、bundle_conflict、selection_conflict、operation_cancelled、operation_expired 与 quota_exceeded。 +HTTP 401/403/404 沿用可见性；非法字段 422、硬限额 413、不支持格式 415、状态/幂等/版本冲突 409、未确认过期 410、配额 429、依赖未就绪 503。202 后的解析或发布错误由 GET 200 的状态对象表达；网络超时始终可能结果未知，先同 key 重放或查 C。429/503 使用 Retry-After。 -HTTP 401/403/404 沿用可见性策略；非法字段 422、单请求硬上限 413、不支持媒体类型 415、幂等/状态冲突及已取消操作 409、过期的未提交内容操作 410、限流 429、依赖不可用 503。202 后发生的解析失败记录在 operation 中，通过 200 状态查询返回 failed 与错误明细，不能伪造先前上传请求的 HTTP 错误。网络超时始终可能意味着结果未知，同键核实。 +分开统计上传、解析完整/部分、T1 接受、发布 pending/blocked/failed/conflicted/completed、索引就绪与下游处理，不把 202 计为导入完成。观测 generation、恢复时延、积压年龄、Source/parts 数和预算，日志关联 commit_id 而指标避免高基数身份。复用仓库 observability 的 operation/unit/outcome/error 规范，记录失败不改变事务结果。 -日志不保存正文、token、完整本机路径、Prompt 或 OCR 图像。指标区分上传字节、识别类型、解析结果、OCR 页数、需确认项、已提交文档/分段、重复命中、解析延迟、提交延迟、checkpoint 冲突与最后成功完成时间。高基数 ID 不作为 metric label。 +## 12. 实施、迁移与验收 -“已保存”只统计取得耐久文档回执的资料；压缩包总览分别显示已保存、失败、未选择、待处理数量。只提取元数据或空正文不算成功。后续生成成本与进度来自处理器自身证据，不能从解析成功推断。 +### 实施顺序与交付门禁 -## 12. 兼容、实施与验收 +1. 固定两项插件依赖版本，登记与原 master 基线的兼容关系；依赖未合入不声称已有完整能力。 +2. 交付 document_commit 输入/受控分段读取/证据验证、Scope 共享 Family 权限及 T2 同事务完成适配器，覆盖最大文档预算。 +3. 实现九项导入持久职责、T1、C 查询/重试、幂等与配额；HTTP/OpenAPI 统一生成，不手改生成代码。 +4. 实现官方 ImportedContent 插件和共享解析组件，保留第 1 节全部 A/B 目标与 C 可选范围；未验收 profile 保持 planned。 +5. 接入 Dashboard/CLI/SDK 和 Source collect/OpenDAL 公开适配器，执行故障恢复与完整文档验收。 -### 实施边界 +不自建第二套 Artifact 版本或 Source 插件进程管理；不把解析 worker 当 Artifact 生成 worker。仅安装插件不能宣称完成文本导入。长图为说明交付物，不嵌入 RFC；本 PR 只提出设计，不修改运行实现或数据库。 -交付采用第 1 节固定的解析族与分支，不再将“确定每种格式使用何种解析器”留给运行环境选择。交付按职责推进，但 A、B 格式范围都是正式目标：先实现文本保存与统一 manifest，再实现 Family 文件操作创建/commit、内部耐久操作账本和标准解析包，再完成 OCR/长尾适配及 OpenDAL。不能仅完成 A 就宣称整个多格式 RFC 已实现。 +### 数据库与兼容 -注册 importedcontent Family、补充操作账本及查询索引，不改变既有 Source/Artifact 引用。历史 Source 不自动包装为 ImportedContent Revision。更新 family allowlist、精确读取、历史、lineage 及 Dashboard 阅读器；不得把 ImportedContent 自动纳入模型生成或 Candidate 流程。旧 OpenDAL binding 不原地换新模式；显式重新导入可生成新的 content Source，不能声称保留旧类型身份。Core Python 支持范围保持不变，较重解析依赖由独立 worker runtime 承担。 +SQLite、embedded seekdb、OceanBase 均需登记导入与依赖框架 schema 的空库初始化、升级、索引、唯一约束和后置校验；不得首次请求临时建表。复用 db-migrate status/plan/apply/verify，新增 revision 与冻结 manifest，不重写已发布迁移。DDL 中断按后端实际事务能力恢复，MySQL 兼容 DDL 不承诺整次回滚。 -### 数据库初始化、升级与恢复 +备份与维护门禁覆盖所有写者；跨进程 checkpoint/文档占位/回执用数据库 CAS 和唯一约束，不只用进程锁。Source/Artifact 历史身份与引用保持不变，既有 content 不自动包装为 ImportedContent。验证 TTL 不删除待发布证据及永久结果映射。实现验收明确记录后端：本地 SQLite 可以先验证，seekdb/OceanBase 无实例时标为未实测，不以 SQLite 结果代替。 -导入的所有持久结构必须同时覆盖 SQLite、embedded seekdb 和 OceanBase。第 3 节的三张核心操作表，加上实际采用的来源绑定、回执、配额账本及查询索引，都纳入受版本管理的 schema；不能把辅助表留给首次请求临时建表。持久资料仍复用 Artifact/Source 表，不新增 ImportedContent 主表或版本表。 +### 必须通过的行为验收 -现有 `server db-migrate status/plan/apply/verify` 提供迁移基础，但当前注册 bundle 仅管理有限表集，不能把它视为已经覆盖全部业务表。实现须在该机制中登记导入 schema bundle/revision、冻结表/索引 manifest 和后置校验，并明确与现有 bundle 的依赖及兼容版本，不直接修改已有发布 revision。 - -| 项目 | 必须落实的约束 | +| 场景 | 验收结果 | | --- | --- | -| 空库与已有库 | 空库初始化和已有部署升级得到相同 schema；升级前核对受管表、索引、历史 revision 和冲突残留，不把任意已有表自动标记为已迁移 | -| 唯一性与查找索引 | 保证会话/操作主键、`(operation_id,item_id)`、稳定来源身份、幂等键分区唯一；按会话创建者、Scope、状态/租约到期、TTL 建有界恢复与清理索引；长 key 按后端索引限制采用有碰撞校验的摘要索引，不静默截断身份 | -| 升级门禁 | API 写入、解析调度、Connector 提交与 GC 均遵守迁移维护门禁；schema 未就绪时不接收导入，不能让一部分副本按新结构写入、另一部分继续按旧结构执行 | -| DDL 中断恢复 | SQLite 按后端能力使用事务；seekdb/OceanBase 的 MySQL 兼容 DDL 可能隐式提交，不能承诺整次 schema 升级可事务回滚。记录步骤和校验结果，中断后先 inspect/plan，再恢复或人工处置，不盲目重复非幂等 DDL | -| TTL 与运行数据 | 清理会话、操作、回执和暂存引用的期限符合第 6、7 节；保留恢复所需 tombstone/回执。稳定来源绑定、已发布 Artifact、Source 和仍被引用的映射不能随操作 TTL 一起删除 | -| 降级与发布 | 功能开关可停止新导入，但不删除持久资料；禁止未经验证的 schema downgrade。注册 bundle 的 verify 成功只证明其受管结构满足后置条件，不替代整库、权限与服务端到端验收 | - -迁移交付必须包含三种后端的空库、已有数据升级、重复执行和中断恢复验收；检查既有 Source/Artifact 与精确引用不变。迁移脚本、manifest、索引及初始化路径必须随功能一同落地，禁止仅在开发 SQLite 自动建表后宣称支持生产后端。 - -### 契约与验收 - -HTTP 验收须覆盖 Scope 路径与父子归属、body 重复身份拒绝、标准 source_type 引用和平铺 Artifact 回执、created=201/updated或reused=200、400/409/410/422 分工、幂等 202 原回执重放、GET 200 的异步失败、parse_status/commit_status 映射，以及通用 Artifact 的 ETag/304/412/428 行为未被导入命令改变。中英文操作清单、参数默认值、请求示例与 OpenAPI 必须一致；长图只作单独说明材料，不嵌入 RFC。 - -实现以 `openapi/powercontext.yaml` 为契约源，执行 `make api-generate`、`make contract-test`，不手改生成代码；运行 `make check`、相关 Source/Server/Access/Dashboard/Connector 测试及 `make docs-test`。影响 Python 支持范围时运行 `tox`，跨组件验收放入 `tests/e2e/`。本 RFC 被接受不表示验收已通过。 - -| 场景 | 验收要求 | -| --- | --- | -| Scope 授权的完整读写链路 | 仅具有 scope.read + scope.contribute 的用户可完成导入并读列表、详情、历史及正文；无需 Artifact owner 或 Prompt 管理员权限；撤权后所有对应路径失效 | -| 通用恢复/合并、直接 POST/PUT、retired 目标 | API 与 Runtime 同时拒绝，不产生孤立 Revision 或丢失 lineage.sources；页面无恢复/合并入口，导入不复活 retired 目标 | -| 普通 content 的角色与生成准入 | working_note=False、internal=None、handoff_receipt=False；伪造保留 metadata 被拒绝；符合既有准入规则的分段进入已配置处理器 | -| 原子保存、幂等重放与通知丢失 | 分段、dirty/pending、Artifact、Head 和回执共同提交或回滚；重放不重复触发，提交后丢失通知仍可恢复处理 | -| 三后端 schema 初始化/升级/中断恢复 | SQLite、embedded seekdb、OceanBase 的表/索引约束一致；已有数据保持可读；DDL 中断不误报 ready，多副本遵守维护门禁 | -| 静态契约、运行能力与 Scope 级发现 | feature contract 不等于启用；普通导入者无 server.observe 也可查目标能力；部分解析器故障只影响相应 profile，关闭新导入后历史仍可读 | -| 版本正文比较与搜索边界 | 按精确 Source 比较正文和语义元信息，超预算明确提示，历史不依赖 staging；未注册 searcher 时明确不支持，namespace 不冒充全文搜索 | -| 必选已有 Scope、缺失/无效目标 | UI 禁止开始导入；API 缺失或非法参数返回 422，不存在/不可见/无权限按 404/403 拒绝，不接受输入或建立操作 | -| 固定会话目标、切换 Scope、跨 Scope 请求 | 会话 scope_id 非空且不可修改；子操作不能覆盖目标，更新资料和来源绑定均校验归属；切换目标使用新会话 | -| 普通导入权限、上传后撤权或目标失效 | scope.read + scope.contribute 足够，无需 server.admin；确认重新校验，失效后不写入或自动补建目标 | -| 多会话累计配额与并发提交 | 从接收起计入同一目标 Scope，多个会话不绕过累计限额；预留/释放原子且无额度迁移 | -| CLI 与 OpenDAL 目标配置 | 新导入缺少已有 scope_id 时在上传前拒绝；恢复会话保持原目标，同时传入冲突目标时失败 | -| 同名不同文档、同文档改名、独立新建同内容 | 文件名/摘要不决定身份；明确更新保持 artifact_id，独立新建不合并 | -| A→B→A、旧请求重放、Head 并发冲突 | 回归内容生成 R3；旧键返回旧回执不回退 Head；冲突 409 不覆盖 | -| parser/filter 变化、路径重命名、容器成员重排 | 配置变更不误换文档；无稳定 ID 的重命名/成员变化不猜测关联 | -| ImportedContent 元信息、parts、lineage 与反查 | 精确版本/分段一致；不重复正文、不伪造全文消费或预填下游关系 | -| ImportedContent 类型与 importedcontent wire name | 持久化、Family 注册、OpenAPI、客户端和 Dashboard 一致；标准四个读取操作可访问 | -| 统一 content Source 与不同输入方式 | 粘贴文本、上传 HTML/PDF、文件成员及 OpenDAL 输入均保存 source_kind=content；读取分段的 SourceRef.source_type 一致，网页获取仍需独立 Connector | -| source_kind 固定值与请求归一 | 省略字段与显式 content 的请求指纹、摘要及缓存语义一致；其他值返回 422；不新增恒定字段的筛选索引 | -| 跨输入方式更新与历史版本 | 文件改为粘贴仍校验 Head 和完整快照语义；各版本 source_kind 均为 content，具体 origin、格式和 parser 独立保留 | -| 新资料预览与已有资料更新中 | 新资料无 Artifact/Source 可见；旧 Head 在解析、失败、重试、取消后保持不变，只有确认成功推进 | -| 通用 Artifact 写入与内部状态篡改 | POST/PUT 不能伪造 ImportedContent；operation 无任意 PUT/PATCH 状态入口，跨会话句柄被拒绝 | -| 私有操作与 Scope 普通读者 | 同 Scope 的其他读者不能读取/确认待提交预览；只能通过 Artifact API 读已接受内容 | -| 操作清理和治理状态 | 清理后 Artifact/Source 精确版本仍可读；进度不产生 Revision，不把 parsing/failed 写入治理状态 | -| 默认路由、后端不可用和一次备用解析 | 冻结 router/profile；不隐式开启 OCR/外发；备用共享预算，冻结后不得覆盖 | -| LiteParse 页数上限、page_errors、复杂度与 OCR | 不把探测信号当执行，不把缺失页当完整；复杂布局使用固定 Docling 分支 | -| XLSB 完整 profile 与仅值 profile | 公式契约不足保持 planned；仅值须显式选择且 needs_review，complete-only 不接受 | -| OOXML 模板/宏变体、ODF/Flat XML、重复行列 | 原容器识别和关系正确；不靠改后缀，不执行宏；节点及展开预算先于读取生效 | -| Office 源码审计与转换定位 | 修订/脚注/隐藏/备注符合选项；不将转换页号映射成不存在的原生位置 | -| 外部服务任务、预签名资源与临时图片 | 实际版本与外发可追溯；保存文字不依赖过期 URL，不承诺原图长期在线 | -| 每种正式宣称支持的格式与格式代际 | 至少有可核对正文、定位与遗漏的真实夹具；只识别扩展名不通过 | -| TXT/代码/配置、UTF-8/UTF-16/GB18030/Big5 | 严格解码、中文与行号正确，错误编码不静默替换 | -| PDF 文本页、扫描页、混合页、多栏和表格 | 页序、OCR 去重和遗漏正确；缺 OCR 不误报完整 | -| Word 表格/脚注/修订/批注 | 符合明确视图与选择，不能静默丢弃所选内容 | -| XLS/XLSX/XLSB、多表、隐藏行列、公式缓存缺失 | 不只首表、不重算、不虚构值，位置可核对 | -| PPT/PPTX/模板、备注、隐藏页 | 顺序与纳入选项正确，不执行宏 | -| HTML/Notebook/EPUB/EML/MSG/MBOX | 无外链抓取或执行，正文选择和子文档边界正确 | -| OCR 图片、多页 TIFF、超大像素与空白页 | 限额生效，元数据不能代替正文，结果可追溯 | -| Office/EPUB 容器与普通 ZIP/TAR | 正确区分；归档路径穿越、嵌套与解压炸弹被限制 | -| 扩展库未安装或仅返回 metadata | capability 不报 available，不制造空 Source | -| 长文、多分段、超限和单段极长 | 拼接 digest 等于完整提取结果；超限整文档失败 | -| 上传后断线、worker crash、旧租约结果返回 | 原 operation 可恢复，旧 generation 不能覆盖冻结输出 | -| OCR 预览后重试、确认并发、profile 升级 | 确认固定 bundle，相同输出复用，新输出保留旧证据 | -| 保存前后摘要、不同 SourceRef/时间戳 | 内容摘要不受保存回执影响，同一文档目标与预期 Head 的并发提交至多一个成功推进版本 | -| 同 Scope 两个 Principal 上传相同内容 | 不复用对方私有 operation；独立新建仍是不同文档；更新同文档须校验权限与 Head | -| 任务过期及幂等映射清理后 worker 恢复 | 新 attempt key 可推进，已提交资料不重复，checkpoint 不提前更新 | -| 发现、冻结选择、部分提交、取消与 GC 竞争 | 状态字段与聚合规则一致；已提交不回滚，未提交不会越过取消/到期 | -| as_text、reparse、附件选择、retry、cancel | API/CLI/UI 契约一致；非法组合拒绝，同文档当前 Head 语义相同的重解析不重复 Source | -| 多副本、多调用端、配额耗尽与 worker crash | 累计预留原子生效，实际删除后释放存储，OCR 已用额度不退款，持久退避可恢复 | -| 用户 A 接受 partial，用户 B 或新的导入尝试命中同一快照 | 本次仍为 needs_review/uncommitted，显示遗漏并显式确认；同文档当前 Head 相同才 reused，独立新建生成新文档；同一次提交重放不重复确认 | -| OpenDAL 命中已保存 partial 或恢复 partial 回执 | complete-only 拒绝自动完成，checkpoint 不推进，不因存在 ImportedContent Revision 绕过覆盖检查 | -| 相同输入以新键上传，旧任务 expired/cancelled/failed/completed | 不合并到旧任务；新 operation 独立创建，持久记录按 coverage 规则复用；同旧键重放仍返回旧结果 | -| 成员发现阶段临时失败且自动重试耗尽 | 无 item_ids 也可 stage=discovery 重试；完整清单原子发布，已冻结清单及成员 ID 不变 | -| 已释放任务槽的 failed（可含部分已提交）重试 | 配额满时 429/503，状态和计数不变；额度恢复后同键准入只占一槽，不重复入队 | -| partial 候选等待确认、容器仅剩失败成员、单文档失败 | 待确认持续占槽；无活动工作的失败可释放任务槽；其他成员的活动槽与暂存计量不受错误释放 | -| 写入中任一分段/记录/pending 故障 | 整份文档回滚，恢复后不重复 Source | -| 容器部分成员失败、用户确认 partial | 逐文档结果真实，遗漏长期保留，不冒充完整导入 | -| 撤权、跨 Scope、跨站请求和恶意文件 | 不能确认写入、越界读取或执行内容 | -| worker checkpoint 与 profile/filter 改变 | 只对完整已提交结果推进，配置变更不静默跳过 | -| SQLite/embedded seekdb/OceanBase、多副本暂存、过期 GC 竞争 | 恢复与提交一致，GC 不破坏正在确认的 bundle | -| 中英文、明暗主题、小屏、关闭后恢复 | 支持清单、预览、遗漏、逐项重试和阅读全文可用 | +| 仅 scope.read+scope.contribute 的用户 | 无需管理员即可预览、确认、查 C 和读取发布内容；部署插件权限独立 | +| T1 任一 Source/清单/回执写入失败 | 全文证据和发布意图均不提交，不产生 Artifact | +| T1 成功后通知丢失、进程重启 | 根据持久意图恢复，返回原 202，不重复 Source | +| T2 任一步骤失败或回复丢失 | Artifact、Head、索引、完成回执共同回滚或共同提交；重试至多一个 Revision | +| 80 页/40 段跨 journal 窗口 | 按 commit 完整装配，缺段/乱序/摘要错阻塞，不先发布半文档 | +| 8 MiB/128 段、2 MiB 清单 | 模式独立预算与 4 MiB draft 上限实测；其他窗口预算不变 | +| 伪造 commit metadata/SourceRef、跨 Scope | 拒绝，不包装普通 content，不扩大证据授权 | +| 两次更新及 A→B→A | 同文档串行准入；R1/R2/R3 正确，旧任务不能覆盖新 Head | +| T1 后撤权、retire、Head 改变 | blocked 或 conflicted；旧版本可读，证据保留，不静默改目标 | +| 插件 disable/unload/reload | 已发布可读，未完成持久保留，旧 generation 不得提交，重新就绪后可恢复 | +| 24 小时到期、取消、30 天请求映射过期 | 仅清理私有输入；C 仍能核实已接受/已发布结果 | +| collect 与 checkpoint 间崩溃 | T1 不冒充完成；用 C 精确回执恢复后 CAS，不重复发布 | +| 文件格式矩阵 | 实测结构、编码、公式/缓存、修订/批注、OCR、容器选择、定位与遗漏；metadata-only 失败 | +| HTTP 与中英文 | 17 项导入操作一致；新增接受 202、复用 200、查询失败状态 200；标准 Artifact 条件请求不变 | +| 数据库与界面 | SQLite/seekdb/OceanBase 独立报告；预览/证据保存/制品完成/索引就绪明确区分 | + +实现须运行 make api-generate、make contract-test、make check、针对 Source/Artifact/Access/Parser/SDK/Dashboard/Connector 的测试，以及 make docs-test；跨组件测试放 tests/e2e，影响 Python 支持范围时运行 tox。设计文档检查不等于运行时验收。 # Drawbacks -广泛格式支持显著增加依赖、样本和安全维护成本。Office 传统格式、复杂表格与各代专有文档的能力不一致，必须维护能力矩阵并保留失败原因，不能使用“支持任意文件”的笼统承诺。 +两阶段允许证据可见而制品尚未生成，发布失败不能回滚已被下游消费的 Source。需要持久交接和独立恢复界面；单文档一个在途提交会阻塞该文档的新更新，但避免未定义的乱序版本语义。 -解析 worker、耐久暂存、租约和分段事务比单份文本捕获复杂；OCR 的速度、准确率和语言覆盖也需要独立预算。保留提取文本和 digest 无法还原原始二进制的所有布局、图像与交互内容。 - -文档原子提交可能同时追加多个 Source，增加单事务成本和后续消费量。8 MiB/128 段等上限必须以三种数据库后端的验收控制，不能无限提高。归档允许部分成功，用户需要理解容器完成与单份文档完成的区别。 - -版本模型保留旧新文本；Head 只表示最近接受的版本，不提供外部实时最新事实选择或删除传播。一次解析正常完成也不能证明模型能准确理解其中所有内容。 +完整文档输入和同事务完成适配需要扩展框架，不能仅靠两个现成插件声明完成。大文档预算、解析依赖和 OCR 成本仍需真实样本验证；插件化不会消除这些成本。 # Rationale and alternatives -**导入资料使用 ImportedContent Artifact + Revision。** 复用既有不可变版本、Head、lineage 和读取能力，比另建 ImportRecord 更容易提供版本历史与资料关系查询；代价是必须注册新 Family、提供专用写事务和阅读器，并补齐来源绑定与查询索引。source_kind 固定为 content，复用现有文本 Source；具体出处、文件格式和解析器独立记录。运行状态由内部可清理操作账本承载，不新增 FileImportJob 业务实体，不因每次解析进度创建 Revision。 - -**显式文档身份与固定已有 Scope。** 手动新建和更新资料明确区分，自动采集使用受控稳定来源键，避免同名/同内容误归并。导入前必须选择已有可贡献 Scope；会话只维护固定目标、私有预览和恢复，不负责创建 Scope、初始化权限或迁移配额。这使上传、确认和配额从开始到结束使用同一权限边界。 +选择两个阶段以复用 Artifact 插件的执行和发布；T1 的原子范围是文档证据与发布意图，T2 是制品及完成记录。等待 HTTP 请求直到插件完成并不能把它们变成一个事务。 -**原生结构与 PDF 布局分开。** Excel、演示、邮件和归档具有原生对象与成员身份，不统一转 PDF；PDF 采用 LiteParse 轻量分支和 Docling 布局/OCR 分支，复杂替代引擎作为独立 profile。接受多种依赖的代价，换取可说明的结构、定位与遗漏。RAGFlow/LlamaIndex 的任务、去重、索引与默认 Reader 不直接承担本 RFC 的保存事务。 +只保存 Source 可直接使用 content.inline，但缺少资料身份、版本和导入结果阅读。Source+Artifact 同步原子提交可以保留另一种产品契约，但需要专用受控发布接口，不能声称直接使用现有 generate 工作流。逐 journal 窗口拼装文档需新增持久聚合与证据回读，且容易提前推进游标；本提案采用完整 commit 作为输入单位。 -**采用可插拔解析能力与固定路由。** 一套只接受 UTF-8 的接口无法覆盖 Office/PDF；一个通用解析器也不能保证所有格式的正文与结构。固定并验收每类适配器比依赖隐式 fallback 更可诊断，也允许未来增加长尾格式。 +解析采用原生结构与 PDF 布局分流，复用成熟解析库；不让 RAGFlow/LlamaIndex 接管版本、权限、任务或事务。正文仅保存一份 Source，ImportedContent 记录精确引用。后续全文索引需要明确的派生投影设计，不用复制全文来绕开当前输入预算。 -**隔离文件解析，复用 ContentSource。** 解析依赖和文件处理风险不进入 Core 请求进程；归一后的文本继续使用已有 Source、Scope、lineage 和 Artifact 处理，无需按每个后缀新增 Source family。 - -**文件异步解析、单份资料同步提交。** OCR 与大文件无法保证在普通 HTTP 超时内完成；创建文件操作返回操作句柄，内部耐久执行记录承担恢复责任。冻结结果后单文档事务可以可靠确认，避免边解析边公开部分 Source。 - -**分段保存而非截断或强迫用户手工拆文件。** 自然边界和定位保留可用性，精确拼接和全量摘要保护完整性。原子保存保证一份文档不出现仅前几页已消费的半提交状态，但不强制所有 Artifact 在同一轮消费整份文档。 - -**提取能力与“理解所有内容”分开。** 正文、表格、OCR 和元数据的覆盖不同。已知缺失必须报告，不能以推测补全文字或把元数据算成正文；用户接受 partial 的决定应留在记录中。 - -**不把库的支持列表直接当产品支持列表。** 产品可用性还取决于格式代际、依赖、选项、输出质量、资源限额与测试。能力发现只声明部署已完成的适配。 # Prior art @@ -1241,16 +1096,10 @@ HTTP 验收须覆盖 Scope 路径与父子归属、body 重复身份拒绝、标 # Unresolved questions -1. 第 1 节已确定各格式的默认解析族和候选边界；具体发布依赖版本、PDF 路由阈值及性能预算仍需同样本验收后冻结。XLSB 公式、传统 Word 修订视图与长尾代际的适配缺口未补齐前保持 planned，不以部分 profile 宣称完整交付。 -2. 50 MiB 原文件、8 MiB 提取结果、500 页和 128 段等默认预算需要在 SQLite/embedded seekdb/OceanBase 与实际部署资源上验证；调整时同时评估事务与下游消费成本。 -3. C 级 OCR 的首批语言包与专有格式代际需按真实资料验收。Apple/WPS 等长尾格式进入 available 必须有正文样本证明，不能只凭类型检测。 -4. 是否长期保留原始二进制及支持下载，需要单独定义存储配额、保留期、精确证据和删除授权；本提案只保留解析暂存。 -5. 如业务要求下游始终仅消费文档 Head 的 Source，需另行定义版本选择、旧 pending 与历史引用处理；本提案的 ImportedContent Head 不自动改变既有 Source 消费规则。 +1. document_commit 的 SDK 命名、物理表映射与适配器装配位置需随框架实现固定；完整证据校验、两阶段事务和预算是不可省略的验收约束。 +2. 50 MiB 文件、8 MiB 正文、128 段及 OCR 配额需真实样本与三种后端分别验证；未通过不得发布该能力。 +3. 标题/出处检索是否作为默认部署声明，需要在首个 searchable schema 注册前确定；正文索引另行定义，不能假定热更新支持。 # Future possibilities -在已有解析注册与能力发现机制下，可以继续接入更多语言、行业文档和专有格式，并单独定义音视频转写、图表理解、公式语义和手写识别。它们的成本和准确性不能隐含在普通文件解析中。 - -语雀、网页和其他知识平台以独立 Connector 接入，明确外部身份、版本、授权与精确读取能力。结构化数据库和超大型数据集应设计有选择条件的采集协议，而不是把任意数据库文件当普通文档无限展开。 - -可进一步支持长期原始文件归档、原文检索及精确位置跳转；资料级生成状态需由各 Family 提供真实 Source 覆盖和产出证据。所有扩展继续保留 Scope 边界、快照身份、遗漏报告和耐久回执。 +可以增加其他可信 Source Connector、长期原始文件归档、明确的全文检索投影及跨文档引用浏览。批量容器仍逐文档发布；下游 Memory 的覆盖与完成由其自身证据报告。所有扩展保持已有 Scope、稳定文档身份、精确版本、显式遗漏和可恢复提交。