Repository navigation
[task] Preserve document metadata in ingestion pipeline tests #77
Description
Checks
- I searched open and closed issues and pull requests and did not find a duplicate.
Area
Tests
Goal
Extend ingestion pipeline test coverage so metadata produced by extractors is verified after content is stored in the memory store.
Anchor has tests for individual extractor behavior and tests for basic ingestion. However, the pipeline-level tests mostly verify that content can be ingested and queried. They do not yet prove that document metadata survives the full path from extractor output to stored chunk metadata.
As file ingestion becomes a public workflow, this metadata path matters for debugging retrieval results and for downstream UI/inspection features.
This issue should be treated as a test-focused follow-up to #75 and #76.
Suggested file locations
tests/fixtures/ingestion/tests/integration/test_ingestion_pipeline.pytests/unit_harness.py
Scope
Add focused tests for metadata preservation across the extraction-to-ingestion boundary.
The tests should cover:
- Plain text extractor metadata survives ingestion.
- Markdown extractor metadata survives ingestion.
- Stored metadata includes
source. - Stored metadata includes
source_format. - Stored metadata includes character offsets when the extractor provides them.
- Stored metadata includes heading metadata for Markdown sections where applicable.
- Existing ingestion metadata such as
questionsandtimestampis still present.
Out of scope
- Do not change extractor behavior unless needed to make an existing documented field flow through.
- Do not add new metadata fields.
- Do not add PDF, Word, or source-code ingestion.
- Do not add directory ingestion.
- Do not introduce new dependencies.
- Do not rewrite the ingestion pipeline.
Definition of done
- Tests fail if extractor metadata is dropped before reaching
MemoryStore.add. - Tests cover at least one
.txtfixture and one.mdfixture. - Tests cover Markdown with heading metadata.
- Tests confirm generated ingestion fields still appear alongside document metadata.
- The tests use deterministic/offline fixtures and do not require a live model.
Activity
HarperZ9 commentedon Jul 18, 2026
Hi Alexander, I can take a focused fixture-only pass if this is still open. Proposed boundary: one tiny
.txtand one.mdfixture with stable metadata, a recorder aroundMemoryStore.add, assertions forsource,source_format, character offsets, Markdown headings,questions, andtimestamp, plus a negative control that drops extractor metadata and proves the test fails.I would start from current main, make no extractor, API, or dependency changes, and run the repository's offline suite. I build provenance and replay tooling at Zentropy Labs, and I can attach an optional fair-source Gather and Forum field-preservation matrix as supplementary evidence, not a dependency or Anchor integration claim.
Should the first contribution cover only metadata already emitted today, or also include a documented xfail for a field that is currently lost?
AlexanderGardiner commentedon Jul 18, 2026
Hi, yes this is still open. Those sound like good starting assumptions, and yes xfail is appropriate. I'm not sure what you mean by Gather and Forum field-preservation matrix, but it is not needed currently.
AlexanderGardiner commentedon Jul 18, 2026
Additionally please make sure to claim the issue if you begin work.
github-actions commentedon Aug 24, 2026
This issue has been quiet for a while. If it is still relevant, add context and it will stay open.
HarperZ9 commentedon Aug 28, 2026
/claim
1 remaining item
github-actions commentedon Aug 28, 2026
@HarperZ9 claimed this issue. Thanks for taking it on.
github-actions commentedon Sep 4, 2026
@HarperZ9 there has been no claimer activity for 7 days. Please reply within 24 hours if you're still working on this issue.
HarperZ9 commentedon Sep 4, 2026
Still working on this. I have rechecked the current issue and will keep the contribution to deterministic .txt/.md fixtures and focused metadata-preservation assertions across the extractor-to-MemoryStore.add boundary, including the agreed xfail where the existing field is currently lost. I will not change extractor behavior, APIs, dependencies, or the ingestion pipeline. I am starting from current main and will report back with the offline test results and PR.
github-actions commentedon Sep 12, 2026
@HarperZ9 there has been no claimer activity for 7 days. Please reply within 24 hours if you're still working on this issue.
github-actions commentedon Sep 14, 2026
No response was received within 24 hours, so this issue has been unclaimed and returned to
status: ready-for-work.HarperZ9 commentedon Oct 15, 2026
/claim
The work for this issue is in #102, which is rebased on current main: 6 tests pass and 4 are expected failures (xfail) that mark the metadata gaps this issue describes.