Repository navigation
Integrate data snapshot pipeline - #44
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Images sharing a filename stem can silently lose metadata through skipped extraction or overwritten output.
1 open issue
What changed in this PR
Integrates a production Data Snapshot pipeline into ai4data, separate from the experimental research package.
Changes:
- Adds PDF snapshot extraction with four detector adapters and stage-specific commands.
- Adds Schema v1.4 metadata extraction through OpenRouter or OpenAI.
- Adds installation extras, packaged resources, documentation, and offline CI tests.
| File | Description |
|---|---|
| tests/data_snapshot/test_yolo26.py | Tests detection normalization and cached weights. |
| tests/data_snapshot/test_yolo11.py | Tests YOLO11 defaults. |
| tests/data_snapshot/test_tfid.py | Tests TF-ID detection and configuration. |
| tests/data_snapshot/test_submission.py | Tests mock submission output. |
| tests/data_snapshot/test_pipeline.py | Tests orchestration and output cleanup. |
| tests/data_snapshot/test_metadata_schema.py | Tests schema examples and version. |
| tests/data_snapshot/test_metadata_extraction.py | Tests metadata requests and failures. |
| tests/data_snapshot/test_metadata_batch.py | Tests JSON writing and replacement. |
| tests/data_snapshot/test_installation.py | Tests installed imports and resources. |
| tests/data_snapshot/test_foundation.py | Tests discovery, rendering, and naming. |
| tests/data_snapshot/test_extraction.py | Tests cropping and duplicate PDF rejection. |
| tests/data_snapshot/test_doclayoutyolo.py | Tests default detector configuration. |
| tests/data_snapshot/test_detection.py | Tests detection validation. |
| tests/data_snapshot/test_cli.py | Tests command arguments and errors. |
| tests/conftest.py | Exempts snapshot tests from classifier mocking. |
| src/ai4data/data_snapshot/submission.py | Adds mock submission. |
| src/ai4data/data_snapshot/rendering.py | Renders PDF pages as RGB images. |
| src/ai4data/data_snapshot/README.md | Documents installation and usage. |
| src/ai4data/data_snapshot/pipeline.py | Coordinates extraction stages and reruns. |
| src/ai4data/data_snapshot/naming.py | Defines deterministic snapshot names. |
| src/ai4data/data_snapshot/metadata_schema/models.py | Defines Schema v1.4 models and validation. |
| src/ai4data/data_snapshot/metadata_schema/__init__.py | Exports schema models. |
| src/ai4data/data_snapshot/metadata_extraction/prompts/user.md | Defines field extraction guidance. |
| src/ai4data/data_snapshot/metadata_extraction/prompts/system.md | Defines image-only evidence boundaries. |
| src/ai4data/data_snapshot/metadata_extraction/extraction.py | Implements metadata requests and validation. |
| src/ai4data/data_snapshot/metadata_extraction/config/openrouter.json | Adds OpenRouter settings. |
| src/ai4data/data_snapshot/metadata_extraction/config/openai.json | Adds OpenAI settings. |
| src/ai4data/data_snapshot/metadata_extraction/batch.py | Writes metadata for image batches. |
| src/ai4data/data_snapshot/metadata_extraction/__init__.py | Exports metadata extraction APIs. |
| src/ai4data/data_snapshot/inputs.py | Discovers PDFs and images recursively. |
| src/ai4data/data_snapshot/extraction.py | Detects, crops, and saves snapshots. |
| src/ai4data/data_snapshot/extract_metadata.py | Adds metadata-only entry point. |
| src/ai4data/data_snapshot/extract_data_snapshot.py | Adds snapshot-only entry point. |
| src/ai4data/data_snapshot/detection.py | Defines the detector contract. |
| src/ai4data/data_snapshot/cli.py | Implements stage-specific commands. |
| src/ai4data/data_snapshot/adapters/yolo26.py | Adds YOLO26 inference and normalization. |
| src/ai4data/data_snapshot/adapters/yolo11.py | Adds YOLO11 configuration. |
| src/ai4data/data_snapshot/adapters/tfid.py | Adds pinned TF-ID inference. |
| src/ai4data/data_snapshot/adapters/doclayoutyolo.py | Adds the default DocLayout-YOLO adapter. |
| src/ai4data/data_snapshot/adapters/__init__.py | Selects detector adapters. |
| src/ai4data/data_snapshot/__main__.py | Adds end-to-end module execution. |
| src/ai4data/data_snapshot/__init__.py | Exposes lazy public APIs. |
| src/ai4data/__init__.py | Exposes the snapshot namespace. |
| README.md | Adds installation and usage links. |
| pyproject.toml | Adds optional runtime dependencies. |
| experimental/data-snapshot/README.md | Distinguishes research and production workflows. |
| .gitignore | Ignores outputs and the TF-ID environment. |
| .github/workflows/data-snapshot.yml | Builds distributions and runs offline tests. |
Review effort: Balanced
Add validation to ensure unique snapshot image filename stems. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Contributor
Author
|
Fixed issues and merge conflicts. Updated the readmes to match the updated docs. @avsolatorio for your review, please :) |
avsolatorio
reviewed
Oct 23, 2026
…iable overrides. Precedence: explicit Python arguments → environment variables → JSON defaults
Contributor
Author
|
@avsolatorio I moved the model defaults to a json file and implemented env variable overrides. The variable names are documented in the readme. Precedence: explicit Python arguments → env variables → JSON defaults |
avsolatorio
approved these changes
Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Integrates the production Data Snapshot pipeline as
ai4data.data_snapshot, separate from the research implementation underexperimental/data-snapshot.Documentation scope
Documentation is limited to repository READMEs covering installation, configuration, commands, outputs, and development.
Concepts, methods, worked examples, and user guides intended for the published site will be covered in a separate PR.
Installation and release
The new extras are absent from
ai4data==0.1.0. The README documents installation from the checkout until a release containing this module is published. After publication, users can install:pip install "ai4data[data-snapshot-doclayoutyolo,data-snapshot-metadata]"Editable installation remains documented for development. TF-ID requires a separate environment because of its older model dependency stack.
Validation
Limitations
Submission remains a mock implementation. The full repository test suite and real inference for every optional adapter were not run.