Integrate data snapshot pipeline

.github/workflows/data-snapshot.yml

@@ -0,0 +1,43 @@
+name: Data Snapshot
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "src/ai4data/**"
+      - "tests/data_snapshot/**"
+      - "tests/conftest.py"
+      - "pyproject.toml"
+      - ".github/workflows/data-snapshot.yml"
+  pull_request:
+    paths:
+      - "src/ai4data/**"
+      - "tests/data_snapshot/**"
+      - "tests/conftest.py"
+      - "pyproject.toml"
+      - ".github/workflows/data-snapshot.yml"
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Build wheel and source distribution
+        run: |
+          pip install build
+          python -m build
+      - name: Install wheel with test dependencies
+        run: |
+          for wheel in dist/*.whl; do
+            pip install "${wheel}[data-snapshot-metadata]" pytest
+          done
+      - name: Test offline from the installed package
+        run: python -m pytest tests/data_snapshot -q

.gitignore

@@ -2,6 +2,7 @@ _build/
 sample.doc.json
 *.DS_Store
 data/
+/output/
 
 # Byte-compiled / optimized / DLL files
 __pycache__/
@@ -127,6 +128,7 @@ celerybeat.pid
 # Environments
 .env
 .venv
+/.venv-tfid/
 env/
 venv/
 ENV/

README.md

@@ -211,6 +211,10 @@ Each capability is an optional extra.
 | `metadata` | Schema mapping, PDF-to-metadata extraction, and metadata augmentation |
 | `discovery` | NADA catalog access, metadata templates, and PDF helpers |
 | `metadata-reviewer` | The five-agent review pipeline, combined with one provider extra: `openai`, `anthropic`, `azure`, or `ollama` |
+| `data-snapshot` | Figure and table images from PDFs |
+| `data-snapshot-metadata` | Validated Data Snapshot Metadata Schema v1.4 metadata from snapshot images through OpenRouter or OpenAI |
+
+For the Data Snapshot pipeline, combine `data-snapshot` and `data-snapshot-metadata`. These extras are absent from `ai4data==0.1.0`; until a release includes them, follow the [source installation instructions](src/ai4data/data_snapshot/README.md#installation). In this checkout, `all` includes DocLayout-YOLO and snapshot metadata extraction. TF-ID is installed separately; the [usage guide](src/ai4data/data_snapshot/README.md) describes its dependency requirements and the three commands.
 
 <details>
 <summary>Example: review a metadata record</summary>
@@ -330,7 +334,7 @@ Synthetic tabular and relational data generation is in a separate repository, [R
 
 ```
 ai4data/
-├── src/ai4data/          Python package: anomaly, data_use, discovery, metadata (augmentation, reviewer)
+├── src/ai4data/          Python package: anomaly, data_use, data_snapshot, discovery, metadata (augmentation, reviewer)
 ├── packages/ai4data/     JavaScript packages: @ai4data/core, @ai4data/search
 ├── reference/end_to_end/ Reference implementation with committed sample runs
 ├── cookbook/             Cookbook sources and their scripts

experimental/data-snapshot/README.md

@@ -1,5 +1,9 @@
 # data-snapshot
 
+This is the research and evaluation package. For the practical PDF-to-snapshot
+and metadata workflow, use [`ai4data.data_snapshot`](../../src/ai4data/data_snapshot/README.md).
+Keep this package's Python 3.10 environment separate from the production module.
+
 Data snapshot toolkit for PDF documents.
 
 This package provides the source code for [Benchmarking Open-Source Layout Detection Models for Data Snapshot Extraction from Institutional Documents](https://arxiv.org/abs/2606.06242).

pyproject.toml

@@ -153,7 +153,43 @@ ollama = [
   "autogen-ext[ollama]>=0.7.5",
 ]
 
-# All capabilities
+# Shared dependencies for metadata-only and alternative extractor installs.
+data-snapshot-core = [
+    "pillow>=11,<13",
+    "pydantic>=2.13.5,<3",
+    "pymupdf>=1.25,<2",
+    "python-dotenv>=1,<2",
+    "tqdm>=4.67,<5",
+]
+data-snapshot-metadata = [
+    "ai4data[data-snapshot-core]",
+    "openai>=2.53,<3",
+]
+# The default Data Snapshot extra bundles DocLayout-YOLO.
+data-snapshot = [
+    "ai4data[data-snapshot-core]",
+    "doclayout-yolo==0.0.4",
+    "huggingface-hub>=0.23,<2",
+]
+data-snapshot-yolo11 = [
+    "ai4data[data-snapshot-core]",
+    "huggingface-hub>=0.23,<2",
+    "ultralytics>=8.4,<9",
+]
+data-snapshot-yolo26 = [
+    "ai4data[data-snapshot-yolo11]",
+]
+# TF-ID uses an older Transformers stack; install separately (see module README).
+data-snapshot-tfid = [
+    "ai4data[data-snapshot-core]",
+    "accelerate>=1,<2",
+    "einops>=0.8,<1",
+    "huggingface-hub>=0.23.2,<1",
+    "timm>=1,<2",
+    "transformers==4.46.0",
+]
+
+# All capabilities with the default snapshot extractor; TF-ID stays opt-in.
 all = [
 	"ai4data[datause]",
 	"ai4data[harmonization]",
@@ -162,6 +198,7 @@ all = [
 	"ai4data[metadata]",
 	"ai4data[discovery]",
 	"ai4data[metadata-reviewer]",
+    "ai4data[data-snapshot,data-snapshot-metadata]",
 ]
 
 # Documentation (Docusaurus site in website/; notebooks are converted to Markdown

src/ai4data/README.md

@@ -26,6 +26,15 @@ Each capability is an optional extra.
 | `metadata` | Schema mapping, PDF-to-metadata extraction, and metadata augmentation |
 | `discovery` | NADA catalog access, metadata templates, and PDF helpers |
 | `metadata-reviewer` | The five-agent metadata review pipeline, combined with one provider extra: `openai`, `anthropic`, `azure`, or `ollama` |
+| `data-snapshot` | Figure and table images from PDFs |
+| `data-snapshot-metadata` | Validated Schema v1.4 metadata from snapshot images through OpenRouter or OpenAI |
+
+For the Data Snapshot pipeline, combine `data-snapshot` and
+`data-snapshot-metadata`. These extras are absent from `ai4data==0.1.0`; until a
+release includes them, install the checkout as described in the
+[Data Snapshot usage guide](https://github.com/worldbank/ai4data/blob/main/src/ai4data/data_snapshot/README.md).
+In this checkout, `all` includes DocLayout-YOLO and snapshot metadata extraction.
+TF-ID requires a separate environment; the guide lists its installation steps.
 
 ## Examples
 
@@ -53,6 +62,9 @@ issues = job.wait_sync(timeout=300)
 The documentation site describes each module: monitoring of data use,
 anomaly detection and explanation, data discoverability, metadata
 augmentation, and the Metadata Reviewer user manual.
+Data Snapshot documentation currently covers repository installation and usage;
+concepts, methods, worked examples, and user guides for the published site are
+planned in a separate PR.
 
 ## License
 

src/ai4data/__init__.py

@@ -14,6 +14,9 @@
 - discovery: NADA catalog, metadata templates, and related discovery helpers
   Install with: uv pip install ai4data[discovery]
 
+- data_snapshot: Figure/table snapshots and Schema v1.4 metadata from PDFs
+  Install with: python -m pip install "ai4data[data-snapshot,data-snapshot-metadata]"
+
 For all capabilities:
   uv pip install ai4data[all]
 """
@@ -37,11 +40,13 @@
 from . import anomaly
 from . import discovery
 from . import metadata
+from . import data_snapshot
 
 __all__ = [
     "__version__",
     "data_use",
     "anomaly",
     "discovery",
     "metadata",
+    "data_snapshot",
 ]