Analyst Report Document Processing Toolkit

Standalone pipeline that inventories a folder of downloaded analyst-report PDFs, parses their filenames into structured report fields, matches them back to a calendar-metadata extract, and detects duplicate reports.

Overview

A folder of exported research PDFs is opaque: filenames encode the real metadata, the same report often arrives more than once, and the calendar metadata lives in a separate table. This toolkit turns that raw folder into a clean, matched, de-duplicated inventory you can review — deterministically and with no external PDF library (page counts and text previews come from raw bytes and the system strings command, with a pure-Python fallback).

Quickstart

python setup_env.py                 # create .venv, install deps, write .env from .env.example
source .venv/bin/activate
python run_all.py                   # run the data/ then analysis/ stages in order
pytest                              # run the tests

The repo ships small synthetic sample PDFs under input/downloaded_reports_sample/pdfs/ and a sample calendar-metadata CSV, and defaults to RUNTIME_MODE=sample, so everything above runs fully offline. Point .env at your own PDF folder and metadata file to run on real data.

Stages

Run in order by run_all.py; each writes a Markdown log to logs/.

Script	Does	Writes to
data/d001_extract_pdf_inventory.py	Scan the PDF folder; record path, size, estimated pages, SHA256, and a text preview	output/data-output/
data/d002_clean_parse_reports.py	Parse filenames into date, ticker, contributor, title, and id; normalize them	output/data-output/
data/d003_match_reports_to_metadata.py	Match reports to calendar metadata by date + pages, scored on ticker/contributor/title	output/data-output/
data/d004_deduplicate_reports.py	Assign exact-hash and metadata-signature duplicate groups; flag which report to keep	output/data-output/
analysis/a001_processing_summaries.py	Build duplicate-group, processing-metric, and contributor-coverage summaries	output/analysis-output/

Layout

src/          shared code: settings (config + paths + schemas), io, logger, utils, validation
data/         d001–d004 data-processing scripts
analysis/     a001 summary script
input/        committed synthetic sample PDFs + calendar metadata (real data gitignored)
output/       data-output/ and analysis-output/ (gitignored)
logs/         one <script>.md per run
tests/        pytest (t001–t006)

How matching works

A cleaned report is matched to a calendar-metadata row only among candidates that share its date and page count. Within that candidate set, each candidate is scored: an exact ticker match, an exact-or-fuzzy contributor match, and an exact-or-fuzzy title match (fuzzy via difflib similarity, title threshold TITLE_SIMILARITY_THRESHOLD, default 0.72). The single highest-scoring candidate wins; reports with no candidate are written to the unmatched table.

How deduplication works

• Exact duplicates: identical SHA256 file hashes are grouped as EXACT-NNN; the first occurrence of each hash is kept (Keep Report = yes), later copies are not.
• Metadata duplicates: rows sharing a (date, contributor, title, pages) signature are grouped as META-NNN, catching likely-repeated reports even when the files differ.

Data & reproducibility

• Inputs: a folder of report PDFs (named <date>-<ticker>-<contributor>-<title>-<id>.pdf) and a calendar-metadata CSV (CALENDAR_METADATA_COLUMNS in src/settings.py).
• The committed samples are synthetic and internally consistent (they include an exact-duplicate pair, a metadata-duplicate pair, and one unmatched report); real inputs are gitignored.

Testing

pytest

Tests cover the PDF inventory helpers, filename parsing and cleaning, metadata matching (matched and unmatched paths), duplicate grouping and keep-flagging, the summary builders, and the validation guards.

License

Apache-2.0. See LICENSE and NOTICE.