Skip to content

feat: external events ingestion and normalization - #42

Merged
selvarajuv merged 1 commit into
mainfrom
external-factors-events
Oct 11, 2026
Merged

selvarajuv merged 1 commit into
mainfrom
external-factors-events

Conversation

@Alexcchip

@Alexcchip Alexcchip commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

#8

Type of change

Addition of ingestion from PredictHQ and AskNews APIs as well as a normalized form that all future events should follow

  • feat — new feature
  • fix — bug fix
  • docs — documentation only
  • refactor — code change that neither fixes a bug nor adds a feature
  • perf — performance improvement
  • test — adding or updating tests
  • chore / build / ci — tooling, deps, or pipeline changes

Changes

  • ingest/normalize.py — the shared contract. NormalizedEvent validates on construction; dedupe_key = normalized title + city slug + local start date. Category is excluded from the key on purpose: the sources disagree on it, and that must not split one event into two rows.
  • ingest/predicthq.py — structured listings near a point, paged by updated.gte. Requests deleted state explicitly so upstream cancellations propagate instead of inflating demand forever.
  • ingest/asknews.py + ingest/textdates.py — infers events from news articles: one scoped search per city, prose date resolution ("August 28-30, 2026", "Q3 2027") with the precision achieved recorded rather than rounded, and a named reject reason for everything dropped.
  • ingest/repository.py — upsert_event, source-agnostic. Resolves by (source, source_ref) before dedupe_key, because a revised start date changes the key and resolving by key first would orphan the original row. PredictHQ outranks AskNews on contested fields.
  • ingest/run.py — ingest CLI, with --dry-run to fetch and print without writing.
  • scripts/run_asknews.py — AskNews runner with the per-run cleaning report; replays recorded batches by default, --live calls the API.
  • docs/event-extraction.md — architecture, file inventory, real run output for both sources, and how to test.
  • Deps: adds httpx, promotes scikit-learn to a runtime dependency.
  • Schema: events (one merged row per occurrence) and event_observations (append-only, one row per source sighting, raw payload kept verbatim).

Database migrations

  • Includes an Alembic migration (just migrate-create "...")
  • Migration runs cleanly (just migrate) and rolls back (just migrate-down)

Checklist

  • Commits follow Conventional Commits
  • Lint passes (just lint) and code is formatted (just format)
  • Pre-commit hooks run clean (just hooks installed)
  • Updated docs / README / .env.example where relevant
  • Tested locally (just dev) and verified the API at http://localhost:8000/docs

How to test

  1. Look at How to test it section of docs/event-extraction.md
  2. This uses API credits!
    cd backend && uv run python scripts/run_asknews.py --dry-run

Notes for reviewers

At the end I added a .commit() to DB to test but this should be removed when we move to an API

Check to see if the normalized form is going to fit what we need

@Alexcchip
Alexcchip force-pushed the external-factors-events branch 2 times, most recently from eb7be70 to d147226 Compare October 7, 2026 22:31
@Alexcchip Alexcchip changed the title External factors events feat: external events ingestion and normalization Oct 7, 2026
@Alexcchip
Alexcchip force-pushed the external-factors-events branch from a6cab78 to 414479a Compare October 7, 2026 22:45
@selvarajuv selvarajuv linked an issue Oct 8, 2026 that may be closed by this pull request
@selvarajuv
selvarajuv force-pushed the external-factors-events branch 3 times, most recently from d4ab031 to 17bf7ff Compare October 11, 2026 00:10
@selvarajuv
selvarajuv force-pushed the external-factors-events branch from 17bf7ff to 11dc877 Compare October 11, 2026 00:21
@selvarajuv
selvarajuv merged commit 3093cac into main Oct 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

External factor integration: Events

3 participants