Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
ad50076
answers: record field, schema, sample check, build merge, answers-che…
SecureCloudGroup Sep 29, 2026
775ff25
answers: generated files for 43 curated feeds and 31 FRED series
SecureCloudGroup Sep 29, 2026
774353d
answers: spec v1.1 — {param} path segments, date type, list row filte…
SecureCloudGroup Sep 29, 2026
034cd5e
answers: top-level lists checked as {items}, RFC 2822 dates are times…
SecureCloudGroup Sep 29, 2026
f4a36e0
Fix curated source records from the answers-authoring evidence
SecureCloudGroup Sep 29, 2026
425430e
eia-petroleum-stocks: unvalidated until the DEMO_KEY rate limit clear…
SecureCloudGroup Sep 29, 2026
747c490
Merge remote-tracking branch 'origin/record-fixes-0929' into library-…
SecureCloudGroup Sep 29, 2026
1a22237
answers: the authored answers for curated JSON/text sources; list-of-…
SecureCloudGroup Sep 29, 2026
bb4a7d6
answers: Open-Meteo daily dates typed as dates
SecureCloudGroup Sep 29, 2026
c4c2ab8
Sports sources after ESPN's refusal: TheSportsDB league and team reco…
SecureCloudGroup Sep 29, 2026
51d5b01
Data fixes from the live e2e run: local sun times, real FRED/feed lab…
SecureCloudGroup Sep 29, 2026
637b3dd
Merge remote-tracking branch 'origin/sports-sources' into library-ans…
SecureCloudGroup Sep 29, 2026
d1d3947
Merge remote-tracking branch 'origin/data-fixes-0929' into library-an…
SecureCloudGroup Sep 29, 2026
9b6bb51
Live-run fixes: temperature words, final-score filter, drought as CSV…
SecureCloudGroup Sep 29, 2026
29fecd8
mlb-team-results: a cancelled game isn't a result (filter on the deta…
SecureCloudGroup Sep 29, 2026
c4c50fa
answers: utc marks a time whose zoneless values are UTC (TheSportsDB …
SecureCloudGroup Sep 29, 2026
bbb2b84
docs: fixed headers, fixed-value filters, utc times
SecureCloudGroup Sep 29, 2026
dbccfa8
answers: a midnight timestamp may be a date; FEMA declarations and CP…
SecureCloudGroup Sep 29, 2026
d007cb2
answers: Hacker News rows read "688 points · 456 comments" without ra…
SecureCloudGroup Sep 29, 2026
d7f4df5
Markets and tech data gaps: metals, software releases, launch filters…
SecureCloudGroup Sep 29, 2026
b05c1c7
Local news, transit advisories, state gas prices, area names, Kp fore…
SecureCloudGroup Sep 29, 2026
62980c0
Merge local-transit-0929: local news, transit advisories, state gas p…
SecureCloudGroup Sep 29, 2026
f85c528
resolvers: reviewed area nicknames are listed on their place (attrs.n…
SecureCloudGroup Sep 29, 2026
4bdf469
answers: quoted keys in paths; BART advisories read as JSON
SecureCloudGroup Sep 29, 2026
ea73db0
mta: record the operator's terms ruling (each person's app fetches fo…
SecureCloudGroup Sep 29, 2026
e29bc3f
crypto entries carry their ticker symbol; Coinbase fills BTC-USD from it
SecureCloudGroup Sep 29, 2026
da3b256
sourcetool: answers spec v1.2 (window, measure, axis, tbd_if, result …
SecureCloudGroup Oct 4, 2026
8b8f1de
taxonomy and resolvers: lottery, geysers, place coast flags, nickname…
SecureCloudGroup Oct 4, 2026
6027b5f
records: answers backfilled to v1.2; new sources for lottery, geysers…
SecureCloudGroup Oct 4, 2026
7c588b7
asks: example asks per source (10) and per route (12) for locate v2
SecureCloudGroup Oct 4, 2026
553c828
asks: no example ask overlaps an evaluation set; a gate keeps it that…
SecureCloudGroup Oct 4, 2026
dc9fc77
check: a resolver value filled into the host must be a plain host; de…
SecureCloudGroup Oct 4, 2026
26d0917
answers: swpc-kp times are UTC; a lint keeps one provider's time form…
SecureCloudGroup Oct 4, 2026
4ba945b
validation: bart-advisories probed ok as JSON from a residential conn…
SecureCloudGroup Oct 4, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
6 changes: 4 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,11 @@ jobs:
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install "httpx==0.28.1" "duckdb==1.5.4"
- name: schema check (every record)
- run: pip install "httpx==0.28.1" "duckdb==1.5.4" "pytest==9.1.1"
- name: schema check (every record + every answers/ file; no live fetch)
run: python -m sourcetool check
- name: answers rules (schema, the sample check, build merge; offline)
run: python -m pytest tests/test_answers.py -q
- name: probe sources changed in this PR (fail on regressions only)
if: github.event_name == 'pull_request'
run: |
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ python3 -m venv .venv && .venv/bin/pip install httpx duckdb
.venv/bin/python -m sourcetool build # build/library.duckdb (+ search index)
.venv/bin/python -m sourcetool lookup "tides for Charleston Harbor today"
.venv/bin/python -m sourcetool coverage # sources per subcategory, empty ones listed
.venv/bin/python -m sourcetool answers-check # fetch each answers/ sample and verify its paths (live)
```

| Path | What |
Expand All @@ -37,6 +38,7 @@ python3 -m venv .venv && .venv/bin/pip install httpx duckdb
| `sources/harvested/` | records harvested from open catalogs (labelled, ranked lower) |
| `sources/suggested/` | users' suggestions, written by `sourcetool ingest` |
| `providers/` | provider records |
| `answers/` | which response paths answer which questions, per curated source (merged into records by `build`) |
| `resolvers/` | lists that turn words into parameters (status pages, stations…) |
| `sourcetool/` | the tool |
| `service/` | the Library API the app submits votes and suggestions to ([README](service/README.md)) |
Expand Down
16 changes: 16 additions & 0 deletions answers/_skipped.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Sources without answers

One line per source: id — why no answers file was written.

- nyt-sports — the feed answers HTTP 200 with zero items (2026-09-28); nothing to verify a list answer against.
- politifact — the feed starts with blank lines before `<?xml`; the app's feed parser refuses it ("XML or text declaration not at start of entity"), so no card can read it until the parser strips leading whitespace.
- powerball-home — an html page (the jackpot, cash value and next draw are page text); the page door reads it, and its readings are in the record's examples.

## Held back (records not published)

One line per record the 2026-09-29 round wanted but did not publish: id — why.

- subreddit-top-rss — reddit.com robots.txt disallows every path for every agent (`Disallow: /`); a hard refusal excludes the source.
- aws-health-currentevents — the AWS site terms bar robots and data mining, and the Health Dashboard documents no public API; terms unclear, not published.
- megamillions-home — the jackpot is filled by script from a POST service (no readable page text), and the site publishes no terms of use to check; Mega Millions numbers come from data.ny.gov (ny-lottery-megamillions) instead.
- thesportsdb-league-round — a round's results for one team need an any-of row filter (home OR away team), which answers spec v1.2 does not have; waits on that schema.
34 changes: 34 additions & 0 deletions answers/abc-top.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"source_id": "abc-top",
"answers": [
{
"name": "latest",
"label": "Headlines",
"kind": "list",
"primary": true,
"words": [
"headlines",
"news",
"breaking",
"top stories",
"latest"
],
"path": "items",
"row": [
{
"path": "title",
"label": "Title",
"type": "text"
},
{
"path": "published",
"label": "Published",
"type": "time"
}
],
"may_be_empty": false
}
],
"sample_url": "https://abcnews.go.com/abcnews/topstories",
"checked": "2026-09-28"
}
66 changes: 66 additions & 0 deletions answers/adsblol-callsign.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
{
"source_id": "adsblol-callsign",
"answers": [
{
"name": "aircraft",
"label": "Flight position",
"words": [
"where",
"position",
"track",
"flight",
"altitude",
"speed",
"location",
"map",
"latitude",
"longitude"
],
"primary": true,
"kind": "list",
"path": "ac",
"row": [
{
"path": "alt_baro",
"label": "Altitude",
"type": "number",
"unit": "ft"
},
{
"path": "gs",
"label": "Ground speed",
"type": "number",
"unit": "kt"
},
{
"path": "lat",
"label": "Latitude",
"type": "number"
},
{
"path": "lon",
"label": "Longitude",
"type": "number"
}
],
"may_be_empty": true
},
{
"name": "updated",
"label": "Last update",
"words": [
"updated",
"last seen",
"when",
"time"
],
"primary": false,
"kind": "value",
"path": "now",
"type": "time",
"window": "now"
}
],
"sample_url": "https://api.adsb.lol/v2/callsign/DAL1234",
"checked": "2026-09-28"
}
132 changes: 132 additions & 0 deletions answers/amtraker-trains.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
{
"source_id": "amtraker-trains",
"answers": [
{
"name": "stops",
"label": "Stops: scheduled vs actual",
"words": [
"late",
"on time",
"delayed",
"running late",
"arrival",
"eta",
"when",
"schedule",
"stops",
"stations"
],
"primary": true,
"kind": "list",
"path": "{train}[0].stations",
"row": [
{
"path": "name",
"label": "Station",
"type": "text"
},
{
"path": "schArr",
"label": "Scheduled",
"type": "time"
},
{
"path": "arr",
"label": "Actual / expected",
"type": "time"
},
{
"path": "status",
"label": "Status",
"type": "text"
}
],
"axis": {
"cell": "schArr",
"step": "hour"
}
},
{
"name": "route",
"label": "Route",
"words": [
"route",
"which train",
"name",
"line"
],
"primary": false,
"kind": "value",
"path": "{train}[0].routeName",
"type": "text",
"window": "now"
},
{
"name": "next_station",
"label": "Next / current station",
"words": [
"where",
"next stop",
"next station",
"location",
"now"
],
"primary": false,
"kind": "value",
"path": "{train}[0].eventName",
"type": "text",
"window": "latest"
},
{
"name": "state",
"label": "Train state",
"words": [
"status",
"running",
"active",
"departed",
"completed",
"state"
],
"primary": false,
"kind": "value",
"path": "{train}[0].trainState",
"type": "text",
"window": "now"
},
{
"name": "speed",
"label": "Speed",
"words": [
"speed",
"how fast",
"moving",
"velocity",
"mph"
],
"primary": false,
"kind": "value",
"path": "{train}[0].velocity",
"type": "number",
"unit": "mph",
"window": "now"
},
{
"name": "updated",
"label": "Last position update",
"words": [
"updated",
"last update",
"as of",
"last seen"
],
"primary": false,
"kind": "value",
"path": "{train}[0].lastValTS",
"type": "time",
"window": "now"
}
],
"sample_url": "https://api-v3.amtraker.com/v3/trains/5",
"checked": "2026-09-28"
}
73 changes: 73 additions & 0 deletions answers/apple-top-albums.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
{
"source_id": "apple-top-albums",
"answers": [
{
"name": "chart",
"label": "Top albums",
"words": [
"top albums",
"albums",
"music",
"top",
"chart",
"charts",
"most popular",
"top 25",
"best"
],
"primary": true,
"kind": "list",
"path": "feed.results",
"row": [
{
"path": "name",
"label": "Album",
"type": "text"
},
{
"path": "artistName",
"label": "Artist",
"type": "text"
},
{
"path": "releaseDate",
"label": "Released",
"type": "date"
}
]
},
{
"name": "number_one",
"label": "Number one",
"words": [
"number one",
"#1",
"top one",
"no. 1",
"at the top",
"first place"
],
"primary": false,
"kind": "value",
"path": "feed.results[0].name",
"type": "text",
"window": "latest"
},
{
"name": "number_one_by",
"label": "Number one artist",
"words": [
"who is number one",
"#1 by",
"top artist"
],
"primary": false,
"kind": "value",
"path": "feed.results[0].artistName",
"type": "text",
"window": "latest"
}
],
"sample_url": "https://rss.marketingtools.apple.com/api/v2/us/music/most-played/25/albums.json",
"checked": "2026-09-28"
}
Loading
Loading