Library API service, sourcetool ingest, and place nicknames

.github/workflows/ci.yml

@@ -57,6 +57,16 @@ jobs:
       - name: "source lookup (development + held-out; the first source offered must be right)"
         run: |
           python -m sourcetool evallookup lookup_asks lookup_holdout || echo "::warning::lookup below 100%, see output"
+  service:
+    name: Library API + ingest tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install -r service/requirements-test.txt
+      - run: python -m pytest service/tests tests/test_ingest.py -q
   revalidate:
     name: weekly re-validation (report only)
     if: github.event_name == 'schedule'

.gitignore

@@ -4,3 +4,5 @@ build/
 __pycache__/
 *.pyc
 .DS_Store
+.venv-service/
+service/data/

README.md

@@ -35,9 +35,11 @@ python3 -m venv .venv && .venv/bin/pip install httpx duckdb
 | `taxonomy/` | categories, subcategories, question kinds, parameter kinds |
 | `sources/curated/` | hand-curated records (JSONL, one per line) |
 | `sources/harvested/` | records harvested from open catalogs (labelled, ranked lower) |
+| `sources/suggested/` | users' suggestions, written by `sourcetool ingest` |
 | `providers/` | provider records |
 | `resolvers/` | lists that turn words into parameters (status pages, stations…) |
 | `sourcetool/` | the tool |
+| `service/` | the Library API the app submits votes and suggestions to ([README](service/README.md)) |
 | `docs/` | [PLAN](docs/PLAN.md), [POLICY](docs/POLICY.md), [SCHEMA](docs/SCHEMA.md) |
 
 Only the maintainer merges changes to `main`. Changes from SmartBrain users arrive through the Library API, and are then reviewed and opened as PRs by the maintainer's tooling.

docs/PLAN.md

@@ -68,7 +68,7 @@ These are the Library records (`docs/SCHEMA.md`).
 | `build` | Compile `build/library.duckdb`: sources, categories, taxonomy and a term index for scoring. |
 | `lookup "words"` | Look up a phrase against the build (the same SQL the app runs). |
 | `coverage` | Count sources per subcategory and list the empty ones. |
-| `ingest` (phase 2) | Pull the API submission queue, validate, aggregate votes, write records, and open a PR for the operator. |
+| `ingest` | Pull the API submission queue, check it again, aggregate votes, write suggestions to `sources/suggested/`, probe them, and clear the queue. The operator opens the PR (`--commit` makes the branch and commit). |
 
 `sourcetool` writes only to a checkout. Publishing is always a PR that the operator merges.
 
@@ -90,16 +90,18 @@ No and "broken" reports go the same way, as votes against a source.
 
 This is a small service on the existing VPS (`smartbrain.securecloudgroup.com`) next to the landing page and broker. It reuses the VPS deploy and TLS.
 
+The service is `service/` in this repo (`service/README.md` has the deploy steps).
+
 | Endpoint | Purpose |
 |---|---|
-| `POST /library/v1/votes` | `{source_id, verdict: yes\|no\|broken}` for known sources |
-| `POST /library/v1/suggestions` | A new source record (from the Library form or a Yes on a web source) |
-| `GET /library/v1/manifest` | The latest signed pack: seq, sha256, size, url |
-| `GET /library/v1/pack/{seq}` | The signed pack (the DuckDB file, or JSONL plus the build step in the app) |
+| `POST /library/v1/votes` | `{source_id, verdict: yes\|no\|broken, app_version}` for known sources |
+| `POST /library/v1/suggestions` | `{record, via: form\|yes, app_version}`: a new source record (from the Library form or a Yes on a web source) |
+| `GET /library/v1/manifest` | The latest pack: tag, url, sha256, bytes (the pack itself stays on the GitHub release, R12) |
+| `GET /library/v1/admin/queue`, `POST /library/v1/admin/ack` | The operator's `sourcetool ingest` (bearer token) |
 
 Security:
 - Anonymous: no account, no install id, no cookies.
-- Rate-limited per IP, with a small proof-of-work on submissions.
+- Rate-limited per IP in memory only (30 votes and 10 suggestions per hour). A proof-of-work is not built yet.
 - Size caps.
 - Every payload runs through the same `validate_record` rules as CI: https, public host, no credentials, known categories.
 - The service **never fetches** a submitted URL; `sourcetool ingest` validates on the operator's side.
@@ -134,7 +136,7 @@ Flow: the queue feeds `sourcetool ingest` on the operator's side. It validates,
 | A | Taxonomy v1, schema, `sourcetool` check, validate, harvest, build and lookup; curated US core; 7 harvesters; DuckDB build | **done** |
 | B | Repo public and protected; CI (schema check plus validation of changed records); scheduled re-validation | this change |
 | C | App: signed pack install, DuckDB tables, lookup in the NI finder, Yes recorded locally, the Library page and form | next (SmartBrain_3000 PR) |
-| D | The Library API on the VPS, `sourcetool ingest`, vote aggregation, and the Yes vote plus suggestions from the app | after C |
+| D | The Library API on the VPS, `sourcetool ingest`, vote aggregation, and the Yes vote plus suggestions from the app | service and ingest built; VPS deploy and the app's calls next |
 | E | Model-drafted classification (R3) to fix keyword misplacements; more harvesters (state 511 and GTFS registries, NWS stations, NOAA stations as resolvers) | rolling |
 
 ## Known gaps (v1)

docs/RESOLVERS.md

@@ -10,7 +10,7 @@ A resolver is a table of entries, one per line in `resolvers/<name>.jsonl`, all
 
 | Resolver | Entries | Open source |
 |---|---|---|
-| place | 32,333 | Census gazetteer plus population estimates |
+| place | 32,333 | Census gazetteer plus population estimates; nicknames from Wikidata and `resolvers/place_nicknames.json` |
 | zip | 33,791 | Census ZIP code areas (ZCTAs) |
 | county | 3,222 | Census gazetteer |
 | us_state | 56 | fixed list |
@@ -33,6 +33,17 @@ The single matcher is `sourcetool/resolve.py`. It works in three ways:
 - **`by_name(many=True)`** returns every distinct thing named, in order: a currency pair, or "compare X and Y".
 - **`near`** returns the nearest entries within the policy's distance. When a near rival differs in what it measures (`differ_on`, such as a different body of water or river), the result is *ambiguous*, which means **ask the user**. When nothing is in range, it's an honest *none*.
 
+### Place nicknames
+People say "NYC", "LA", "Philly", "Vegas", "NOLA", "Chi-town" or "the Big Apple". These are extra aliases on the place:
+- **Wikidata**: English short names (P1813) and nicknames (P1449) of items with a GNIS id (P590), joined to the Census place by the gazetteer's GNIS code (ANSICODE). The join keeps only items that are a Census place. A nickname is kept when it is a code typed in capitals ("NYC", "L.A."), a single word of five or more letters ("Philly"), or a phrase of up to three words ("Mile High City"); slogans, anything with a digit, short everyday words ("Jeff", "The Hub") and state names are dropped. A Wikidata nickname never shadows another place's own name ("Frisco" stays Frisco, Texas).
+- **Reviewed list** `resolvers/place_nicknames.json`: the common ones Wikidata lacks (it has none for Chicago), plus an `exclude` list for Wikidata nicknames we refuse.
+
+Two matcher rules keep short nicknames honest:
+- A **two-letter** place alias counts only when typed in capitals: "LA weather" is Los Angeles, the word "la" never is.
+- A capitalised **state code that is also a place nickname** ("LA", "DC") names that place only when no other place is named. "LA weather" is Los Angeles; "Lafayette LA" is Lafayette, Louisiana, because another place is named and the code is its state.
+
+In source lookup, a code typed in capitals that names the ask's place ("NYC weather") is the place, not a ticker, unless the ask has a ticker cue ("NYC stock").
+
 Every result is `resolved`, `ambiguous` (the card asks one question, offering the candidates) or `none`. Nothing is guessed silently.
 
 ## Fills: how every source parameter gets its value
@@ -74,15 +85,15 @@ Resolver sets:
 
 | Set | Size | Result |
 |---|---|---|
-| Development | 60 asks | 100% |
+| Development | 71 asks (11 nickname asks added) | 100% |
 | Held-out | 44 asks | 82% on its first run, then 100% after general fixes |
 | Sealed | 40 asks, run once | **90%** |
 
 Lookup sets (top-1):
 
 | Set | Size | Result |
 |---|---|---|
-| Development | 45 asks | 100% |
+| Development | 47 asks | 100% |
 | Held-out | 40 asks | 85% on its first run, then 95% after general fixes |
 | Sealed | 30 asks, run once | **100%** |
 

docs/SCHEMA.md

@@ -1,6 +1,6 @@
 # Source record schema (v1)
 
-Each source is one JSON object per line in `sources/**/*.jsonl`. `sourcetool/schema.py` holds the rules, and CI, `sourcetool` and the app all run the same validator.
+Each source is one JSON object per line in `sources/**/*.jsonl`. Users' suggestions arrive through the Library API and `sourcetool ingest` writes them to `sources/suggested/<date>.jsonl` with tier `harvested` and origin `{by: "user-suggestion"}`. `sourcetool/schema.py` holds the rules, and CI, `sourcetool` and the app all run the same validator.
 
 | Field | Meaning |
 |---|---|
@@ -29,8 +29,8 @@ Each source is one JSON object per line in `sources/**/*.jsonl`. `sourcetool/sch
 
 Rules that reject a record:
 - non-https URLs;
-- private or `.local` hosts;
-- a credential embedded in the URL;
+- private or `.local` hosts, and private, loopback or link-local IP addresses;
+- a credential embedded in the URL (a key parameter or `user:password@`);
 - unknown categories, kinds or parameter kinds;
 - undeclared `{params}`;
 - duplicate ids.