Skip to content

Library API service, sourcetool ingest, and place nicknames - #4

Merged
SecureCloudGroup merged 2 commits into
mainfrom
service-and-nicknames
Sep 28, 2026
Merged

SecureCloudGroup merged 2 commits into
mainfrom
service-and-nicknames

Conversation

@SecureCloudGroup

Copy link
Copy Markdown
Owner

Library API (ruling R7)

service/ is a small FastAPI + SQLite service for https://smartbrain.securecloudgroup.com/library/v1/. It is not deployed; service/README.md has the steps.

Endpoint
POST /votes {source_id, verdict: yes|no|broken, app_version}
POST /suggestions {record, via: form|yes, app_version}: a new source, parameter values removed
GET /manifest {tag, url, sha256, bytes} from a JSON file the operator updates
GET /healthz
GET /admin/queue?since=, POST /admin/ack bearer token (LIBRARY_ADMIN_TOKEN, constant-time compare)
  • Checks: suggestions run the same validate_record as CI, plus sourcetool/submission.py. That means no parameter values, no credential in the URL, headers or user:pass@, public hosts only (private and loopback IPs refused), only known fields, and size caps. The client's tier, votes, validation, terms status, authority, origin and id are replaced server-side. The service never fetches a submitted URL.
  • Privacy: no accounts, cookies or identifiers. The queue holds the payload, the app version and the day. Client IPs exist only in the in-memory rate limiter (30 votes and 10 suggestions per hour). There is no access log.
  • Hardening:
    • 16 KB body cap, JSON-only POSTs, strict models;
    • no CORS, no docs endpoints, security headers;
    • a queue cap;
    • X-Forwarded-For is trusted only from the proxy network.
  • Deploy files: a hardened systemd unit (unprivileged sblibrary user), a Caddy handle block for the existing landing site, and examples for the env and manifest files. Caddy runs in the sb_caddy container, so the service binds the docker0 address rather than 127.0.0.1.

sourcetool ingest

  • Pulls the queue and checks every item again.
  • Adds votes to records' votes. A broken report counts as a No and is listed.
  • Writes new suggestions to sources/suggested/<date>.jsonl (tier harvested, origin user-suggestion) and probes them. A suggestion of a source the Library already has counts as a Yes for that source.
  • Clears the queue.
  • --dry-run fetches and reports without writing or clearing. --commit makes a local branch and commit and never pushes.
  • check and build now read sources/suggested/.

Place nicknames

  • Sources: Wikidata P1813/P1449 in English, joined to the Census place by GNIS id (P590 = gazetteer ANSICODE), plus the reviewed resolvers/place_nicknames.json. 69 places gain 108 aliases.
  • Matcher rules (documented in RESOLVERS.md):
    • A two-letter place alias counts only when typed in capitals.
    • A capitalised state code that is also a nickname ("LA", "DC") names the place only when no other place is named. "LA weather" is Los Angeles; "Lafayette LA" is Louisiana.
    • In lookup, a code that names the ask's place is not a ticker ("NYC weather" gets weather sources, "NYC stock" still gets stock sources).

Results

  • check: 8,631 records, 0 invalid.
  • Service + ingest tests: 53 passed. This PR adds them as a new CI job.
  • evalres: development 71/71 (11 nickname asks added), held-out 44/44. Sealed: 38/40, unchanged from main.
  • evallookup: development 47/47, held-out 38/40, sealed 30/30. Held-out and sealed are unchanged from main.

The Library API (ruling R7): service/ is a small FastAPI + SQLite service for
https://smartbrain.securecloudgroup.com/library/v1/ that takes votes and
suggestions from the app, serves the pack manifest, and exposes a token-guarded
queue to the operator. Suggestions run the same validate_record rules as CI plus
sourcetool/submission.py (no parameter values, no credentials, public hosts
only, known fields, size caps); trust fields are set server-side. No accounts,
cookies or identifiers; client addresses live only in the in-memory rate
limiter. Deploy files (systemd unit, Caddy snippet, env and manifest examples)
and steps in service/README.md; not deployed.

sourcetool ingest pulls the queue, re-checks every item, adds votes to records,
writes new suggestions to sources/suggested/<date>.jsonl (tier harvested, origin
user-suggestion), probes them, and clears the queue. check and build now read
sources/suggested/. The schema also refuses private IP literals and user:pass@.

Place nicknames: Wikidata short names and nicknames joined to Census places by
GNIS id, plus a reviewed list (resolvers/place_nicknames.json); 69 places gain
108 aliases. A two-letter place alias counts only in capitals, and a state code
that is also a nickname ("LA", "DC") names the place only when no other place
is named. A code that names the ask's place is not a ticker ("NYC weather").
… prior, then id)

Tied sources came back in arbitrary order, so the first source could
differ between runs and between the Library and the app's port. Evals
unchanged: dev 47/47, holdout 38/40, sealed 30/30.
@SecureCloudGroup
SecureCloudGroup merged commit 5acae25 into main Sep 28, 2026
3 checks passed
@SecureCloudGroup
SecureCloudGroup deleted the service-and-nicknames branch September 28, 2026 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant