Repository navigation
Library API service, sourcetool ingest, and place nicknames - #4
Merged
Merged
Conversation
The Library API (ruling R7): service/ is a small FastAPI + SQLite service for https://smartbrain.securecloudgroup.com/library/v1/ that takes votes and suggestions from the app, serves the pack manifest, and exposes a token-guarded queue to the operator. Suggestions run the same validate_record rules as CI plus sourcetool/submission.py (no parameter values, no credentials, public hosts only, known fields, size caps); trust fields are set server-side. No accounts, cookies or identifiers; client addresses live only in the in-memory rate limiter. Deploy files (systemd unit, Caddy snippet, env and manifest examples) and steps in service/README.md; not deployed. sourcetool ingest pulls the queue, re-checks every item, adds votes to records, writes new suggestions to sources/suggested/<date>.jsonl (tier harvested, origin user-suggestion), probes them, and clears the queue. check and build now read sources/suggested/. The schema also refuses private IP literals and user:pass@. Place nicknames: Wikidata short names and nicknames joined to Census places by GNIS id, plus a reviewed list (resolvers/place_nicknames.json); 69 places gain 108 aliases. A two-letter place alias counts only in capitals, and a state code that is also a nickname ("LA", "DC") names the place only when no other place is named. A code that names the ask's place is not a ticker ("NYC weather").
… prior, then id) Tied sources came back in arbitrary order, so the first source could differ between runs and between the Library and the app's port. Evals unchanged: dev 47/47, holdout 38/40, sealed 30/30.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Library API (ruling R7)
service/is a small FastAPI + SQLite service forhttps://smartbrain.securecloudgroup.com/library/v1/. It is not deployed;service/README.mdhas the steps.POST /votes{source_id, verdict: yes|no|broken, app_version}POST /suggestions{record, via: form|yes, app_version}: a new source, parameter values removedGET /manifest{tag, url, sha256, bytes}from a JSON file the operator updatesGET /healthzGET /admin/queue?since=,POST /admin/ackLIBRARY_ADMIN_TOKEN, constant-time compare)validate_recordas CI, plussourcetool/submission.py. That means no parameter values, no credential in the URL, headers oruser:pass@, public hosts only (private and loopback IPs refused), only known fields, and size caps. The client's tier, votes, validation, terms status, authority, origin and id are replaced server-side. The service never fetches a submitted URL.X-Forwarded-Foris trusted only from the proxy network.sblibraryuser), a Caddyhandleblock for the existing landing site, and examples for the env and manifest files. Caddy runs in thesb_caddycontainer, so the service binds thedocker0address rather than 127.0.0.1.sourcetool ingestvotes. Abrokenreport counts as a No and is listed.sources/suggested/<date>.jsonl(tierharvested, originuser-suggestion) and probes them. A suggestion of a source the Library already has counts as a Yes for that source.--dry-runfetches and reports without writing or clearing.--commitmakes a local branch and commit and never pushes.checkandbuildnow readsources/suggested/.Place nicknames
resolvers/place_nicknames.json. 69 places gain 108 aliases.Results
check: 8,631 records, 0 invalid.evalres: development 71/71 (11 nickname asks added), held-out 44/44. Sealed: 38/40, unchanged frommain.evallookup: development 47/47, held-out 38/40, sealed 30/30. Held-out and sealed are unchanged frommain.