Assess Denver, and freeze the Stage 1 inventories for replication (#96, #59)

[jonfroehlich]

First cities through the location-precision gate filed as #96, plus the frozen Stage 1 inputs the
ICCV paper should have shipped with. Analysis and tooling only — no pipeline change, no retrain, no
city sourced. Full suite 516 passed (was 304; +212 tests).

Denver is assessed and passes. Seattle is partially assessed and produced a methodological
correction instead of a verdict. Four things previously asserted in this repo are contradicted
below; every correction is recorded in place rather than quietly patched.

---

1. Denver is Good — median 0.29 m, 92.3% within 1 m

All 59 chips reviewed by Jon in one sitting, against the §5e rubric. Wilson intervals throughout,
because at n≈55 with a rate near 5% the normal approximation runs below zero.

	value	95% CI
Offset median	0.29 m	—
p75 / p90 / max	0.64 / 0.95 / 2.07 m	—
Within 1 m ("on the ramp")	92.3% (48/52)	81.8–97.0
Within 0.5 m	63.5% (33/52)	49.9–75.2
Phantom (readable corner, no ramp)	5.5% (3/55)	1.9–14.9
Unjudgeable	6.8% (4/59)	2.7–16.2

The threshold is stated rather than assumed, since the paper published none: Good = ≥90% within
1 m, phantom under 10%. A ramp is 1.2–1.8 m deep, so "within 1 m of centre" is very nearly "on the
concrete". This is our threshold, not the paper's — Denver's Table 1 peers cannot be re-scored
against it without repeating the exercise.

It also settles §5d for Denver. Zero chips where the reviewer saw more ramps than Denver
publishes; 1.37 ramps seen per corner against 1.21 records per corner. Denver's low pairing ratio is
ramp design vocabulary, not missing records — the benign reading, now confirmed on imagery.

Denver passing moves the usable pool from 276,615 to 349,385 — the single largest available
addition. It does not reach 500,000, and §6 shows nothing available does.

2. ⚠️ Denver is mid-pack, not weak, on per-ramp vs per-corner — I benchmarked against one city

inventory_geometry.py. NYC calibrates it: it publishes rampid and cornerid, so 6 m
single-link clustering can be scored against the publisher's own grouping — P 0.976 / R 0.973.
That is what licenses running it on cities with no corner key, and every remaining city gets it free
before any reviewer time.

I first called Denver "weak on pairing" by comparing it to NYC alone. NYC is the outlier:

City	Records	Rec/corner (6 m)	vs published corner key
NYC — in training	217,679	1.61	P .976 / R .973
Portland — in training	46,101	1.38	—
Seattle	38,364	1.34	—
Bend — in training	14,805	1.31	—
Sioux Falls	19,991	1.22	—
Denver	72,770	1.21	—
Arlington	10,342	1.12	—
Charlotte	40,600	1.09	P .621 / R .406 ⚠️
Minneapolis	18,453	1.09	P .924 / R .859
Boston	24,022	1.08	—
San Francisco	50,096	6.64	⚠️ broken

The real result is about the corpus, not the city: all three training cities sit above every
candidate, so any addition dilutes pairing density. And the mechanism is mostly benign — Sioux
Falls' Diagonal-only corners hold 1.004 records against Directional-only 1.365, so a low
ratio is largely design vocabulary, which is the diversity §8 asks for.

Never benchmark a candidate against one reference city. That was the error.

3. ⚠️ Seattle's "systematic shift" is not a registration error — and the statistic needed a null

Seattle's first eleven measurable chips read median 2.33 m with a mean offset vector of 2.06 m
against a mean magnitude of 2.37 m — an 87% systematic share. I reported that as a registration
error. The direction is real; the conclusion was not.

The share statistic had no null. It was documented as "~0 for noise", but under random headings
the mean vector shrinks only as 1/sqrt(n), so the expected share is ~0.9/sqrt(n):

city	n	share	null median	p
Seattle	11	87%	39%	0.0013
Denver	52	24%	16%	0.221

Seattle's lean survives its null — and Denver's 24% never meant anything, being inside its own.

A direction still doesn't say which side is wrong, so measure the other two legs of a triangle
that must close:

(ramps vs imagery)  =  (ramps vs centrelines)  +  (centrelines vs imagery)
 review sheet, n=11      n≈31,000                   n≈2,500

Leg 1 — inventory_centerline_offset.py (new). A ramp sits half a roadway from the centreline on
one side or the other, so median(east side) + median(west side) = twice the shift while the unknown
half-width cancels and returns as a sanity check.

	resultant shift	half-width E / N	samples
Seattle	0.00 m	6.24 / 6.26 m	31,430 / 30,809
Denver (control)	0.12 m	7.35 / 7.46 m	49,594 / 49,404

Denver is the calibration and it lands: 0.12 m here against 0.10 m from its 52 reviewer clicks,
measured independently. Flat across a 10°–30° sweep of the cardinal cutoff. Seattle's coordinates
are unbiased against Seattle's own street network — tighter than Denver's.

Leg 2 — verify_chip_georeference.py --sites-from-verdicts (new). Orthorectification error is
local, so a five-neighbourhood average cannot clear the places the verdicts came from. Measured
under the review itself, all eleven chips clear at ≤0.32 m, mean east +0.02 m — including the
8.79 m click, at 0.11 m.

The triangle predicts 0.0–0.3 m and the sheet says 2.06 m. It does not close, and that is the
result. No city-wide displacement can hide from 31,430 samples or a per-chip imagery check. The
sheet's own quartiles agree once you stop averaging: p25 is 0.47 m against a 2.33 m median, and a
uniform shift cannot produce a chip that is dead-on. What made 87% was a handful of large
per-record errors sharing a heading.

Consequently: the SHIFT framing is withdrawn; there is no constant to subtract, which was the
optimistic branch; Seattle's Poor rating is neither confirmed nor overturned, because eleven
chips on a basemap already declared inadequate for grading settle nothing; and the
62.9%-installed-after-2019 confound is now the leading candidate.

The durable fix: inventory_review_summary.py prints the null and p-value, refuses to call a shift
on its own, and names the two instruments that can answer it.

4. ⚠️ The obvious basemap is not good enough, and it fails silently

inventory_review_sheet.py renders the §5 positional instrument. The first version used Esri World
Imagery, which over Denver is leaf-on, hazy and effectively ~1 m, and at z=21 serves "Map data not
yet available" as a flat grey tile the fetcher pasted in as evidence. Denver's own
Aerial2016_tilecache is leaf-off and 0.057 m/px.

probe_basemap.py exists because the lesson kept recurring in new forms — King County advertises
maxLOD 23 and 404s above z20, so the deepest level must be probed, never read. Every King County
year is leaf-on at 27–41% vegetation cover against Denver's 7.2%, which is why Seattle's sheet
carries an explicit "adequate to size a large error, not to grade a Good city" caveat.

Every city needs its municipal basemap located and probed before its sheet is worth reviewer time.
Charlotte's is found and is good (leaf-off, 0.061 m/px); its sheet is built and unreviewed.

5. ⚠️ Two disqualifications and a supply correction, found by reading data — no imagery

• San Francisco is disqualified for Stage 1: 50,096 rows but only 7,553 distinct coordinates,
  1:1 with cnn (intersection node id) ⇒ coordinates are intersection centroids, ~4.7 identical
  labels per pixel. Separately, 14,414 rows are crexist=0 — confirmed absence, wrong polarity for
  Stage 1 and right for RampNet 2.0: measurement, condition inference, and tagging — we already collect the supervision and discard it at ingest #86.
• Charlotte's coordinates disagree with its own corner key (P .621 / R .406): of its multi-record
  published corners, 52.5% span >6 m, 14.4% >20 m, p99 308 m. Its sheet must test this.
• Charlotte has 5,505 RP_Type=NoRamp rows ⇒ usable 35,095, not 40,601.

Unassessed pool is 180,673, not ~236,000 (−55,327). So Good (276,615) + every unassessed city,
all passing, = 457,288 < 500,000: the route avoiding both the OK tier and the state DOTs is
closed.

Also fixed a silent truncation found the same way: SF names its OID field ObjectId, so ID-paging
stopped at 2,000 of 50,096 and wrote a normal-looking snapshot. find_oid_field now reads it from
the layer, and a short fetch refuses to write without --allow-partial.

6. ⚠️ Denver's "2022 imagery" claim does not survive contact with the data

96.2% of records are UPDATE_STATUS=NC, 74.4% carry a 2015 CREATEDATE, only 84 carry 2022, and
Modify is used zero times across 72,770 records. Either everything was re-verified against 2022
imagery or a 2015 delineation is being carried forward; zero M favours the second, which is a ~7-year
gap comparable to what disqualified DC. §5c's ✅ for Denver is now marked unconfirmed. Nothing in
the schema records removals either, so phantoms have no upper bound from this data.

Footprint passes: 85.2% strictly inside Denver County, 98.8% of the rest within 1 km, 128 records
(0.18%) beyond 2 km — the mountain parks.

7. Frozen Stage 1 inputs — data/inventories/

The three paper inventories were never in this repo; the README points at live portals serving current
data. Eleven snapshots are now frozen, each with endpoint, exact query, fetch date,
declared-vs-retained count and a sha256.

Snapshot	Records	vs paper Tab. 1
nyc-ny	217,679	−0.0005% — effectively frozen
portland-or	46,101	+1.7% drift
bend-or	14,805	+8.8% drift
+ Denver, Seattle, SF, Charlotte, Boston, Sioux Falls, Minneapolis, Arlington		candidates
+ seattle-wa-centerlines (34,484), denver-co-centerlines (7,866)		reference geometry, not supply

Two gzip header fields are pinned (mtime=0, filename="") so identical records hash identically — a
real bug the test caught, since otherwise the digest tracked where the file was written.

Centrelines are frozen on the same terms because an analysis that exonerates a city's coordinates must
not depend on a live endpoint for the reference it exonerated them against. Each city's centrelines
must come from the same publisher and CRS as its ramps — Seattle's SND and curb ramps are both
EPSG:2926 from org ZOyb2t4B0UYuYNYH via the same server, which is what makes a shared datum fault
cancel and a ramp-layer defect show.

Two limits stated rather than implied: these are not the paper's files (Portland and Bend have
drifted, so a re-run reproduces today's dataset, not ICCV's — recovering the paper-exact files from the
supplemental stays open), and basemap imagery is not redistributable, so verdicts.json records tile
URLs and keys instead of shipping tiles.

What this does not settle

• Recall is structurally invisible to this instrument — it samples published records, so it can
  only find records with no ramp, never ramps with no record. §5e says so explicitly.
• One rater, no second pass. The benchmark's standing top follow-up applies here too.
• Denver's phantom interval reaches 14.9%, which across 72,770 records is the difference between
  ~4,000 and ~10,800 false labels. Narrowing it needs a larger sample, not a better instrument.
• Our Good/OK/Poor scale is still a scale of one. NYC is the remaining route to placing it on the
  paper's, and its imagery service is dynamic-only — the /export fetcher is not built.

Review

• docs/curb_ramp_data_sourcing.md §5d–§5i and §9 carry the numbers and caveats.
• data/inventories/README.md documents the snapshot format and how to add a city.
• Everything is regenerable; the review sheet is not committed because it embeds municipal tiles.

🤖 Generated with Claude Code (claude-opus-5[1m])

[jonfroehlich]

Correction: Denver is mid-pack, not weak — and the unassessed supply is 23% smaller than counted.

Ran the geometry gate over all nine frozen inventories (pushed to PR #97). Two things I said in the
comment above are now wrong, and one of them is my error rather than new data.

⚠️ The Denver framing was wrong

I compared Denver against NYC alone. Against the full set, NYC is the outlier:

City	Records	Rec/corner (6 m)	Singleton	vs published corner key
NYC — Good, in training	217,679	1.61	0.402	P .976 / R .973
Portland — Good, in training	46,101	1.38	0.635	—
Bend — Good, in training	14,805	1.31	0.726	—
Sioux Falls	19,991	1.22	0.785	—
Denver	72,770	1.21	0.794	—
Arlington	10,342	1.12	0.887	—
Charlotte	40,600	1.09	0.917	P .621 / R .406 ⚠️
Minneapolis	18,453	1.09	0.917	P .924 / R .859
Boston	24,022	1.08	0.924	—
San Francisco	50,096	6.64	0.038	⚠️ see below

Denver is second-highest of the candidates. The actual result is about the corpus, not the city:
all three training cities sit above every candidate, so any addition dilutes pairing density
relative to what RampNet trains on today. Minneapolis's own published corner key confirms 1.09 is
real, so this is not a clustering artifact.

And the mechanism is mostly benign. The two cities publishing ramp type let it be decomposed:
Sioux Falls' Diagonal-only corners hold 1.004 records, Directional-only hold 1.365;
Charlotte agrees from the other side (1.059 with a diagonal type vs 1.176 without). A low ratio is
largely ramp-design vocabulary, not under-recording — which is the diversity §8 asks for, and
plausibly the thing Paterson and Gainesville punished us for lacking. The counter-risk is real
though: fewer paired examples to learn the pair separation #46 found us failing.

⚠️ San Francisco is disqualified for Stage 1 — no imagery needed

Its 50,096 records carry only 7,553 distinct coordinates, mapping 1:1 to cnn (SF's intersection
node id). Every ramp at an intersection is stamped with the intersection's point — modal 6–8 rows
per coordinate, up to 29. Stage 1 would project ~4.7 identical labels onto one pixel, none on a ramp.
Separately, 14,414 rows are crexist = 0 — confirmed absence, wrong polarity for Stage 1 (right
polarity for #86).

⚠️ Charlotte's coordinates disagree with Charlotte's own corner key

Recovery is P .621 / R .406, far below NYC and Minneapolis. Cause is spread: of its 3,839
multi-record published corners, 52.5% span >6 m, 14.4% >20 m, p99 = 308 m. Big arterial corners
explain ~30 m, not 308 m. Charlotte's review sheet must test this specifically.

Supply correction — and it closes a route

City	Listed in §3	Corrected	Why
San Francisco	50,096	0	intersection centroids
Charlotte	40,601	35,095	5,505 rows are RP_Type = NoRamp
others			live drift only
Total	~236,000	180,673	−55,327

Good (276,615) + every unassessed city, all passing, = 457,288 — still short of 500,000. The
route that avoided both the OK tier and the state DOTs is now closed: 500k requires accepting the
tier the paper rejected, or the state-DOT tail with its Richmond/NYC clipping hazards.

Both corrections came from reading the data, not reviewing imagery — the automated checks pay for
themselves before a reviewer is booked. Also fixed a silent fetch truncation found the same way: SF
names its OID field ObjectId, so paging stopped after 2,000 of 50,096 and wrote a normal-looking
snapshot. Short fetches now refuse to write.