Skip to content

README.md

Latest commit

 

History

History
121 lines (88 loc) · 9.16 KB

README.md

File metadata and controls

121 lines (88 loc) · 9.16 KB
Raw

Local OCR benchmark

The committed release-qualification spec, expected rows, manifest, and nine PNGs contain synthetic examples generated by render_fixtures.js. The renderer reads the spec, not personal photos. Original user screenshots, their ground truth, and runtime reports stay in ignored local files and are not needed for this suite.

Large-image regression (September 27, 2026)

The release qualification suite now includes 1260 × 2736 iPhone screenshots with units and repeated stops, a tall no-address image, and a 1600 × 1600 no-address image. These exercise EasyOCR's bounded detector canvas. Generate them with node output/ocr-benchmark/render_fixtures.js --spec output/ocr-benchmark/release-qualification-spec.json --outdir output/ocr-benchmark/fixtures/release-qualification, then run the final-image validator with manifests/release-qualification-manifest.json and release-qualification-expected.json. Paths to those two files are relative to this benchmark directory; validator arguments are relative to the repo root.

probe-memory.py is an offline diagnostic for the application container. It prints engine timings, line counts, and cgroup peak memory without printing OCR text. The strict Java runner remains the release acceptance check.

The benchmark runs the production AddressOcrService.extractAddressCandidates path against screenshot manifests. It compares ordered, normalized address strings without correcting OCR characters, and keeps repeated addresses as repeated rows.

For scoring comparisons only, ASCII apostrophe U+0027 and curly single quotation marks U+2018 and U+2019 normalize to the same ASCII apostrophe. Reports retain the literal OCR text. No letter or digit substitutions are applied, so O and 0, and 330 and 350, remain distinct.

Original screenshots

Correction: The earlier 27/28-row and 6/7-case counts below used an incorrect APT 350 expectation. Independent review against corrected truth found the current original Java candidate matches all 7 screenshot cases and all 28 visible rows, including APT 330 and S APT 3B. The preserved pre-correction results/original-java-baseline.json and prior direct-Tesseract measurements are historical artifacts evaluated against the wrong truth, not current accuracy reports.

manifests/original-only.json covers seven checked-in route screenshots. Their manually transcribed truth is in expected-addresses.json: 28 visible address rows, including one duplicate from an overlapping scroll viewport.

The historical Java and direct-Tesseract reports used an incorrect APT 350 label. The user confirmed APT 330; independent transcription verified that the historical Java output matches all 28 corrected rows in all seven images.

The historical reports remain for traceability and must not be cited as an OCR failure on that apartment number.

Synthetic training suite

train-spec.json is the source for 12 deterministic, synthetic screenshots. It covers directional and single-digit numbers, fractions, hyphenated numbers, highway and mile roads, unit labels, ZIP/locality text, apostrophes, wrapped long addresses, duplicate ordering, numeral distractors, top and bottom crops, and a no-address rejection case. Every case carries an accept, review, or reject label plus its expected warnings and ordered rows.

training-expected.json maps case IDs to expected visible addresses. manifests/train.json supplies the images and labels to the Java runner. These fixtures and labels are frozen: do not regenerate them to make a candidate pass. New coverage belongs in a separately named spec, manifest, and expected file, followed by visual label review before OCR runs.

The historical single-engine run in results/train-report.json matched 17 of 37 visible rows, with 3 false positives and 20 false negatives. It lacks the current consensus warning metadata. Candidate v10's three-engine run is results/final-training-v10.json: all 12 screenshots and 37 ordered rows pass, with zero unit or duplicate false positives/negatives.

The Java benchmark runner uses strict OCR output parsing and requires a compatible three-engine consensus wrapper; plain Tesseract TSV is rejected. For a host-side run, set OCR_STRICT_WRAPPER to the path of that wrapper. The final-image helper uses /app/ocr/run inside the image.

From the repository root, run the Java service scorer against the frozen images:

python3 output/ocr-benchmark/benchmark.py run \
  --tesseract "$OCR_STRICT_WRAPPER" \
  --manifest output/ocr-benchmark/manifests/train.json \
  --expected output/ocr-benchmark/training-expected.json \
  --report output/ocr-benchmark/results/train-report.json \
  --fail-on-mismatch

If the Maven test classpath cannot be built in the current worktree, provide an existing dependency classpath file with --classpath-file PATH. --tesseract PATH selects the strict consensus wrapper executable and --language LANG selects the language used by the service. The scorer marks OCR consensus metadata unavailable for historical runner outputs that lack the strict schema-v1 engine contract and candidate metadata; it does not report missing historical metadata as zero consensus.

The runtime p95 is the nearest-rank percentile, using the ceiling rank for the case count.

The synthetic suite is a development set, not an independent accuracy estimate. Keep held-out specifications and labels separate from OCR implementation work; report their result only after the implementation is frozen.

Frozen acceptance sets

Set Manifest under manifests/ Expected file Cases / complete rows
Original screenshots original-only.json Omit --expected; uses filename mapping 7 / 28
Training train.json training-expected.json 12 / 37
Development 1 heldout.json heldout-expected.json 12 / 29
Development 2 final-holdout-manifest.json final-holdout-expected.json 11 / 27
Qualification regression qualification-manifest.json qualification-expected.json 8 / 20
Untouched release qualification release-qualification-manifest.json release-qualification-expected.json 6 / 16

The last set was visually labeled independently, then first evaluated against frozen v10. Earlier holdouts became development sets when their failures informed fixes. All reports and failed candidates remain available. The aggregate has 56 screenshots and 157 complete expected rows. See the release record for accepted results and deployment identity.

Final application image check

After building the final application image and a clean target/test-classes, run the same test-only runner against the image's /app/app.jar:

python3 output/ocr-benchmark/validate-final-image.py \
  --image "$FINAL_IMAGE" \
  --tesseract /app/ocr/run \
  --manifest output/ocr-benchmark/manifests/original-only.json \
  --report output/ocr-benchmark/results/final-image-original.json

python3 output/ocr-benchmark/validate-final-image.py \
  --image "$FINAL_IMAGE" \
  --tesseract /app/ocr/run \
  --manifest output/ocr-benchmark/manifests/train.json \
  --expected output/ocr-benchmark/training-expected.json \
  --report output/ocr-benchmark/results/final-image-train.json \
  --fail-on-mismatch

python3 output/ocr-benchmark/validate-final-image.py \
  --image "$FINAL_IMAGE" \
  --tesseract /app/ocr/run \
  --manifest output/ocr-benchmark/manifests/heldout.json \
  --expected output/ocr-benchmark/heldout-expected.json \
  --report output/ocr-benchmark/results/final-image-heldout.json

The helper runs Linux/amd64 with network access disabled, UID/GID 10001, a read-only root filesystem, a 128 MiB /tmp, a two-CPU limit, and a 2 GiB memory limit. It mounts only the benchmark directory, src/test/resources, and target/test-classes from the worktree, all read-only. The raw runner JSON is written to a unique temporary Docker volume mounted at the image-owned /app/cache, copied to the host beside the report, and the helper removes only its labeled temporary container and volume. Container output is kept separately in a .container.log file.

The scorer is imported from benchmark.py, so the report uses the existing scoring and regression logic. The raw runner file ends in .runner.json; --report must name a new file. Held-out runs require --expected and should be started only after OCR changes are frozen.

Strict acceptance is the default. It requires exact ordered rows, zero OCR runtime errors, the engine contract, observed evidence from all three engines across positive cases, expected review flags, and zero unit/duplicate false positives or negatives. Missing detections may abstain; no synthetic vote is created. Use --diagnostic only for investigation, never release acceptance. Verify the gate itself with python3 -B output/ocr-benchmark/self_check.py.

Run at most two OCR containers concurrently on the development host; do not build or export Docker images during inference. Four concurrent v9 suites caused timeouts and exit-137 failures. Keep the 45-second OCR timeout and 2 GiB memory gate unchanged. The helper binds runs to an immutable local image ID, records per-case timings, and captures OCR_MEMORY_PEAK_BYTES in each container log.