Skip to content

bench: vision OCR accuracy baseline (#416) - #427

Open
Supermonters wants to merge 1 commit into
masterfrom
inter-v2/09-vision-ocr-accuracy
Open

Supermonters wants to merge 1 commit into
masterfrom
inter-v2/09-vision-ocr-accuracy

Conversation

@Supermonters

Copy link
Copy Markdown
Collaborator

Closes #416

Mục tiêu

Đo CER/WER vision OCR trên corpus scan tiếng Việt; so với baseline Tesseract historical (bench/REPORT.md §4). Chỉ đo & report — không đổi prompt/model.

Thay đổi

  • Thêm bench/REPORT_OCR_VISION.md — kết quả fileconv accuracy (5 mẫu).

Terminal evidence

Xem comment đầy đủ trên #416 (format inter-v2-07). Tóm tắt:

File Label CER Accuracy ms
vn_printed.png image-print-OCR 0.013 98.7% 5454
vn_lowres.png image-lowres-OCR 0.053 94.7% 5300
vn_hand_caveat.png handwrite-OCR 0.150 85.0% 6371
vn_hand_dancing.png handwrite-OCR 0.005 99.5% 5133
gold-004.pdf pdf-scan-OCR 0.111 88.9% 4579

TB 5 mẫu: CER ~0.066 (~93.4% accuracy); ms/file TB ~5367.

Contrast Tesseract (historical) vs vision

Nguồn: bench/REPORT.md §4 (2026-06-30). Tesseract removed 2026-08-10.

Kịch bản Tesseract (historical) Vision (2026-08-21)
image-print-OCR 99.2% 98.7%
image-lowres-OCR 99.0% 94.7%
handwrite-OCR (TB) ~47.9% 92.2%
pdf-scan ~1200 ms/trang 88.9% / ~4.6 s

Lệnh recreate

bash bench/make_vn_images.sh          # macOS: DejaVu ~/Library/Fonts/, dùng magick
bash bench/download_pdfium.sh
export FILECONV_OCR_API_KEY=...

cat > bench/vn_corpus/manifest_ocr.tsv <<'TSV'
# file	ground_truth	label
vn_printed.png	ref.txt	image-print-OCR
vn_lowres.png	ref.txt	image-lowres-OCR
vn_hand_caveat.png	ref.txt	handwrite-OCR
vn_hand_dancing.png	ref.txt	handwrite-OCR
../markhand_web/golden/documents/gold-004.pdf	../markhand_web/golden/markdown/gold-004.md	pdf-scan-OCR
TSV

cargo build --release -p fileconv-cli
./target/release/fileconv accuracy bench/vn_corpus/manifest_ocr.tsv bench/REPORT_OCR_VISION.md

macOS note: make_vn_images.sh upstream hardcode Linux font path; local workaround: DEJAVU=$HOME/Library/Fonts/DejaVuSans.ttf + magick (không commit trong PR này).

Acceptance checklist

Made with Cursor

Establish quality baseline for OpenRouter vision OCR on vn_corpus images
and gold-004 PDF scan per inter-v2-09 acceptance criteria.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Supermonters
Supermonters force-pushed the inter-v2/09-vision-ocr-accuracy branch from 16dc218 to 89102d7 Compare August 21, 2026 04:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[inter-v2-09] Benchmark OCR vision vs baseline Tesseract

1 participant