[Parser] Cyrillic and non-Latin characters from OCR not cleaned

[ilhamfp]

Parser Improvement: Cyrillic and non-Latin characters from OCR not cleaned

Severity: MEDIUM
File: ocr_correct.py
Function: correct_ocr_errors
Estimated errors fixed: 1

Current Behavior

Cyrillic characters like 'Д', 'Ж', 'Ѽ' and other non-Latin characters like 'Œ' appear in output when OCR misreads graphical elements (logos, signatures, decorative elements) or digits/dates. These are never valid in Indonesian legal text.

Proposed Fix

Add patterns to remove or replace Cyrillic characters and other non-Latin script characters that should never appear in Indonesian legal text. Isolated non-Latin characters (on their own line or surrounded by spaces) should be removed entirely as they represent graphical artifacts. Cyrillic characters embedded within otherwise Latin text should be mapped to their visual Latin equivalents where possible.

Code Before

    # Ligature and encoding artifacts
    (re.compile(r'fi'), 'fi'),
    (re.compile(r'fl'), 'fl'),
    (re.compile(r'ff'), 'ff'),
    (re.compile(r'\u00a0'), ' '),  # Non-breaking space → regular space

Code After

    # Ligature and encoding artifacts
    (re.compile(r'fi'), 'fi'),
    (re.compile(r'fl'), 'fl'),
    (re.compile(r'ff'), 'ff'),
    (re.compile(r'\u00a0'), ' '),  # Non-breaking space → regular space

    # Cyrillic and non-Latin artifacts from OCR misreading graphics/logos/signatures
    # Remove lines that are just isolated non-Latin characters (graphical artifacts)
    (re.compile(r'^\s*[\u0400-\u04FF\u0500-\u052F\u0152\u0153\u2000-\u200F]+\s*$', re.MULTILINE), ''),
    # Remove isolated Œ/œ (common OCR artifact for logos/signatures)
    (re.compile(r'^\s*[Œœ]+\s*$', re.MULTILINE), ''),
    # Map common Cyrillic->Latin lookalikes when embedded in Latin text
    (re.compile(r'А'), 'A'),  # Cyrillic А -> Latin A
    (re.compile(r'В'), 'B'),  # Cyrillic В -> Latin B
    (re.compile(r'С'), 'C'),  # Cyrillic С -> Latin C
    (re.compile(r'Е'), 'E'),  # Cyrillic Е -> Latin E
    (re.compile(r'Н'), 'H'),  # Cyrillic Н -> Latin H
    (re.compile(r'К'), 'K'),  # Cyrillic К -> Latin K
    (re.compile(r'М'), 'M'),  # Cyrillic М -> Latin M
    (re.compile(r'О'), 'O'),  # Cyrillic О -> Latin O
    (re.compile(r'Р'), 'P'),  # Cyrillic Р -> Latin P
    (re.compile(r'Т'), 'T'),  # Cyrillic Т -> Latin T
    (re.compile(r'Х'), 'X'),  # Cyrillic Х -> Latin X

---

Generated by the Pasal.id Correction Agent (Opus 4.6) after analyzing 5 parser feedback entries.