[EllianCarlos]

Problem

Step 10 undoes the srcML 1.1.0 mojibake (#97). srcML 1.1.0 reads UTF-8 as Latin-1, so a token holds café where the source holds café. srcML develop (#104) reads UTF-8 right. After #104, the repair changes correct text. Example: the literal u8"// café naïve" in srcML's test_srcml_unit_parse_encoding.cpp becomes café naïve. The token gets shorter, so the walk moves.

Step 10 gets no tokenizer identity. blobExec keeps the identity in the blob map as an opaque digest. Sharded mode keeps none. The digest does not name the srcML version.

Change

• The repair runs only when the source does not hold the token text as is (SourceReader.holds_next, misread_by_srcml). The comparison skips whitespace, as the walk does.
• A 1.1.0 token with non-ASCII text never matches strict UTF-8 source. Thus 1.1.0 data gets the same repair as before.
• A correct token always matches. Thus the gate needs no flag. It also works on a mix of old and new tokens.
• DATASET.md describes the gate.

Not changed

• Old datasets (tag corpus-c100-2026-10) and the repair of 1.1.0 tokens.
• The .rs exclusion. The Rust tokenizer reads strict UTF-8 (read_to_string), so it is correct. The gate also protects Rust.
• Tokenizers, the tokenizer identity, blobExec and the pipeline script.

Tests

• New: the source holds Hö as is, and the token stays as is.
• New: one file with café and the literal u8"café". Develop tokens and 1.1.0 tokens give the same rows and positions.
• New: holds_next unit tests.
• devenv shell -- pytest -q --junitxml=...: 82 passed (74 before), 0 failures, 0 errors.
• The CI shell suites pass: tokenize_gate 50, pipeline_workdir_guard 79, ensure_artifacts 42, retokenize_passthrough 34, reblame_passthrough 15.
• Sweep of 721 files through process_blame_file. With 1.1.0 tokens, the repair changes 26 tokens in 10 files, as before, and all 10 files now give the same rows as develop (before: 9). With develop tokens, the repair changes 0 tokens (before: 1).