[Afloat16]

What & why

Plain input scanned every control token even when no byte could become a protocol token, and allocated a per-byte permission bitmap that could never allow a match.

Skip forbidden scans and the bitmap when all nonempty fragments are plain. Preserve standalone user-defined tokens, protocol fragments and explicit special parsing. This is independent of ordinary-token BPE merge scheduling.

Area

Engine, CLI or server (edge0-infer/).

How was it tested?

moe_test_text_bpe

The target passes. Cases with 512 control tokens cover empty protocol fragments, untrusted control spellings, user-defined tokens and explicit protocol parsing. A local benchmark repeats "<control_7> untrusted body " 512 times (13824 bytes), calls encode_input 100 times and checks every result. With 256 byte tokens and 512 controls, five samples have median 0.242569 s on main versus 0.108728 s after correction (2.23x for this synthetic case). Vocabulary setup is untimed; both produce 1382400 tokens across all iterations. Whole-model throughput was not measured.

All 14 L0 checks passed in a local integration with the other focused fixes. git diff --check passed.

Local C++ environment: Linux x86_64, GCC 14.2, CPU preset. Targeted builds downgrade pre-existing ignored-attributes, dangling-reference, range-loop-construct and misleading-indentation warnings (plus maybe-uninitialized in the existing diagnostics source); these accommodations are not source changes. The all-target build is blocked by existing assets_test.cpp use of std::count_if without . A clean full L0/L1 result, GPU tests and model-asset execution are not claimed.

Checklist

• Full ctest --preset cpu -L "L0|L1" passes locally
• Added/updated focused regressions and ran the affected target
• No public interface/configuration change requiring additional documentation