Unicode data

[adh]

No description provided.

[Copilot]

🟡 Changes recommended

The current generator/runtime path drops conditional SpecialCasing rules (e.g., Final_Sigma), and Character>>whitespace? misses some standard whitespace control code points, leading to incorrect Unicode behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR introduces generated, compact Unicode tables to support Unicode-aware casing and category queries in the ListTalk VM, and wires the generation step into the Meson build.

Changes:

• Add a Python generator that converts Unicode UCD inputs into a compact C table implementation (unicode_data.c) plus a new internal header.
• Implement new Unicode-aware String primitives (caseFold, lower, upper, title) and Character primitives (simple casing, category, and several ... ? predicates).
• Extend the test suite with new Unicode casing/category/property checks.

File summaries

File	Description
tools/generate_unicode_data.py	Adds generator for compact Unicode property/case tables consumed by the VM.
meson.build	Adds a Meson custom_target() to generate and compile unicode_data.c into the VM library.
src/utils/unicode_data.h	Introduces internal API for Unicode category/casing/casefold lookups.
src/classes/String.c	Adds Unicode-aware String casing and case-folding primitives backed by generated tables.
src/classes/Character.c	Adds Unicode-aware Character casing/category/property primitives backed by generated tables.
tests/eval-objects.lt	Adds coverage for new casing/category/property behavior.
data/CaseFolding.txt	Adds Unicode UCD input used by the generator.
data/SpecialCasing.txt	Adds Unicode UCD input used by the generator.

Review details

• Files reviewed: 8/9 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

The Unicode data generator has verified correctness issues (guard ordering and size-limit checking) and skips conditional special-casing rules like Final_Sigma, which can produce incorrect casing behavior.

Review details

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

tools/generate_unicode_data.py:120

• generate() converts property_indices to bytes before enforcing the 8-bit index constraint. If len(distinct_properties) > 256, bytes(...) will raise a generic ValueError before the intended "Unicode tables no longer fit in eight-bit indices" error, making the guard ineffective.

    distinct_properties, property_indices = unique_with_indices(properties)
    raw_blocks = [
        bytes(property_indices[offset : offset + BLOCK_SIZE])
        for offset in range(0, CODEPOINT_COUNT, BLOCK_SIZE)

tools/generate_unicode_data.py:133

• The case-fold table size check runs before appending the current mapping and doesn't account for len(mapping). If the table ever approaches the 16-bit limit, fold_values.extend(mapping) can push it past the limit despite the guard.

    for codepoint, mapping in case_folding:
        if len(fold_values) > 0xffff or len(mapping) > 0xff:
            raise ValueError("Case-fold tables no longer fit compact offsets")
        fold_rows.append(
            f"    {{UINT32_C(0x{codepoint:x}), {len(fold_values)}, {len(mapping)}}},"
        )
        fold_values.extend(mapping)

tools/generate_unicode_data.py:80

• parse_special_casing() currently skips all conditional SpecialCasing entries (if fields[4]: continue). That drops language-insensitive context rules like Final_Sigma, so String>>lower/String>>title will never produce U+03C2 (final sigma) in word-final positions, diverging from Unicode default case algorithms.

            fields = [field.strip() for field in content.split(";")]
            if len(fields) < 5:
                raise ValueError(f"{path}:{line_number}: expected at least 5 fields")
            if fields[4]:
                continue

• Files reviewed: 8/9 changed files
• Comments generated: 0 new
• Review effort level: Lite