Skip to content

feat(std): add String::toScalars and deprecate the byte-wise toChars - #441

Merged
0xErwin1 merged 1 commit into
mainfrom
feat/string-to-scalars
Oct 28, 2026
Merged

0xErwin1 merged 1 commit into
mainfrom
feat/string-to-scalars

Conversation

@0xErwin1

Copy link
Copy Markdown
Collaborator

String::toChars() returns one char per byte without decoding, so "é" comes back as two chars. Its doc says so, but the name suggests characters: the std::regex parser used it and split non-ASCII scalars until the tests caught it.

What changes

  • String::toScalars(): Vector<char> decodes the UTF-8 into Unicode scalars. A byte that doesn't start a well-formed sequence decodes as U+FFFD and advances one byte, the same as charAt, so the result never splits a scalar.
  • toChars, toVectorChars and str's toChars extension are marked @deprecated. The message points to toScalars() for scalars and to toBytes() for bytes. Their docs now state the byte-wise behavior up front.
  • The compiler's two callers move off it:
    • test_runner: the failed-test output limit used toChars().length(), which is the byte count, and cuts with substring, which takes byte offsets. It now uses length(), so the behavior is identical.
    • LSP leadingChars: "the first count characters" now takes scalars, so a completion prefix no longer cuts a non-ASCII character in half.

Note

The deprecation lint (A0124) only fires for calls through a path. Instance-method calls like text.toChars() are a documented gap in ignis/analyzer/lint.ign, so existing callers get no warning yet. The attribute still shows in the docs and in ignis doc.

Evidence

  • New tests: toScalarsDecodesUtf8 ("añ😀" is 7 bytes and 3 scalars), toScalarsReplacesMalformedBytes, toScalarsOfEmptyIsEmpty.
  • test-std 645/645; check-std and ignis check are clean.
  • With the compiler built from this tree: --filter completion 22, test_runner 20, lsp 116, all passing.
  • The doc baselines are regenerated.

String::toChars (and toVectorChars, and str's toChars extension) copy one
char per byte without decoding, so "é" gives two chars, which the name
hides; the regex parser tripped over it. String::toScalars decodes the
UTF-8 into Unicode scalars (a malformed byte becomes U+FFFD and advances one
byte), and the byte-wise helpers are marked @deprecated, pointing to
toScalars or toBytes.

The compiler's two callers move off it: the failed-test output limit counts
bytes, as the substring it cuts with does, and the language server's
leadingChars takes scalars.
@0xErwin1
0xErwin1 merged commit 3a439a1 into main Oct 28, 2026
9 checks passed
@0xErwin1
0xErwin1 deleted the feat/string-to-scalars branch October 28, 2026 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant