Rust I18N Rewrite

[logan-gatlin]

Rewrites node I18N in Rust. The C++ implementation defers to ICU and simdutf for some conversions. ICU bindings are provided by rust_icu while simdutf bindings are written by hand (I filed a PR to the existing simdutf bindings to make their build work for us, but was ignored).

JSG Additions

• Uint8Array::from_buffer
• new_zeroed_with - allocates a buffer, zeroes it, then writes to it from a closure. This allows Rust to safely create and initialize buffers. Since the buffer is created in the same function it is written to, there are provably no other references to the buffer so mutating is safe.

[ask-bonk]

Since last review: 0 resolved, 0 still open, 0 new.
LGTM!

Not re-run: tests, docs, jsg-gc, kj-style, memory-safety, rust-first (no author changes in their files since the last review)

---

_{Reviewed commit: db9e3e66 · github run}

[logan-gatlin]

I'm Bonk, and I've done a quick review of your PR.

Rewrites Node I18N transcoding in Rust behind an autogate.

1. High: src/rust/i18n/lib.rs:139 exposes uninitialized backing-store bytes through result.buffer. Posted one inline suggestion.

Targeted tests could not build because the runner lacks libc++ headers. Time for a pun! That buffer had too many loose bytes.

github run

fixed

[danlapid]

Still too much unsafe - can't we drop it down more? we really want our rust code to be safe

[codecov-commenter]

Codecov Report

❌ Patch coverage is 79.79381% with 98 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.14%. Comparing base (e775a96) to head (db9e3e6).

Files with missing lines	Patch %	Lines
src/workerd/api/node/tests/buffer-nodejs-test.js	0.00%	72 Missing ⚠️
src/rust/i18n/ffi.rs	95.08%	9 Missing ⚠️
src/workerd/api/node/i18n.c++	76.92%	2 Missing and 4 partials ⚠️
src/rust/i18n/dispatch.rs	96.29%	5 Missing ⚠️
src/rust/jsg/ffi.c++	57.14%	2 Missing and 1 partial ⚠️
src/rust/jsg/v8.rs	94.73%	2 Missing ⚠️
src/rust/i18n/error.rs	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7322      +/-   ##
==========================================
+ Coverage   41.04%   41.14%   +0.09%     
==========================================
  Files         986      990       +4     
  Lines      284720   285204     +484     
  Branches    27618    27628      +10     
==========================================
+ Hits       116877   117353     +476     
- Misses     151757   151761       +4     
- Partials    16086    16090       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[logan-gatlin]

Still too much unsafe - can't we drop it down more? we really want our rust code to be safe

Yes we can! Most of the unsafe was bindings to icu and simdutf. We only use a small slice of either library, so I rewrote what we needed in safe Rust.

[logan-gatlin]

After discussing with Mike and James, and running some benchmarks to compare simdutf with my Rust rewrite, I've decided to keep the C bindings to ICU and simdutf.

For ICU it is a compatibility concern, if the ICU C library changes to include new additions to unicode, we need to pick those up. A Rust rewrite puts that maintenance burden on us.

For simdutf it is a performance concern. Rust CAN achieve the same performance as C, but not without unsafe. The compiler intrinsics that simdutf uses are unsafe in Rust.