engine: move protobuf code from prost to buffa

[bpowers]

Why

The Rust side of project_io.proto now uses buffa instead of prost. Nothing else in the workspace depended on prost, so this removes prost and prost-derive (and their itertools 0.14) from Cargo.lock entirely. It adds buffa, smoothutf8 + simdutf8 (buffa's UTF-8 validation), and second versions of foldhash (0.1) and hashbrown (0.15). The TypeScript side (google-protobuf, src/server/schemas) is untouched, and the proto schema changes only in a comment.

What changed

• Generator. scripts/gen-rust-protobufs.sh drives buffa-build (still via protoc), pinned to exactly the buffa version Cargo.lock resolves. Generated code calls runtime internals that semver does not cover, so generator and runtime must move together. The options keep the generated types close to prost's:

  • one file per package: project_io.gen.rs keeps its hash header and freshness test.
  • no zero-copy views.
  • unknown fields dropped: no hidden bookkeeping field, so struct literals still compile.
  • no with_* setters.
  • idiomatic field names: viewBox -> view_box.
  • oneof variants stored inline.

• Types. Open enum fields are EnumValue<E> (were i32), and singular message fields are MessageField<T> (were Option<T>). Oneofs, strings, repeated fields and proto3 optional scalars are unchanged. serde.rs converts with as_known().unwrap_or_default(), which keeps the old fallback of unknown enum values to the default variant. Enum variants keep their prost idiomatic spellings (SimMethod::Euler) through buffa's const aliases, so the conversion matches still read the same.

• Downstream crates. libsimlin and simlin-cli use the engine's pub use buffa re-export instead of declaring their own protobuf dependency, so the engine alone owns the runtime version.

• Encode. simlin_project_serialize_protobuf uses try_encode_to_vec: buffa's panicking encode would abort the process (panic = abort) on a project past the 2 GiB limit rather than return an error.

• Decode limits. Decoding keeps buffa's defaults, including a 32 MiB budget on the memory repeated fields may materialize. prost had no such bound. Measured with a throwaway harness that binary-searched the smallest budget each model decodes under:

  model	encoded	budget needed
  C-LEARN v77	1,099,061 B	1,471,096 B
  FREE6	168,032 B	610,208 B
  WRLD3-03	113,663 B	357,224 B
  fishbanks.protobin	2,167 B	14,416 B

  C-LEARN, the largest model in the corpus, uses under a twentieth of the budget, and the budget bounds what a crafted payload can make a decode allocate. A project roughly 20x C-LEARN's size would be refused with a decode error, which prost did not do.

• CLI fix (separate commit). cli: read protobuf input to the end, found while building the wire check below. open_binary read with read_until(0), which stops at the first zero byte, so the CLI could not read any real .pb file, on main as well. It had no test; it now has two.

Evidence

• Wire compatibility across the corpus. Built simlin at main (prost) and on this branch (buffa), both with the open_binary fix applied, and ran every .xmile/.stmx/.itmx/.mdl/.protobin under test/ through convert --to protobuf. That is 511 files; the 23 the prost CLI cannot convert at all are excluded, leaving 488.
  • Encoding: byte-identical for 485 of 488.
  • Buffa reading prost's bytes: decoding and re-encoding matches prost doing the same, 488 of 488.
  • Prost reading buffa's bytes: matches buffa doing the same, 488 of 488. For the 3 below this was checked on the identical buffa-encoded file.
  • The 3 encodings that differ are MDL imports that are nondeterministic run to run (engine: MDL importer subscript/dimension classification is non-deterministic across process hash seeds #859): the same CLI emits different dimension names across runs, and a different set of models showed up in each of two passes.
• Older blobs. test/fishbanks.protobin and test/logistic-growth.protobin decode through the LTM tests and the json_proptest protobin round trip.
• Tests.
  • serde.rs unit tests, including the open-enum fallback rows and the legacy-field rows.
  • The protobuf idempotence proptests in json_proptest/json_sdai_proptest, which assert stable bytes after the first round trip.
  • The simulate corpus's protobuf round-trip assert_eq! on every corpus model.
  • The libsimlin integration suite (269 tests).
  • The new CLI open_binary_tests.
  • The full pre-commit hook on each commit.
• Size.
  • Browser wasm (wasm-release, --no-default-features, then build.sh's wasm-opt -O3 flags): 9,987,598 -> 10,017,186 bytes (+0.30%). Gzip -9: 3,181,543 -> 3,193,811 bytes (+12 KB).
  • Native release CLI: 7,557,496 -> 7,491,960 bytes.

What this does not establish

• No encode/decode throughput benchmark was run.
• Production blobs from the database were not sampled. Compatibility rests on the corpus encodings, the two checked-in older .protobin files, and both implementations being conformant to the same wire format.
• The browser download grows by about 12 KB compressed. That comes from buffa's expanded generated code (11.4k lines against prost's 1.1k-line derive input), and I did not try to shrink it further.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RqCF6kitvGj4xBVs5wZYw5

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-14T23:42:20.557700Z	a3709e1	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[codecov]

Codecov Report

❌ Patch coverage is 89.24051% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.05%. Comparing base (40ae5e9) to head (a3709e1).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/libsimlin/src/serialization.rs	25.00%	6 Missing ⚠️
src/simlin-engine/src/serde.rs	95.45%	5 Missing ⚠️
src/libsimlin/src/project.rs	40.00%	3 Missing ⚠️
src/simlin-cli/src/main.rs	89.65%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1068      +/-   ##
==========================================
- Coverage   93.29%   92.05%   -1.25%     
==========================================
  Files         282      282              
  Lines      165608   171663    +6055     
==========================================
+ Hits       154509   158025    +3516     
- Misses      11099    13638    +2539     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[claude]

Code Review

Reviewed the non-generated changes: the generator config in scripts/gen-rust-protobufs.sh, the serde.rs conversions, the FFI decode_from_slice / try_encode_to_vec swaps in libsimlin, the compat_from_proto signature change, the enum/message-field mechanics (EnumValue::as_known().unwrap_or_default() and MessageField), the CLI open_binary fix and its new tests, and the test-only encode_to_vec migrations.

No blocking findings.

Behavior notes worth flagging (already documented in the PR description, not defects):

• The FFI decode now inherits buffa's default 32 MiB budget on repeated-field materialization, where prost had no bound. This is a hard cap that would refuse a project roughly 20x C-LEARN's size. The PR notes real corpus models sit well under this.
• simlin_project_serialize_protobuf now correctly uses try_encode_to_vec because panic = abort in the release profile (Cargo.toml:31) would otherwise abort the process on a >2 GiB project rather than surface a decode error.
• The open_binary fix (using read_to_end instead of read_until(0)) resolves a pre-existing latent bug in the CLI protobuf import path — good catch, and the new open_binary_tests module covers both the whole-read path and truncation refusal.

Overall correctness verdict: correct

The wire compatibility across 488 corpus models (byte-identical on 485; the 3 differing encodings are attributable to nondeterminism in MDL imports, issue 859), the older .protobin fixtures continuing to decode, and the full test suite plus proptests all pass. The semantic behavior of enum-value fallbacks (try_from(i32).unwrap_or_default() -> EnumValue::from(i32).as_known().unwrap_or_default()) is preserved, and the MessageField/Option conversions are used consistently.

🤖 Generated with Claude Code