fix(expr): fix memory accounting gaps in coercion, attributes, and slices - #418
Merged
Merged
Conversation
…ices
The evaluator tracks memory by adding each value's size when it is
created and subtracting it when it is consumed. Several places got this
wrong, and two places could turn a memory or operation limit error into
an ordinary error that the evaluator then ignored.
- eval_node coerced a value to the target type after the value had been
tracked, so it was tracked at the old size and later released at the
new size. When coercion grows a value (int to string, range_expr to
list[int]) the release subtracted too much, and a range_expr coerced
to a list was never checked against the memory limit at all. eval_node
now releases the original and tracks the coerced value.
- eval_attribute replaces errors from its base evaluation and property
dispatch with friendlier ones (undefined variable, property not
available). Limit errors went through the same replacement and became
ordinary errors, which an if/else with an unresolved test or an
and/or past an unresolved operand then absorbed. eval_call does the
same for a failed method call whose name is also a property. Both now
leave limit errors unchanged.
- Omitted slice bounds were passed to dispatch as Null values that were
never tracked, and dispatch subtracted 64 bytes for each. They are now
tracked. The slice's early exits for an unresolved receiver or bound
also left the receiver and bounds tracked after discarding them; they
are now released.
- slice_string collected the input into a Vec<char> (4 bytes per
character) and an index Vec<usize> (8 per selected element), neither
tracked nor checked against the limit, then built the result from an
iterator whose capacity grew past the actual length. It now computes
the number of selected characters arithmetically, checks
min(input bytes, 4 x count) before allocating, copies the characters
directly from the input, and shrinks the buffer so the tracked size is
exact. slice_list gets the same check before allocating. While testing
this, collect_indices was found to overflow on a step near i64::MAX
('hello'[1::9223372036854775807] panicked in debug builds); it now
saturates.
One existing pinned figure moves by the two placeholders in [::-1]
(128000160 -> 128000288). New tests: nine in test_memory.rs for the
evaluator fixes, one with a custom library for the eval_call case, two
for slice budgeting, two unit tests in comparison.rs for slice_len and
the character copy, and extreme-step and multi-byte cases in
test_slicing.rs. specs/expr/evaluator.md and
specs/expr/function-library.md are updated to match.
Signed-off-by: Mark <399551+mwiebe@users.noreply.github.com>
mwiebe
force-pushed
the
fix/accounting-followups
branch
from
September 30, 2026 15:12
bff3e16 to
f9da3d6
Compare
leongdl
previously approved these changes
Sep 30, 2026
step.unsigned_abs() as usize truncates to zero on 32-bit targets such as wasm32 when the step is a multiple of 2^32 (including i64::MIN), and step_by(0) panics. Reproduced on i686-pc-windows-msvc with the existing extreme-step test. Clamp with usize::try_from(..).unwrap_or(usize::MAX): when more than one character is selected the step is below the string length and fits; when one is selected the stride only needs to be non-zero. Adds 2^32 steps to the slicing test. Signed-off-by: Mark <399551+mwiebe@users.noreply.github.com>
slice_list checked count x 64 bytes up front but then allocated an index Vec<usize> by repeated push and collected the result through a filter_map, whose size hint is zero, so the result Vec grew by doubling. Peak allocation was about 2.25x the checked amount, and the elements were all cloned before any per-element check. The indices are now produced lazily by an iterator (slice_indices, replacing collect_indices), and the result is built into a BudgetedVec::with_capacity that reserves the exact count once and charges each element as it is pushed, the same way slice_range builds its reverse walk. A new test slices 100 strings of 100 KB and fails at the 20th push (12009056 bytes) where the previous code built all 100 first (20009056 bytes). Signed-off-by: Mark <399551+mwiebe@users.noreply.github.com>
mwiebe
enabled auto-merge (squash)
September 30, 2026 16:23
leongdl
approved these changes
Sep 30, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was the problem/requirement? (What/Why)
The expression evaluator in
openjd-exprenforces a memory limit so that an expression like'A' * 100000000fails instead of exhausting the host. It does this with a running total: each value's size is added when the value is created and subtracted when it is consumed. That total is only useful if both sides agree on every value's size and every allocation goes through it.Four places did not meet that bar:
intto astring, or arange_exprto alist[int]), it was added at the old size and later subtracted at the new size. For growing coercions this subtracted too much, hiding memory that was still in use. Arange_exprcoerced to a list was never checked against the limit at all.x.name) and method calls replace low-level errors with clearer ones like "Undefined variable" or "'name' is a property, not a method". A memory- or operation-limit error went through the same replacement and came out as an ordinary error. The evaluator deliberately ignores ordinary errors in some places (anif/elsewhose condition is not yet known evaluates both branches and tolerates one failing), so a limit error could be silently dropped.x[:], the omitted bounds were passed along as placeholder values that were never added to the total but were subtracted from it afterwards, 64 bytes each.slice_stringbuilt two temporary vectors (4 bytes per input character, 8 bytes per selected character) that were never counted or checked, then built a result whose buffer capacity had grown past its actual length.What was the solution? (How)
eval_nodereleases the original value, coerces it, and tracks the coerced value.eval_attributeandeval_callcheck whether an error is a limit error before replacing it, and pass limit errors through unchanged.slice_stringcomputes the number of selected characters arithmetically, checksmin(input bytes, 4 × count)against the limit before allocating, copies the characters straight from the input with no temporaries, and shrinks the result buffer so its tracked size is exact.slice_listgets the same pre-check. While testing this, the shared index loop was found to overflow on a step neari64::MAX(a debug-build panic on'hello'[1::9223372036854775807]); it now saturates.What is the impact of this change?
The memory limit is enforced more accurately. Expressions that grow a value through coercion, or that slice large strings, can now hit the limit where before they slipped under it. Limit errors can no longer be hidden by a friendlier error message. The
peak_memorymetric reported byevaluate_with_metricschanges for affected expressions.Value semantics do not change. The only user-visible message changes are byte counts in memory-limit diagnostics; one existing pinned figure moves by 128 bytes.
How was this change tested?
cargo test --workspace, pluscargo clippy --all-features --all-targets --workspace -- -D warningsandcargo fmt --all -- --check.size_of::<ExprValue>()and the heap-size rules and match the output exactly.eval_callcase (fix 2), which was added as a result.Was this change documented?
specs/expr/evaluator.md(Target Type Propagation, Attribute, Call, Speculative evaluation, Subscript) andspecs/expr/function-library.md(Preflighting Output Budgets) describe the new behavior.Is this a breaking change?
No. No public signatures change. Expressions that previously passed a memory limit only because of an accounting error may now fail it; that is the limit working as intended.
Does this change impact security?
This change tightens the memory bound the evaluator enforces on untrusted template expressions. It does not create or modify files and does not need threat modeling.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.