Skip to content

fix(expr): fix memory accounting gaps in coercion, attributes, and slices - #418

Merged
mwiebe merged 3 commits into
OpenJobDescription:mainfrom
mwiebe:fix/accounting-followups
Sep 30, 2026
Merged

mwiebe merged 3 commits into
OpenJobDescription:mainfrom
mwiebe:fix/accounting-followups

Conversation

@mwiebe

@mwiebe mwiebe commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

What was the problem/requirement? (What/Why)

The expression evaluator in openjd-expr enforces a memory limit so that an expression like 'A' * 100000000 fails instead of exhausting the host. It does this with a running total: each value's size is added when the value is created and subtracted when it is consumed. That total is only useful if both sides agree on every value's size and every allocation goes through it.

Four places did not meet that bar:

  1. Type coercion. When a value is coerced to a target type after being created (an int to a string, or a range_expr to a list[int]), it was added at the old size and later subtracted at the new size. For growing coercions this subtracted too much, hiding memory that was still in use. A range_expr coerced to a list was never checked against the limit at all.
  2. Friendly error messages. Attribute access (x.name) and method calls replace low-level errors with clearer ones like "Undefined variable" or "'name' is a property, not a method". A memory- or operation-limit error went through the same replacement and came out as an ordinary error. The evaluator deliberately ignores ordinary errors in some places (an if/else whose condition is not yet known evaluates both branches and tolerates one failing), so a limit error could be silently dropped.
  3. Slice bounds. For x[:], the omitted bounds were passed along as placeholder values that were never added to the total but were subtracted from it afterwards, 64 bytes each.
  4. String slicing. slice_string built two temporary vectors (4 bytes per input character, 8 bytes per selected character) that were never counted or checked, then built a result whose buffer capacity had grown past its actual length.

What was the solution? (How)

  • eval_node releases the original value, coerces it, and tracks the coerced value.
  • eval_attribute and eval_call check whether an error is a limit error before replacing it, and pass limit errors through unchanged.
  • Slice placeholders are tracked when created. The two early exits for an unresolved receiver or bound also release the operands they discard.
  • slice_string computes the number of selected characters arithmetically, checks min(input bytes, 4 × count) against the limit before allocating, copies the characters straight from the input with no temporaries, and shrinks the result buffer so its tracked size is exact. slice_list gets the same pre-check. While testing this, the shared index loop was found to overflow on a step near i64::MAX (a debug-build panic on 'hello'[1::9223372036854775807]); it now saturates.

What is the impact of this change?

The memory limit is enforced more accurately. Expressions that grow a value through coercion, or that slice large strings, can now hit the limit where before they slipped under it. Limit errors can no longer be hidden by a friendlier error message. The peak_memory metric reported by evaluate_with_metrics changes for affected expressions.

Value semantics do not change. The only user-visible message changes are byte counts in memory-limit diagnostics; one existing pinned figure moves by 128 bytes.

How was this change tested?

  • Have you run the unit tests? Yes: cargo test --workspace, plus cargo clippy --all-features --all-targets --workspace -- -D warnings and cargo fmt --all -- --check.
  • 15 new tests. Each fix has at least one test that fails when only that fix is reverted, verified by reverting each hunk individually. Byte figures in the new tests were derived by hand from size_of::<ExprValue>() and the heap-size rules and match the output exactly.
  • An independent review of the change re-derived the figures and confirmed the per-hunk test isolation. It found the eval_call case (fix 2), which was added as a result.

Was this change documented?

  • Are relevant docstrings in the code base updated? Yes. specs/expr/evaluator.md (Target Type Propagation, Attribute, Call, Speculative evaluation, Subscript) and specs/expr/function-library.md (Preflighting Output Budgets) describe the new behavior.

Is this a breaking change?

No. No public signatures change. Expressions that previously passed a memory limit only because of an accounting error may now fail it; that is the limit working as intended.

Does this change impact security?

This change tightens the memory bound the evaluator enforces on untrusted template expressions. It does not create or modify files and does not need threat modeling.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@mwiebe
mwiebe requested a review from a team as a code owner September 30, 2026 15:10
mwiebe added a commit to mwiebe/openjd-rs that referenced this pull request Sep 30, 2026
…ices

The evaluator tracks memory by adding each value's size when it is
created and subtracting it when it is consumed. Several places got this
wrong, and two places could turn a memory or operation limit error into
an ordinary error that the evaluator then ignored.

- eval_node coerced a value to the target type after the value had been
  tracked, so it was tracked at the old size and later released at the
  new size. When coercion grows a value (int to string, range_expr to
  list[int]) the release subtracted too much, and a range_expr coerced
  to a list was never checked against the memory limit at all. eval_node
  now releases the original and tracks the coerced value.

- eval_attribute replaces errors from its base evaluation and property
  dispatch with friendlier ones (undefined variable, property not
  available). Limit errors went through the same replacement and became
  ordinary errors, which an if/else with an unresolved test or an
  and/or past an unresolved operand then absorbed. eval_call does the
  same for a failed method call whose name is also a property. Both now
  leave limit errors unchanged.

- Omitted slice bounds were passed to dispatch as Null values that were
  never tracked, and dispatch subtracted 64 bytes for each. They are now
  tracked. The slice's early exits for an unresolved receiver or bound
  also left the receiver and bounds tracked after discarding them; they
  are now released.

- slice_string collected the input into a Vec<char> (4 bytes per
  character) and an index Vec<usize> (8 per selected element), neither
  tracked nor checked against the limit, then built the result from an
  iterator whose capacity grew past the actual length. It now computes
  the number of selected characters arithmetically, checks
  min(input bytes, 4 x count) before allocating, copies the characters
  directly from the input, and shrinks the buffer so the tracked size is
  exact. slice_list gets the same check before allocating. While testing
  this, collect_indices was found to overflow on a step near i64::MAX
  ('hello'[1::9223372036854775807] panicked in debug builds); it now
  saturates.

One existing pinned figure moves by the two placeholders in [::-1]
(128000160 -> 128000288). New tests: nine in test_memory.rs for the
evaluator fixes, one with a custom library for the eval_call case, two
for slice budgeting, two unit tests in comparison.rs for slice_len and
the character copy, and extreme-step and multi-byte cases in
test_slicing.rs. specs/expr/evaluator.md and
specs/expr/function-library.md are updated to match.

Signed-off-by: Mark <399551+mwiebe@users.noreply.github.com>
@mwiebe
mwiebe force-pushed the fix/accounting-followups branch from bff3e16 to f9da3d6 Compare September 30, 2026 15:12
Comment thread crates/openjd-expr/src/functions/comparison.rs Outdated
Comment thread crates/openjd-expr/src/functions/comparison.rs Outdated
leongdl
leongdl previously approved these changes Sep 30, 2026
step.unsigned_abs() as usize truncates to zero on 32-bit targets such as
wasm32 when the step is a multiple of 2^32 (including i64::MIN), and
step_by(0) panics. Reproduced on i686-pc-windows-msvc with the existing
extreme-step test. Clamp with usize::try_from(..).unwrap_or(usize::MAX):
when more than one character is selected the step is below the string
length and fits; when one is selected the stride only needs to be
non-zero. Adds 2^32 steps to the slicing test.

Signed-off-by: Mark <399551+mwiebe@users.noreply.github.com>
slice_list checked count x 64 bytes up front but then allocated an
index Vec<usize> by repeated push and collected the result through a
filter_map, whose size hint is zero, so the result Vec grew by doubling.
Peak allocation was about 2.25x the checked amount, and the elements
were all cloned before any per-element check.

The indices are now produced lazily by an iterator (slice_indices,
replacing collect_indices), and the result is built into a
BudgetedVec::with_capacity that reserves the exact count once and
charges each element as it is pushed, the same way slice_range builds
its reverse walk. A new test slices 100 strings of 100 KB and fails at
the 20th push (12009056 bytes) where the previous code built all 100
first (20009056 bytes).

Signed-off-by: Mark <399551+mwiebe@users.noreply.github.com>
@mwiebe
mwiebe enabled auto-merge (squash) September 30, 2026 16:23
@mwiebe
mwiebe merged commit 3740a96 into OpenJobDescription:main Sep 30, 2026
22 checks passed
@mwiebe
mwiebe deleted the fix/accounting-followups branch September 30, 2026 16:47
@github-actions github-actions Bot mentioned this pull request Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants