[char0n]

Description

• Criterion records. Every StepRunRecord now has criteria: one CriterionRecord per criterion the step's last attempt evaluated.
  • Order: successCriteria first, then the criteria of the onSuccess / onFailure actions it considered.
  • Each record holds { source, action?, condition, met, error? }; error is { reason, message }.
  • No request or response data is recorded.
  • StepExecutionResult carries the same list.
• Outcome from the evaluator. CriterionEvaluator.evaluate returns a CriterionOutcome ({ met, error? }) instead of a boolean. The CriterionEvaluationError it used to discard (runner: criterion evaluation errors must fail the criterion, not abort the step (Arazzo 1.1.0 MUST) #120) now reaches the record.
• No short-circuit. Every criterion in a list is evaluated, so the record explains each unmet criterion, not only the first. CriterionPredicate also receives the action the criterion belongs to.
  • A step that calls a workflow evaluates its successCriteria even when the called run failed.
  • Side effect: a malformed criterion (unknown type, missing context, …) after an unmet one now throws. Before, the unmet criterion hid it, and the step just failed or the action just didn't match. The same applies to the successCriteria of a workflow-calling step whose called run failed, which were not evaluated at all before.
• Softened successCriteria. In a step that calls a workflow, a successCriterion reading $statusCode, $url, $method, $request.* or $response.* no longer aborts the run.
  • It fails with an unavailable-expression error recorded, so onFailure applies.
  • Action criteria already behaved this way, and now record the error too.
  • outputs still throw.

Motivation and Context

Arazzo 1.1.0 says an implementation SHOULD log or report a criterion it cannot evaluate. Until now the runner either swallowed the error (#120) or aborted the run (#228). The record makes failed criteria visible without either.

Closes #231

BREAKING CHANGE (in the squash footer):

• CriterionEvaluator.evaluate returns a CriterionOutcome;
• StepExecutor.evaluateCriteria returns CriterionRecord[];
• StepExecutor.selectActions returns { matchedActions, criteria }.
• StepRunRecord and StepExecutionResult gain a required criteria field, which matters only to code that constructs them.

How Has This Been Tested?

packages/runner npm test: 720 passing. npm run lint and npm run typescript:declaration are clean.

New and updated specs:

• the evaluator returns the error with its reason for each of the 5 evaluation-error kinds;
• StepExecutor records every successCriterion and action criterion, with action names and diagnostics, and an empty list for a step without criteria;
• ActionResolver evaluates every criterion and passes the action;
• a workflow-calling step records unavailable-expression for its action criteria;
• a workflow-calling step fails its successCriteria with a recorded error, and takes onFailure;
• outputs reading $response.body still throw.

Website (after release), _reference/runner.md:

• ~405–415, StepRunRecord table: add the criteria row.
• ~492: HTTP-exchange expressions in successCriteria no longer throw. The criterion fails with a recorded unavailable-expression; only outputs throw.
• ~628–639: StepExecutionResult.criteria, and the new return types of evaluateCriteria and selectActions.
• ~819: CriterionEvaluator.evaluate returns a CriterionOutcome; mention CriterionRecord.

Checklist

My PR contains...

• No code changes (src/ is unmodified: changes to documentation, CI, metadata, etc.)
• Dependency changes (any modification to dependencies in package.json)
• Bug fixes (non-breaking change which fixes an issue)
• Improvements (misc. changes to existing features)
• Features (non-breaking change which adds functionality)

My changes...

• are breaking changes to a public API (config options, System API, major UI change, etc).
• are breaking changes to a private API (Redux, component props, utility functions, etc.).
• are breaking changes to a developer API (npm script behavior changes, new dev system dependencies, etc).
• are not breaking changes.

Documentation

• My changes do not require a change to the project documentation.
• My changes require a change to the project documentation.
• If yes to above: I have updated the documentation accordingly.

Automated tests

• My changes can not or do not need to be tested.
• My changes can and should be tested by unit and/or integration tests.
• If yes to above: I have added tests to cover my changes.
• If yes to above: I have taken care to cover edge cases in my tests.
• All new and existing tests passed.

[char0n]

Adversarial review (lean)

Profile lean, 3 dimensions (spec-adherence, code-quality, risk-robustness), 17 agents, Sonnet, 0 failures.
12 survived · 0 refuted · 0 no-quorum · 1 unrefuted (miss-challenge), plus 6 informational findings (not listed).

Suggestions

• Removing short-circuiting surfaces errors that were hidden (spec-adherence#0, risk-robustness#1, plus the unrefuted miss-challenge finding).
  • A structural CriterionError in a criterion after an unmet one now aborts the run. Before, the unmet criterion hid it.
  • The same applies to successCriteria of a workflow-calling step whose called run failed.
  • This is the intended trade-off of evaluating every criterion. It is now pinned by a test and stated in the PR body.
• The website _reference/runner.md is stale (code-quality#0).
  • It says successCriteria reading HTTP-exchange roots throw, and it doesn't document criteria.
  • This is deferred until after the release, per the docs-sync rule; the edits are listed in the PR body.

Nits

• StepActionSelection lists were mutable while the other result shapes are readonly (code-quality#1).
• The README didn't say records cover only the last attempt (code-quality#2).
• The BREAKING CHANGE list omits the new required criteria field on StepRunRecord / StepExecutionResult (risk-robustness#0).
• CriterionRecordError.reason is optional though every producer sets it (spec-adherence#3). Left as is: the optionality comes from CriterionError.reason, and an injected evaluator may omit it.

Contradiction reconciled

Findings disagreed on whether evaluating successCriteria after a failed called run adds throw paths. It does, but only for structural errors (the first suggestion above). Missing references resolve to undefined, and HTTP-exchange roots are now softened.

Auto-applied fixes

• 5197efb test(runner): pins that a malformed criterion after an unmet one throws (spec-adherence#0 / risk-robustness#1).
• ca7bfb8 refactor(runner): makes the StepActionSelection lists readonly (code-quality#1).
• c44a393 docs(README): says criterion records cover the last attempt (code-quality#2).
• PR body: states the side effect and lists the new required criteria fields under breaking changes (risk-robustness#0). The squash footer should include them.

[char0n]

Adversarial review: simplification pass (lean)

Profile lean, 3 dimensions (source simplicity, test simplicity, public API and docs minimality), 17 agents, Sonnet, 0 failures.
17 survived · 0 refuted · 0 no-quorum, plus 17 unrefuted miss-challenge findings. Many findings are duplicates; the distinct ones are below.

Auto-applied fixes

• 6493cf4 refactor(runner):
  • CriterionEvaluator.evaluate assigns met in each case and returns { met } once. The switch diff is now just return → met = …; break;.
  • selectActions returns an inline { matchedActions, criteria } type, so StepActionSelection is no longer exported.
  • The unavailable-expression catch comment is shortened.
  • The unrelated doc edit to RuntimeExpressionContext.ts is reverted.
• 6d4c086 test(runner): removes the new "no error" spec, which repeated the existing $statusCode == 404 spec. That spec now asserts { met: false }.
• 495b26f docs(README): the criterion records paragraph is two sentences, without evaluator internals. The pre-existing CriterionEvaluationError sentence is kept.

Squash footer: commit 1ad851a's BREAKING CHANGE text still names StepActionSelection. When squashing, say that selectActions returns { matchedActions, criteria }.

For the author (design / public API, not applied)

• Drop CriterionRecord.source (surface-minimality#0).
  • Whether a record belongs to an action can be read from action, and whether the action is on the success or failure path from the step's successful.
  • Trade-off: a record read on its own no longer says which list it came from.
• Narrow CriterionRecord.error (surface-minimality#1). Make it error?: string, or make reason a required string, instead of the optional CriterionErrorReason | 'unavailable-expression' union. Consumers would lose typed branching on the reason.
• Avoid a second error shape (unrefuted). CriterionOutcome carries the error object, while the record carries plain { reason, message }. The evaluator could return { met, reason?, message? } instead.
  • Counterpoint: the record is plain on purpose. It must not carry cause, which can hold XPath parser output.

Not applied (judged not worth it)

• Mapping { reason, message } once in #evaluateCriterion (source-simplicity#0): it needs a cast on RuntimeExpressionError.reason, which is typed string, to save one line.
• Merging the duplicated criteria and action code in StepExecutor.execute and the workflow-step path (source-simplicity#2): it would need a new public method.
• Collapsing the composition spec's assertions: cosmetic.
• No spec for a success criterion reading $response.body (test-simplicity#2): not a regression. calledResponseOutput was always an outputs case.
• The website is out of date: already listed in the PR body as a follow-up after release.