Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 47 additions & 1 deletion crates/astra-test-harness/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,10 @@ check results, not a claimed root-cause analysis. Raw prompts, tool/provider bod
and free-form judge diagnostics remain private; public artifacts do not include
them. Checks without a safe numeric projection explicitly say that their detailed
payload is withheld. Full private diagnostics remain in the local suite report.
Journal tool-outcome count failures publish only the expected bounds and actual
count, using the same deduplicated records as the verdict. Tool arguments,
results, and file contents remain private; an unavailable capture reports no
count instead of inventing zero.

Prompt-cache scope checks retain their grading-time observation count, post-cold
cache-read sum, and bounded per-execution capture status, identity-match flags,
Expand Down Expand Up @@ -405,7 +409,7 @@ loss or file-write failure cancels via the existing runner and fails closed.
Selected custom source revisions must support this option and the repository
helper; unsupported sources fail explicitly, without an old-log fallback.

Progress uses `astra.harness_progress.v5`. Unavailable judges report the canonical
Progress uses `astra.harness_progress.v7`. Unavailable judges report the canonical
error kind and a fixed failure stage; unknown causes remain `unknown`. Required
quorum failures retain the first typed cause and still fail closed. Raw errors
and individual vote diagnostics remain private. Argument failures include bounded
Expand Down Expand Up @@ -1085,6 +1089,48 @@ harness scheduling pressure, a heartbeat without new typed evidence is an
efficiency/non-convergence signal, and a terminal report with incomplete
evidence is never promoted to success merely because the process stayed alive.

The CI safe progress stream (`astra.harness_progress.v7`) includes an
`execution_diagnostic` on each terminal row and step. Failed rows print it in
the job log and summary, and retain it in `summary.json` and
`progress.safe.jsonl`. It preserves recognized terminal state, interruption,
error kind/code, admission status, and available journal/live-stream event
counts and the last captured row's event type (`last_event`, in artifact merge
order for journals). Unknown tags become `other`; missing capture
remains `null`. The journal's last provider failure retains its typed error,
admission/provider/settlement stage, deadline phase, dispatch/completion flags,
and whether its reported run matches the root or another run. Failure ordering
uses producer UTC wall-clock time (`ts` in canonical rows, epoch-millisecond
`created_at` in typed step rows), not artifact merge order. This is captured
time ordering, not proof of causality across producers. `last_provider_failure`
is `null` when there are no captured failures, the capture is incomplete
(skipped/truncated rows or integrity errors), any retained failure lacks a valid
timestamp, or the latest timestamp has conflicting safe failure projections.
The loader rejects typed events with missing or unrepresentable producer time
and records capture corruption. Invocation scoping also marks reliably owned
but unorderable rows as corrupt before excluding them; further scopes retain
that uncertainty. Invalid rows never become certified invocation evidence.
Retained-event counters and corruption/truncation counters remain visible;
they do not reconstruct dropped failures. A failed-event tag remains a failure
observation even when its optional error kind is missing; a successful completed
event's null error kind does not replace a failure. Per-step journals require matching
session and run identities and use the canonical invocation scope, including
causally linked descendants; an unavailable association remains `null` rather
than borrowing another step's journal. This is a
captured observation, not a root-cause verdict. No stderr, response bodies,
event text, identities, or tool arguments are published. Live termination counters count observed events,
not unique children or verified adoption, and captured events may be partial.
Diagnostic fields do not regrade outcomes or change runtime retries/timeouts.
Corrupt captures fail closed for durable evidence and session cleanup.
The journal also projects `loop_boundaries`: counts of recognized model-round
finish reasons and direct-child dependency outcomes, grouped as root, another
run, or unattributed. This distinguishes recovery loops, superseded responses,
and result staging/adoption without publishing trace attributes or identities.
Missing entries mean no such observation was captured, not proof that the
underlying transition did not happen.
In particular, a cancellation observed after an outer timeout is not proof of
the original failure cause. Push a new commit to exercise new diagnostics;
rerunning an older workflow run still checks out that run's original PR SHA.

## Preflight checks

Before running cases, the harness validates:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ prompt: |
requested_model_policy={"mode":"fixed","selector":{"kind":"configured_name","model_name":"glm-99.99"}}。
glm-99.99 是服务端配置模型名称,不是 Offering ID;保持 configured_name,
不要改为 offering_id,也不要省略模型策略。
这是一条模型 admission 负向测试,spawn 参数必须合法。使用以下完整参数:
{"action":"spawn","description":"Check unavailable model","prompt":"Reply ASTRA-MISSING-MODEL-MUST-NOT-RUN","requested_model_policy":{"mode":"fixed","selector":{"kind":"configured_name","model_name":"glm-99.99"}}}
子任务指令放在 prompt;不要添加 message、message_type、to 或 agent_id,
这些是其他 action 的字段,会在模型检查之前导致参数校验失败。
不要设置 execution 或自行编造其他模型字段。这个 spawn 应由运行时 admission
拒绝;不要先调用 model_catalog,不要换模型,不要重试,也不要用文字模拟
拒绝。最终如实说明失败原因,并包含 ASTRA-MISSING-MODEL-NOT-AVAILABLE。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ description: |
prompt: |
只尝试一次委派,让 glm-99.99 模型的子代理回答 ASTRA-MISSING-MODEL-MUST-NOT-RUN。
使用 requested_model_policy={"mode":"fixed","selector":{"kind":"configured_name","model_name":"glm-99.99"}},不要设置 execution 或其他模型字段。
这是一条模型 admission 负向测试,spawn 参数必须合法。使用以下完整参数:
{"action":"spawn","description":"Check unavailable model","prompt":"Reply ASTRA-MISSING-MODEL-MUST-NOT-RUN","requested_model_policy":{"mode":"fixed","selector":{"kind":"configured_name","model_name":"glm-99.99"}}}
子任务指令放在 prompt;不要添加 message、message_type、to 或 agent_id,
这些是其他 action 的字段,会在模型检查之前导致参数校验失败。
若委派失败,不要重试或换模型。之后由当前代理调用 model_catalog 查询当前可用的 Chat 模型,
使用这次真实返回的目录结果,如实告诉我委派结果及可用模型。
debug_log: true
Expand Down
6 changes: 1 addition & 5 deletions crates/astra-test-harness/src/criteria.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3535,11 +3535,7 @@ fn evaluate_one_with_primary_cache(
let Some(session) = session else {
return missing_required_session(c, "journal_tool_outcome_count");
};
let count = session
.journal_tool_calls()
.iter()
.filter(|call| call.name == *name && call.ok == Some(*ok))
.count() as u32;
let count = session.journal_tool_outcome_count(name, *ok) as u32;
let passed = count >= *min && count <= *max;
CriterionResult {
criterion: c.clone(),
Expand Down
Loading
Loading