Skip to content

test(harness): clarify child evidence and preserve unknown counts - #1154

Merged
XuPeng-SH merged 2 commits into
matrixorigin:mainfrom
loveRhythm1990:test/harness-evidence-contract-20261028
Oct 28, 2026
Merged

XuPeng-SH merged 2 commits into
matrixorigin:mainfrom
loveRhythm1990:test/harness-evidence-contract-20261028

Conversation

@loveRhythm1990

@loveRhythm1990 loveRhythm1990 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Capability assessment: Unchanged: this change tightens existing harness evidence checks without changing runtime capabilities or authority.

Summary

The workspace-evidence case now supplies an explicit read range covering the target section and asks for the literal edition without giving the expected value. Its existing causal-child criterion already requires exactly one successful child read. New regression coverage verifies that failed, unknown, unlinked and same-run reads cannot satisfy it. Delegation and final-answer checks remain unchanged.

Tool-count progress diagnostics now leave the count unknown for incomplete or corrupt captures, matching the grader’s existing fail-closed behavior instead of displaying a misleading zero.

Related issue

Follow-up to #1139. That PR merged at b5799a3e while this final test-only supplement was being pushed; this supplement was not included in its merge. This branch starts from current main 0fe324c6.

Change type

  • Bug fix
  • Test

User and compatibility impact

No product, authentication, database or deployment changes. Existing diagnostic schema and unknown-value representation are preserved. No timeout, retry or acceptance criterion is relaxed.

Architecture and complexity delta

The existing SessionCapture integrity metadata and shared tool-outcome counter remain canonical. No parallel counter or evidence projection is introduced. Existing grader and progress consumers were reviewed.

Verification

  • Final follow-up a79ec693: all 641 harness library tests passed, with 1 existing ignored test, including progress, case contracts and the new causal-child read regression. The first exact-name Cargo filter selected zero tests; it is not counted as validation. The subsequent full library test executable ran all 642 tests (641 passed, 1 ignored) with serial execution.
  • cargo clippy -p astra-test-harness --lib --tests -- -D warnings, formatting and diff checks passed.
  • The final diff preserves the existing successful causal-child assertion and adds no global tool-count acceptance check.
  • Unhappy paths cover skipped, truncated and conflicting captures; their verdict remains failed and count diagnostic remains null.
  • Database verification is not applicable: no database code changed.
  • The first supplement repetition (https://github.com/matrixorigin/Astra/actions/runs/40638389210) failed 0/3: every answer and original criterion passed, but an incorrectly added global outcome count saw two read records. The original causal-child criterion already requires a successful read. This follow-up removes the redundant broader count and tests the owning criterion instead; the failed run is not counted as a pass. The corrected three-repetition run on a79ec693 passed 3/3, with all original criteria passing: https://github.com/matrixorigin/Astra/actions/runs/40639899672 . The artifact confirms the exact source SHA. Full automatic PR CI remains pending; three repetitions do not prove that every historical intermittent failure is eliminated.
  • fix(memory): enforce per-user authority for turn observation #1139’s merged head passed its Test Suite, static checks and all 18 live regression cases. Those are parent-head results, not this draft’s CI approval.

Final checklist

  • Tests exercise the existing owning layer and unhappy paths.
  • No public contract or documentation update is needed.
  • Diff reviewed for secrets, private data and generated files.
  • Conventional Commit title and feature-branch delivery.

@loveRhythm1990 loveRhythm1990 changed the title test(harness): require successful reads and preserve unknown counts test(harness): clarify child evidence and preserve unknown counts Oct 28, 2026
@loveRhythm1990
loveRhythm1990 marked this pull request as ready for review October 28, 2026 06:23

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for one evidence-contract gap in the workspace case. The incomplete-capture diagnostic change and the existing causal-success checks otherwise look consistent with their current owners. The gap below predates this PR, but this PR explicitly tightens the case prompt and adds a case-specific regression test, so the case can still report a false pass for the behavior it is meant to certify.

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I withdraw my previous requested change. The existing case is a smoke test for causal child workspace execution and the final answer; this PR clarifies its prompt and preserves that existing acceptance contract. Requiring a new file/range-to-child-call evidence join as a merge blocker would extend the scope and add complexity beyond this change. The current checks do not prove strict answer provenance, so the pass should be interpreted within that limit. The incomplete-capture diagnostic uses the existing integrity owner, and the added tests cover the relevant causal and unhappy paths. CI and the live regression passed on this head.

@mergify mergify Bot added the queued label Oct 28, 2026
@mergify

mergify Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

🚫 Left the queue

1 min in the queue, with no time running CI.

So far

  • ✅ Entered the queue Wed 28 Oct, 09:02 · rule main · triggered by rule Automatic queue on approval for main
  • 🚫 Left the queue 09:03 · a79ec69

Reason

Pull request #1154 has been merged manually at 6b8f370

Hint

You were too fast!

Tick the box to put this pull request back in the merge queue (same as @mergifyio queue).

  • Requeue this pull request

Times in UTC · dashboard · how the merge queue works

@mergify
mergify Bot force-pushed the test/harness-evidence-contract-20261028 branch from a79ec69 to d5571f5 Compare October 28, 2026 09:03
@XuPeng-SH
XuPeng-SH merged commit 6b8f370 into matrixorigin:main Oct 28, 2026
8 checks passed
@mergify mergify Bot removed the queued label Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants