Skip to content

evaluation(workbuddy): add capture ownership replay and contamination report #2122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tmt-123 wants to merge 12 commits into
base: master
Choose a base branch
from
+11,089 −23
Open

evaluation(workbuddy): add capture ownership replay and contamination report #2122

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
2c21132
eval(workbuddy): add capture ownership replay and decision report
tmt-123 Oct 19, 2026
a73a4d9
eval(workbuddy): publish paired replay findings and source support audit
tmt-123 Oct 19, 2026
c3f2e80
Merge remote-tracking branch 'upstream/master' into eval/workbuddy-ca…
tmt-123 Oct 19, 2026
5ddffc7
docs(eval): document evidence limits and production follow-up contract
tmt-123 Oct 19, 2026
11ea32d
docs(eval): trim duplicate capture ownership reports
tmt-123 Oct 20, 2026
f71f695
Merge upstream/master and resolve evaluation configuration conflicts
tmt-123 Oct 22, 2026
55713ea
docs(eval): clarify frozen report revision scope
tmt-123 Oct 22, 2026
17ecf1d
fix(eval): enforce capture ownership replay and scoring invariants
tmt-123 Oct 23, 2026
843f6fc
Merge branch 'master' into eval/workbuddy-capture-ownership
tmt-123 Oct 23, 2026
e1002a7
docs(eval): restore memory index final newline
tmt-123 Oct 23, 2026
e522d49
chore: merge upstream/master for desktop qualification
tmt-123 Oct 25, 2026
f08ef54
fix(eval): refresh nested lock package metadata
tmt-123 Oct 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,12 +58,13 @@ e2e-test: ## Run CLI to Client SDK to Server end-to-end tests.

.PHONY: evaluation-unit-test
evaluation-unit-test: ## Run the evaluation project's unit tests, the work-continuity benchmark included.
@uv sync --project evaluation --frozen
@uv run --project evaluation pytest -c evaluation/pyproject.toml \
@uv sync --project evaluation --frozen --extra capture-ownership
@uv run --project evaluation --extra capture-ownership pytest -c evaluation/pyproject.toml \
evaluation/artifact_evolution/tests \
evaluation/coding/swebench_pro/tests/unit \
evaluation/coding/work_continuity/tests/unit \
evaluation/memory/longmemeval_v2/tests -m "not live" -q
evaluation/memory/longmemeval_v2/tests \
evaluation/memory/capture_ownership/tests -m "not live" -q

.PHONY: code-seekdb-test
code-seekdb-test: ## Exercise native code indexing against a real embedded seekdb instance.
Expand Down
12 changes: 7 additions & 5 deletions evaluation/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Python environment and build configuration.
| Directory | Purpose | Suites |
| --- | --- | --- |
| [`coding/`](coding/README.md) | Coding-task completion and continuation quality | [SWE-bench Pro](coding/swebench_pro/README.md), [Work continuity](coding/work_continuity/README.md) |
| [`memory/`](memory/README.md) | Memory retrieval, answer quality, and write-gate evidence | [LoCoMo](memory/locomo/README.md), [LoCoMo-Plus](memory/locomo_plus/README.md), [LongMemEval-V2](memory/longmemeval_v2/README.md), [Memory write gate](memory/write_gate/README.md) |
| [`memory/`](memory/README.md) | Memory retrieval, answer quality, and write-gate evidence | [LoCoMo](memory/locomo/README.md), [LoCoMo-Plus](memory/locomo_plus/README.md), [LongMemEval-V2](memory/longmemeval_v2/README.md), [Memory write gate](memory/write_gate/README.md), [Capture ownership](memory/capture_ownership/README.md) |
| [`performance/`](performance/README.md) | Load, latency, storage growth, and capacity limits | None yet |
| [`skills/`](skills/README.md) | Agent instruction routing and authorization regressions | [Claude Code skill-up](skills/skill-up/README.md), [ZCode native CLI](zcode_guidance/README.md) |
| [`artifact_evolution/`](artifact_evolution/README.md) | Evaluation-owned Dream recording and faithful replay | [Artifact evolution](artifact_evolution/README.md) |
Expand Down Expand Up @@ -61,6 +61,7 @@ uv sync --project evaluation --locked
uv run --project evaluation swebench-pro --help
uv run --project evaluation work-continuity --help
uv run --project evaluation longmemeval-v2 --help
uv run --project evaluation --extra capture-ownership capture-ownership --help

# Memory quality: inspect a bundled, provider-free smoke plan.
uv run python -m evaluation.memory.locomo --help
Expand Down Expand Up @@ -94,10 +95,11 @@ uv run --directory evaluation ty check
uv build --project evaluation
```

The `powercontext-eval` distribution contains four packages: `powercontext_eval_artifact_evolution`,
`powercontext_eval_swebench_pro`, `powercontext_eval_work_continuity`, and `powercontext_eval_longmemeval_v2`.
Editable installs and wheels expose the independent `artifact-evolution`, `swebench-pro`, `work-continuity`, and
`longmemeval-v2` commands and their suite modules.
The `powercontext-eval` distribution contains five packages: `powercontext_eval_artifact_evolution`,
`powercontext_eval_swebench_pro`, `powercontext_eval_work_continuity`, `powercontext_eval_longmemeval_v2`, and
`powercontext_eval_capture_ownership`. Editable installs and wheels expose the independent `artifact-evolution`,
`swebench-pro`, `work-continuity`, `longmemeval-v2`, and `capture-ownership` commands and their suite modules.
Capture ownership requires the `capture-ownership` extra shown above.
LoCoMo behavior tests remain under [`tests/`](../tests/), including [`tests/evaluation/`](../tests/evaluation/).
Skill-up and ZCode guidance keep their own validation tests. The [Bub replay harness](../e2e/bub/README.md) and
[product E2E tests](../tests/e2e/) cover cross-component acceptance.
Expand Down
1 change: 1 addition & 0 deletions evaluation/memory/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ the shared `evaluation/` environment.
| [LongMemEval-V2](longmemeval_v2/README.md) | Long-horizon memory retrieval, prepared context, reader answers, and official scoring. |
| [LoCoMo-Plus](locomo_plus/README.md) | Factual and cognitive memory evaluation with bounded smoke and full profiles. |
| [Memory write gate](write_gate/README.md) | Offline evidence-sufficiency calibration, held-out checks, and promotion decision. |
| [Capture ownership](capture_ownership/README.md) | WorkBuddy joined-prompt evidence ownership, candidate contamination and current-fact retention. |
| [Source poisoning observation](source_poisoning/README.md) | Versioned synthetic inputs, immutable citations, and actual host answers from a bounded live-model run. |

Add dataset-specific Memory quality suites here. Put latency, throughput, storage, and capacity measurements in
Expand Down
Loading
Loading