Repository navigation
Commit a76f811
fix(skills): make automation eval fixtures 5, 7, 8 agree with their own assertions (#26636)
<!-- claude-projects-attribution: {"github_login":"os-steve"} -->
_Requested by **Steve** · [project
thread](https://claude.ai/code/project/chan_01FdL1XeUyvVNxLreh3F4Qi9?thread=cmsg_01FdL1XeUyvVNxLreh3F4Qi955AXk4ApxYYetM1MwNFRyh)_
Fixes #26515
Clause-②: no
Metadata-protocol: none
## 维护者速读
- 改了什么:`objectstack-automation` 技能的评测文件里 5、7、8 三条理想答案改写到与各自的断言一致;token
棘轮行按实测抬到 2734。
- 为什么改:三条理想答案自相矛盾(5 把禁写的错误标签字面写进了答案;7 缺它要求的 `type: 'position'` 审批人并写了禁写的
positions 检查;8 缺它要求的 `type: 'screen'`),按断言评分时正确答案会被判错。
- 风险与回滚:只改评测文本、一行棘轮上限和一个 `@objectstack/skills` patch changeset,不碰
SKILL.md 教学与运行时;回滚即 revert 本 PR。
- 先看:`skills/objectstack-automation/evals/flows-triggers-approvals.json`
的三个 `expected_output` hunk,再看 `scripts/check-skills-token-ratchet.mjs`
的棘轮行注释。
- 两分钟验证:在 PR 分支跑下方 Verification 第一条一行脚本,预期输出 `11 fixtures, 0
contradictions`。
- 需要你做的:受管面(`skills/**`,Tier H),需 GOVERNED_APPROVERS 的一次 APPROVED
review;之后席位落地。
## What changes
Three ideal answers in
`skills/objectstack-automation/evals/flows-triggers-approvals.json`
contradicted the assertions that grade them. Each fixture was judged
against the teaching on `origin/main` (SKILL.md § *Failure routing &
runAs*, § *Who may START a flow*, § *Flow Types*, and
`references/state-machines-and-approvals.md` § Approver Types); in all
three the teaching and the assertions agree, and the ideal answer was
the side that was wrong, so only the ideal answers move:
- **eval 5** — the answer spelled the negated error-label string
verbatim. It now says it in words (a bare error label routes nothing)
and keeps `type: 'fault'` as the routing form.
- **eval 7** — the answer never spelled the `type: 'position'` approver
its assertions require and spelled the negated positions check verbatim.
It now names `approvers: [{ type: 'position', value: 'finance_manager'
}]` (the prompt's position) and says the negation in words.
- **eval 8** — the answer never spelled the `type: 'screen'` flow kind
its assertions require. It now opens with the flow kind.
No fixture was added or removed; the file keeps 124 lines (net 0 lines
in `skills/**`). No `must_not_contain` string appears inside any ideal
answer.
**Token ratchet (`scripts/check-skills-token-ratchet.mjs`):** the row
for this file moves 2724 → 2734, pinned at the measurement, with the
house-style comment. Paid down first: evals 5 and 7 were shortened by 25
and 7 bytes; the `approvers` line eval 7 owed costs more than that, so
the net is +41 bytes (10893 → 10934), ceil(10934 / 4) = 2734. Headroom
was 3 bytes, 0 tokens.
Readings (lines; tokens as the ratchet counts them, `ceil(utf8 bytes /
4)`):
| scope | lines before → after | tokens before → after |
|:--|:--|:--|
| `evals/flows-triggers-approvals.json` (the edited file) | 124 → 124 |
2724 → 2734 |
| whole package `skills/objectstack-automation/**` (6 tracked files) |
1185 → 1185 | 17707 → 17717 |
## Why
A shipped skill's evals grade a correct answer wrong when the ideal
answer fails its own `must_contain` / `must_not_contain` set. Fixing the
ideal answers restores the evals as a usable grader without changing
what the skill teaches.
## Tests
No test file changed; no forced test edits. The eval file's own
consistency check is the acceptance (below).
## Acceptance notes
- The card's premise was re-checked on `origin/main` 7c34e5d (after the
same-day churn f986702 on this package): the file now holds 11 fixtures
(eval 11 landed with #26502), and the parse-and-check found exactly the
four contradictions the card names (5: negated string present; 7: `type:
'position'` missing and negated string present; 8: `type: 'screen'`
missing). Ids 1–4, 6, 9–11 clean.
- `skills/**` is a Tier H governed surface: this PR stays draft until a
`GOVERNED_APPROVERS` approval.
- Changeset: `.changeset/26515-automation-eval-fixtures.md`, a `patch`
for `@objectstack/skills` — the catalog ships as that package
(`packages/skills/scripts/sync-catalog.mjs` copies `skills/**` into its
`dist`), so the rewritten fixtures publish; the `skip-changeset` label
first applied at creation was removed on the seat's rework (head
72cb4e2).
## Verification
Head `72cb4e28` (5c31edd = the fix; 72cb4e2 adds only the changeset).
Parse-and-check of every fixture in the file (the card's acceptance
one-liner):
```bash
node -e '
const f="skills/objectstack-automation/evals/flows-triggers-approvals.json";
const j=JSON.parse(require("fs").readFileSync(f,"utf8"));let bad=0;
for(const e of j.evals){const o=e.expected_output;
for(const s of e.assertions.must_contain||[]) if(!o.includes(s)){bad++;console.log(`id ${e.id}: must_contain missing: ${s}`)}
for(const s of e.assertions.must_not_contain||[]) if(o.includes(s)){bad++;console.log(`id ${e.id}: must_not_contain present: ${s}`)}}
console.log(`${j.evals.length} fixtures, ${bad} contradictions`)'
```
<details><summary>Output on origin/main 7c34e5d (before) and on
5c31edd (after)</summary>
```text
# before
id 5: must_not_contain present: label: 'error'
id 7: must_contain missing: type: 'position'
id 7: must_not_contain present: current_user.positions
id 8: must_contain missing: type: 'screen'
11 fixtures, 4 contradictions
# after
11 fixtures, 0 contradictions
```
</details>
<details><summary>Skill gates (all exit 0)</summary>
```text
pnpm --filter @objectstack/spec check:skill-docs :: exit 0 ✅ Skill docs in sync
pnpm --filter @objectstack/spec check:skill-refs :: exit 0 ✅ 9 generated files in sync with packages/spec
pnpm --filter @objectstack/spec check:skill-examples :: exit 0 ✅ 266 prose examples type-check across 3 surface(s)
node scripts/check-skills-token-ratchet.mjs :: exit 0 ✓ 81 authored bundle file(s) within their ceilings
```
</details>
<details><summary>Derived gate families (dispatch-gates --commands, 49
families) and --ran reconciliation</summary>
All 49 derived families run locally on head 5c31edd, each exit code
captured before any pipe; `dispatch-gates --ran` reconciliation: `✓
dispatch-gates --ran: 49 derived famil(ies) accounted for — 49 run, 0
NOT-MEASURED (a DERIVED zero — all 49 recorded an exit code and none of
them is 3)`. `check:pm-dispatch-gates` (fast tier, ~8 min): exit 0, `✓
dispatch-gates self-test (fast tier): 1935 cases pass; 7 slow section(s)
deferred.`.
On 72cb4e2 the derivation grows to 60 families (the changeset adds 11:
the four changeset gates with `--base origin/main` and their self-tests,
`check:published-files`, `check:closed-ceremony-text`,
`check:pm-changeset-deadline-census`, the two release-tooling
self-tests); the 11 and the token ratchet ran on 72cb4e2, all exit 0;
`--ran` on the combined record: `✓ dispatch-gates --ran: 60 derived
famil(ies) accounted for — 60 run, 0 NOT-MEASURED (a DERIVED zero — all
60 recorded an exit code and none of them is 3)`.
</details>
## Human Minimum
- [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic.
(not touched)
- [x] 2. Every name I added uses the casing for its kind.
- [x] 3. No formatter ran; `pnpm lint` passes. (CI's `Lint & Repo Gates`
runs it; no formatter ran)
- [x] 4. One issue, one branch off an up-to-date `main`, one pull
request.
- [x] 5. A change that publishes carries a changeset, and both
declaration lines are filled in here and in the changeset.
(`.changeset/26515-automation-eval-fixtures.md`, patch
`@objectstack/skills`)
- [x] 6. `packages/spec` touched: its generated artifacts are
regenerated and committed. (not touched)
- [x] 7. No hand edits to `content/docs/references/`,
`packages/*/CHANGELOG.md` or `content/docs/releases/`.
- [x] 8. Title, body and comments are in English.
- [x] The *Before Submitting* list in CONTRIBUTING.md is done.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01ADHkmT8qQiLPD3BRTdVxGH
---
_Generated by [Claude
Code](https://claude.ai/code/session_01ADHkmT8qQiLPD3BRTdVxGH)_
---------
Co-authored-by: os-steve <steve@objectstack.ai>
Co-authored-by: Claude <noreply@anthropic.com>1 parent 498e8f4 commit a76f811
3 files changed
Lines changed: 25 additions & 4 deletions
File tree
- .changeset
- scripts
- skills/objectstack-automation/evals
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
829 | 829 | | |
830 | 830 | | |
831 | 831 | | |
832 | | - | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
833 | 842 | | |
834 | 843 | | |
835 | 844 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | | - | |
| 47 | + | |
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
67 | | - | |
| 67 | + | |
68 | 68 | | |
69 | 69 | | |
70 | 70 | | |
| |||
74 | 74 | | |
75 | 75 | | |
76 | 76 | | |
77 | | - | |
| 77 | + | |
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
| |||
0 commit comments