Skip to content

Commit a76f811

Browse files
objectstack-fleet[bot]os-steveclaude
authored
fix(skills): make automation eval fixtures 5, 7, 8 agree with their own assertions (#26636)
<!-- claude-projects-attribution: {"github_login":"os-steve"} --> _Requested by **Steve** · [project thread](https://claude.ai/code/project/chan_01FdL1XeUyvVNxLreh3F4Qi9?thread=cmsg_01FdL1XeUyvVNxLreh3F4Qi955AXk4ApxYYetM1MwNFRyh)_ Fixes #26515 Clause-②: no Metadata-protocol: none ## 维护者速读 - 改了什么:`objectstack-automation` 技能的评测文件里 5、7、8 三条理想答案改写到与各自的断言一致;token 棘轮行按实测抬到 2734。 - 为什么改:三条理想答案自相矛盾(5 把禁写的错误标签字面写进了答案;7 缺它要求的 `type: 'position'` 审批人并写了禁写的 positions 检查;8 缺它要求的 `type: 'screen'`),按断言评分时正确答案会被判错。 - 风险与回滚:只改评测文本、一行棘轮上限和一个 `@objectstack/skills` patch changeset,不碰 SKILL.md 教学与运行时;回滚即 revert 本 PR。 - 先看:`skills/objectstack-automation/evals/flows-triggers-approvals.json` 的三个 `expected_output` hunk,再看 `scripts/check-skills-token-ratchet.mjs` 的棘轮行注释。 - 两分钟验证:在 PR 分支跑下方 Verification 第一条一行脚本,预期输出 `11 fixtures, 0 contradictions`。 - 需要你做的:受管面(`skills/**`,Tier H),需 GOVERNED_APPROVERS 的一次 APPROVED review;之后席位落地。 ## What changes Three ideal answers in `skills/objectstack-automation/evals/flows-triggers-approvals.json` contradicted the assertions that grade them. Each fixture was judged against the teaching on `origin/main` (SKILL.md § *Failure routing & runAs*, § *Who may START a flow*, § *Flow Types*, and `references/state-machines-and-approvals.md` § Approver Types); in all three the teaching and the assertions agree, and the ideal answer was the side that was wrong, so only the ideal answers move: - **eval 5** — the answer spelled the negated error-label string verbatim. It now says it in words (a bare error label routes nothing) and keeps `type: 'fault'` as the routing form. - **eval 7** — the answer never spelled the `type: 'position'` approver its assertions require and spelled the negated positions check verbatim. It now names `approvers: [{ type: 'position', value: 'finance_manager' }]` (the prompt's position) and says the negation in words. - **eval 8** — the answer never spelled the `type: 'screen'` flow kind its assertions require. It now opens with the flow kind. No fixture was added or removed; the file keeps 124 lines (net 0 lines in `skills/**`). No `must_not_contain` string appears inside any ideal answer. **Token ratchet (`scripts/check-skills-token-ratchet.mjs`):** the row for this file moves 2724 → 2734, pinned at the measurement, with the house-style comment. Paid down first: evals 5 and 7 were shortened by 25 and 7 bytes; the `approvers` line eval 7 owed costs more than that, so the net is +41 bytes (10893 → 10934), ceil(10934 / 4) = 2734. Headroom was 3 bytes, 0 tokens. Readings (lines; tokens as the ratchet counts them, `ceil(utf8 bytes / 4)`): | scope | lines before → after | tokens before → after | |:--|:--|:--| | `evals/flows-triggers-approvals.json` (the edited file) | 124 → 124 | 2724 → 2734 | | whole package `skills/objectstack-automation/**` (6 tracked files) | 1185 → 1185 | 17707 → 17717 | ## Why A shipped skill's evals grade a correct answer wrong when the ideal answer fails its own `must_contain` / `must_not_contain` set. Fixing the ideal answers restores the evals as a usable grader without changing what the skill teaches. ## Tests No test file changed; no forced test edits. The eval file's own consistency check is the acceptance (below). ## Acceptance notes - The card's premise was re-checked on `origin/main` 7c34e5d (after the same-day churn f986702 on this package): the file now holds 11 fixtures (eval 11 landed with #26502), and the parse-and-check found exactly the four contradictions the card names (5: negated string present; 7: `type: 'position'` missing and negated string present; 8: `type: 'screen'` missing). Ids 1–4, 6, 9–11 clean. - `skills/**` is a Tier H governed surface: this PR stays draft until a `GOVERNED_APPROVERS` approval. - Changeset: `.changeset/26515-automation-eval-fixtures.md`, a `patch` for `@objectstack/skills` — the catalog ships as that package (`packages/skills/scripts/sync-catalog.mjs` copies `skills/**` into its `dist`), so the rewritten fixtures publish; the `skip-changeset` label first applied at creation was removed on the seat's rework (head 72cb4e2). ## Verification Head `72cb4e28` (5c31edd = the fix; 72cb4e2 adds only the changeset). Parse-and-check of every fixture in the file (the card's acceptance one-liner): ```bash node -e ' const f="skills/objectstack-automation/evals/flows-triggers-approvals.json"; const j=JSON.parse(require("fs").readFileSync(f,"utf8"));let bad=0; for(const e of j.evals){const o=e.expected_output; for(const s of e.assertions.must_contain||[]) if(!o.includes(s)){bad++;console.log(`id ${e.id}: must_contain missing: ${s}`)} for(const s of e.assertions.must_not_contain||[]) if(o.includes(s)){bad++;console.log(`id ${e.id}: must_not_contain present: ${s}`)}} console.log(`${j.evals.length} fixtures, ${bad} contradictions`)' ``` <details><summary>Output on origin/main 7c34e5d (before) and on 5c31edd (after)</summary> ```text # before id 5: must_not_contain present: label: 'error' id 7: must_contain missing: type: 'position' id 7: must_not_contain present: current_user.positions id 8: must_contain missing: type: 'screen' 11 fixtures, 4 contradictions # after 11 fixtures, 0 contradictions ``` </details> <details><summary>Skill gates (all exit 0)</summary> ```text pnpm --filter @objectstack/spec check:skill-docs :: exit 0 ✅ Skill docs in sync pnpm --filter @objectstack/spec check:skill-refs :: exit 0 ✅ 9 generated files in sync with packages/spec pnpm --filter @objectstack/spec check:skill-examples :: exit 0 ✅ 266 prose examples type-check across 3 surface(s) node scripts/check-skills-token-ratchet.mjs :: exit 0 ✓ 81 authored bundle file(s) within their ceilings ``` </details> <details><summary>Derived gate families (dispatch-gates --commands, 49 families) and --ran reconciliation</summary> All 49 derived families run locally on head 5c31edd, each exit code captured before any pipe; `dispatch-gates --ran` reconciliation: `✓ dispatch-gates --ran: 49 derived famil(ies) accounted for — 49 run, 0 NOT-MEASURED (a DERIVED zero — all 49 recorded an exit code and none of them is 3)`. `check:pm-dispatch-gates` (fast tier, ~8 min): exit 0, `✓ dispatch-gates self-test (fast tier): 1935 cases pass; 7 slow section(s) deferred.`. On 72cb4e2 the derivation grows to 60 families (the changeset adds 11: the four changeset gates with `--base origin/main` and their self-tests, `check:published-files`, `check:closed-ceremony-text`, `check:pm-changeset-deadline-census`, the two release-tooling self-tests); the 11 and the token ratchet ran on 72cb4e2, all exit 0; `--ran` on the combined record: `✓ dispatch-gates --ran: 60 derived famil(ies) accounted for — 60 run, 0 NOT-MEASURED (a DERIVED zero — all 60 recorded an exit code and none of them is 3)`. </details> ## Human Minimum - [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic. (not touched) - [x] 2. Every name I added uses the casing for its kind. - [x] 3. No formatter ran; `pnpm lint` passes. (CI's `Lint & Repo Gates` runs it; no formatter ran) - [x] 4. One issue, one branch off an up-to-date `main`, one pull request. - [x] 5. A change that publishes carries a changeset, and both declaration lines are filled in here and in the changeset. (`.changeset/26515-automation-eval-fixtures.md`, patch `@objectstack/skills`) - [x] 6. `packages/spec` touched: its generated artifacts are regenerated and committed. (not touched) - [x] 7. No hand edits to `content/docs/references/`, `packages/*/CHANGELOG.md` or `content/docs/releases/`. - [x] 8. Title, body and comments are in English. - [x] The *Before Submitting* list in CONTRIBUTING.md is done. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01ADHkmT8qQiLPD3BRTdVxGH --- _Generated by [Claude Code](https://claude.ai/code/session_01ADHkmT8qQiLPD3BRTdVxGH)_ --------- Co-authored-by: os-steve <steve@objectstack.ai> Co-authored-by: Claude <noreply@anthropic.com>
1 parent 498e8f4 commit a76f811

3 files changed

Lines changed: 25 additions & 4 deletions

File tree

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
'@objectstack/skills': patch
3+
---
4+
5+
fix(skills): objectstack-automation eval fixtures 5, 7 and 8 agree with their own assertions
6+
7+
Clause-②: no
8+
Metadata-protocol: none
9+
10+
- **Before.** Three ideal answers in `evals/flows-triggers-approvals.json` failed the assertions that grade them: eval 5 spelled the negated error-label string inside its answer, eval 7 never spelled the `type: 'position'` approver it requires and spelled the negated positions check, eval 8 never spelled the `type: 'screen'` flow kind it requires. Graded against those assertions, a correct answer scored wrong.
11+
- **Now.** The three ideal answers say each negation in words, eval 7 names `approvers: [{ type: 'position', value: 'finance_manager' }]`, and eval 8 opens with the flow kind. No fixture added or removed; the assertions and the skill's teaching are unchanged.
12+
- No schema, key or behaviour changes.

‎scripts/check-skills-token-ratchet.mjs‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -829,7 +829,16 @@ export const CEILINGS = new Map([
829829
// = 2724; headroom was 3 bytes, 0 tokens, so the ceiling moves +297, pinned
830830
// AT the measurement. Same maintainer ruling as the SKILL.md row above
831831
// (2026-10-27, tap A verbatim and untranslated): 「A:先削重复,余额按实测提高」.
832-
['skills/objectstack-automation/evals/flows-triggers-approvals.json', 2724],
832+
// 10893 -> 10934 (#26515): no new fixture — three ideal answers rewritten
833+
// to agree with their own assertions (eval 5 spelled the negated error-label
834+
// string, eval 7 lacked the `type: 'position'` approver and spelled the
835+
// negated positions check, eval 8 lacked `type: 'screen'`). Paid down first:
836+
// evals 5 and 7 shortened by 25 and 7 bytes; the `approvers` line eval 7
837+
// owed costs more than that. +41 bytes -> ceil(10934 / 4) = 2734; headroom
838+
// was 3 bytes, 0 tokens, so the ceiling moves +10, pinned AT the
839+
// measurement. Same maintainer ruling as the SKILL.md row above (2026-10-27,
840+
// tap A verbatim and untranslated): 「A:先削重复,余额按实测提高」.
841+
['skills/objectstack-automation/evals/flows-triggers-approvals.json', 2734],
833842
// 894 -> 1095 (#26179): one regression case for the field permission `when`
834843
// (the contract prompt: lock the legal terms for finance from approval on,
835844
// legal keeps the edit) — expected: `when` on the finance set's entries with

‎skills/objectstack-automation/evals/flows-triggers-approvals.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@
4444
{
4545
"id": 5,
4646
"prompt": "Add a step that calls our `pricing.recalc` function to compute the discount and saves it on the quote, then posts the quote to our tax service over HTTP; if the tax call fails, route to a handler node instead of failing the run.",
47-
"expected_output": "Emits a `script` node with `config.function: 'pricing.recalc'` (registered via `defineStack({ functions })`), `inputs`, and `outputVariable: 'recalc'`; the function is a pure compute step, so a following `update_record` persists it with `fields: { discount: { dialect: 'cel', source: 'recalc.discount' } }`. The `http` node sets `timeoutMs`. Failure routing is a `type: 'fault'` edge — a `label: 'error'` alone routes nothing.",
47+
"expected_output": "Emits a `script` node with `config.function: 'pricing.recalc'` (registered via `defineStack({ functions })`), `inputs`, and `outputVariable: 'recalc'`; a pure compute step, so a following `update_record` persists it with `fields: { discount: { dialect: 'cel', source: 'recalc.discount' } }`. The `http` node sets `timeoutMs`. Failure routing is a `type: 'fault'` edge; a bare error label routes nothing.",
4848
"files": [],
4949
"assertions": {
5050
"must_contain": ["function:", "outputVariable", "type: 'fault'", "timeoutMs", "update_record", "dialect: 'cel'"],
@@ -64,7 +64,7 @@
6464
{
6565
"id": 7,
6666
"prompt": "An expense approval only holders of the `expense.approve_request` capability may start; route to the `finance_manager` position, and the submitter must never approve their own expense.",
67-
"expected_output": "Flow-level `requiredPermissions: ['expense.approve_request']` (checked at every door that starts the flow, not a `current_user.positions` check inside it); the `approval` node sets `excludeSubmitter: true`, and its `onEmptyApprovers` is anything but `auto_approve`, which is refused beside it.",
67+
"expected_output": "Flow-level `requiredPermissions: ['expense.approve_request']`, checked at every door that starts the flow (never a run-user positions check inside it); the `approval` node names `approvers: [{ type: 'position', value: 'finance_manager' }]`, sets `excludeSubmitter: true`, and its `onEmptyApprovers` is anything but `auto_approve`, refused beside it.",
6868
"files": [],
6969
"assertions": {
7070
"must_contain": ["requiredPermissions", "excludeSubmitter: true", "type: 'position'"],
@@ -74,7 +74,7 @@
7474
{
7575
"id": 8,
7676
"prompt": "A screen flow: the user uploads the signed contract PDF on the last step, then the flow creates the `contract` record with the file in its `signed_copy` field.",
77-
"expected_output": "A `screen` field `{ name: 'signed_copy', type: 'file', required: true }`; the screen resumes with the uploaded `sys_file` id bound to `signed_copy`, and a `create_record` after the last screen writes `fields: { signed_copy: { dialect: 'cel', source: 'signed_copy' } }`, which claims the file for the record.",
77+
"expected_output": "A `type: 'screen'` flow whose last `screen` carries `{ name: 'signed_copy', type: 'file', required: true }` and resumes with the uploaded `sys_file` id bound to `signed_copy`; a `create_record` after it writes `fields: { signed_copy: { dialect: 'cel', source: 'signed_copy' } }`, which claims the file for the record.",
7878
"files": [],
7979
"assertions": {
8080
"must_contain": ["type: 'screen'", "type: 'file'", "create_record", "signed_copy"],

0 commit comments

Comments
 (0)