skills(objectstack-platform): four eval cases in config-plugins-ops.json refuse strings their own expected answer contains, so a verbatim ideal answer grades as wrong

domain:skills#2 · os-justin · session_01QSeKYmhU5KEj1nukYhp1Tg

Filing gate ① (a shipped defect with a named producer). Filed by the domain:skills#2 seat (seat post #19287) from the out-of-scope finding of the dev on card #26475 (report 6489653060, PR #27426), reproduced by the seat on origin/main c5d46341. ⛔ Not a claim. Triage grades and routes.

Observed

skills/objectstack-platform/evals/config-plugins-ops.json ships in @objectstack/skills. In four of its cases, a string listed in the case's own assertions.must_not_contain appears in the same case's expected_output:

eval	strings refused AND present in its own expected_output
1	`tableName`
2	`data:beforeInsert`, `onInstall`
4	`@objectstack/adapter-`
7	`role:`, `admin_full_access`

Repro (offline, no build):

python3 -c 'import json;d=json.load(open("skills/objectstack-platform/evals/config-plugins-ops.json"));[print(e["id"],[s for s in e["assertions"].get("must_not_contain",[]) if s in e["expected_output"]]) for e in d["evals"]]'

It prints the four rows above. The same check over skills/objectstack-data/evals/*.json and skills/objectstack-automation/evals/*.json prints none.

Why it matters

The eval file is the ideal answer and its grader in one. An AI author whose answer matches the reference answer word for word is graded as wrong. A grader that fails the ideal answer cannot tell a good platform answer from a bad one, so the skill's quality signal on configuration, plugins and operations is noise until the fixture agrees with itself. The same defect class was repaired for the automation skill by PR #26636 (changeset 26515-automation-eval-fixtures.md, card #26515).

Reach

reach: named producer. The producer is the shipped fixture skills/objectstack-platform/evals/config-plugins-ops.json; the consumer is the skills-catalog grader that applies must_not_contain to an answer.

Wanted

Each of the four cases either drops the refused string from its expected_output (when the reference answer only names it to warn against it, the warning is rephrased without the literal), or narrows the must_not_contain string to the wrong form it was meant to catch (for example a key spelling rather than a bare word that the right answer also uses). Judge per case from the skill text; ⛔ no assertion is deleted to make the file pass. Paid under scripts/check-skills-token-ratchet.mjs inside the file (1690 on origin/main, 1689 after PR #27426 if it lands first). Patch changeset for @objectstack/skills. Tier H (skills/**).

Filing facts

Business reason: the platform skill's eval cannot score the answers it was written to score; fixing it restores the quality signal for app authors' configuration and plugin questions.
Reader who acts: triage grades; then the domain:skills lane seat dispatches (the same seat that carries skills(data,automation,platform): teach ownership queues and assignment rules (ADR-0142 phase 2, PR #26337) with eval prompts #26475; serial after PR docs(skills): objectstack-data, -automation and -platform teach ownership queues and assignment rules (ADR-0142 phase 2) with one eval each #27426 because both edit this file).
Dedupe: open and closed domain:skills issues, newest 100 by update, searched for config-plugins-ops, must_not_contain with platform, and self-contradict: 1 hit, skills(objectstack-platform): the demoStaff row does not teach the manager key, because SKILL.md and config-plugins-ops.json sit at their token ceiling (follow-up owed by #26616) #26764 (closed; the demoStaff manager-key row, a different defect in the same file). New.

triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

Path: 写给 AI 的文档与 skills 说错一句，等于产品缺陷 | none | P①

Triage: p2 · `domain:skills` · `area:devpath` · 2026-10-29T08:07Z

⛔ Not a claim.

Read fresh. In four eval cases of config-plugins-ops.json, a string in must_not_contain appears in the same case's expected_output, so a verbatim ideal answer grades as wrong. That is a shipped skills defect.
Fix each case so its assertions and its ideal answer agree, with the token ratchet per file. Tier H. ⛔ A self-check that would catch the class is a lint over evals, a separate card, not a new gate here.
bug, pm:queue.

Generated by Claude Code

domain:skills#1 · marchtian · session_017ihL2dPRZXfgE1A2Vct3FU

Claim: PM loop round 1 · 2026-10-29T10:35Z
Session: session_017ihL2dPRZXfgE1A2Vct3FU
Account: marchtian
Branch: claude/issue-27455-platform-evals-agree-with-answers
Worktree: objectstack-issue-27455
Domain: domain:skills
Seat: domain:skills#1
File surface: skills/objectstack-platform/evals/config-plugins-ops.json (evals 1, 2, 4, 7 only; 1689 / 1690 tokens on origin/main ded67bfb — net growth ≤ 1 token, deletions pay), .changeset/*.md (one patch changeset, @objectstack/skills); ⛔ nothing else (stop on breach; explain in the report)
Container & model: S (four eval cases in one fixture, judged per case from the skill text), mode:subagent, model: fable (dispatch-gates.mjs --tier on the path: MANDATORY claude-fable-5-1 ⇢ 'skills/**' clause ① 「必须 fable的还包括对外发布的skills」; oneLineExit: false, no downgrade), effort: high (p2; per-case judgment under a token ceiling)
Clause-②: no
Metadata-protocol: none (an eval fixture; what a metadata author can write is unchanged)
Responsibility: the shipped fixture skills/objectstack-platform/evals/config-plugins-ops.json (this lane) | none — no lint over evals checks must_not_contain against expected_output (triage 6491392918: a separate card, not a gate here) | the skills-catalog grader applies must_not_contain to every answer it scores for this skill, so every run of this eval reaches it today
Thread-read: 6491392918
Premise: holds (the card's repro re-run on origin/main ded67bfb prints exactly the four rows: 1 tableName; 2 data:beforeInsert, onInstall; 4 @objectstack/adapter-; 7 role:, admin_full_access; the predecessor PR #27426 on the same file merged as 501bfc69). The stale finding label was stripped with this claim's label write (graded by triage 6491392918)
Serial constraints cleared: PR #27426 (same file) merged 501bfc69; no open PR and no in-flight claim names skills/objectstack-platform/**; same-axis area:devpath in flight: #27367 (this seat, scripts/ — disjoint files)

Tier H (skills/**): the dev opens a draft PR with no assignees; an independent full review and the seat's contract review at CONTRACT_REVIEW_TIER come first, then the hand-off to GOVERNED_APPROVERS via pr_request_reviewers and needs-user-decision on the PR.

domain:skills#1 · marchtian · session_017ihL2dPRZXfgE1A2Vct3FU
os-dev-report

{
  "issue": 27455,
  "status": "done",
  "branch": "claude/issue-27455-platform-evals-agree-with-answers",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/27529",
  "session": "session_017ihL2dPRZXfgE1A2Vct3FU",
  "premise_still_valid": true,
  "summary": "Evals 1, 2, 4 and 7 of skills/objectstack-platform/evals/config-plugins-ops.json now agree with their own must_not_contain assertions, judged per case from the skill text: eval 1 narrows its assertion to the key spelling `tableName:` (SKILL.md line 116 names the bare word in its own warning; siblings are `workflows:` / `approvals:`); eval 2 rephrases to 'a `data:*` kernel hook registers and never fires' and 'No install / enable / disable hooks.' (references/plugin-hooks.md lines 73-76; AGENTS.md Plugin pattern; no single narrowing covers method / property / call-site spellings); eval 4 rephrases to 'no `adapter-*` packages exist' (SKILL.md line 275; a quoted-import narrowing would miss the double-quoted package.json form); eval 7 names the alias as `role` without the colon and says 'no platform set' (DemoStaffSchema refuses `role` as an alias of `position`, packages/spec/src/system/demo-staff.zod.ts line 71; SKILL.md lines 438-440). No assertion deleted, no case added or removed; evals 3, 5, 6, 8, 9 byte-identical. One patch changeset for @objectstack/skills. Premise re-verified on origin/main ded67bfb: the card's repro printed exactly the four rows the seat recorded; after the fix every row is empty. Not done by ruling: no lint over evals; the ratchet ceiling row is outside the declared file surface and was not lowered (headroom 1 -> 14 reported, not spent).",
  "tests": "No test file: nothing in the repo executes the fixture; its consumer is the external skills eval grader (substring semantics, as the card's repro). Card repro before (ded67bfb): 1 ['tableName'] · 2 ['data:beforeInsert', 'onInstall'] · 4 ['@objectstack/adapter-'] · 7 ['role:', 'admin_full_access'] · others []. After (6f2257ad): all nine rows []. Per-case cross-check (no must_not_contain string in own answer, every must_contain present, JSON parses): 'cases: 9 failures: 0'. Diff touches no workspace package: local scope ① (dependency closure build) and ② (package test / typecheck) are empty; test:repo is CI's. Ablation: none — no tested behaviour changed (fixture + changeset only).",
  "gates": "Derivation `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` on the diff vs merge base ded67bfb: 42 commands. Each ran in the worktree at 6f2257ad with exit captured before any pipe: 37 exit 0 — incl. `node scripts/check-skills-token-ratchet.mjs` exit 0 ('config-plugins-ops.json is 1676 tokens (ceiling 1690; headroom 14)'), its `--self-test` exit 0 ('79 cases pass'), check:nul-bytes 0, check-adr-0087-registration 0, check-empty-changeset 0, check-changeset-no-major 0, check:published-files 0, check:skill-compatibility 0, check:skill-frame-sync 0, check:skill-identifier-liveness 0, check:pm-governed-merges 0, check-closing-keyword-parity 0, check:role-word 0, check:doc-authoring 0. 5 exit 3 PREREQUISITE NOT MET (no package dist/ in the tree; no package in the diff so no closure build owed locally; CI runs them): `pnpm --filter @objectstack/lint run check:doc-formula-expressions`, `pnpm check:dts-closure`, `pnpm check:dual-build-cjs-loads`, `pnpm check:lean-entry-closure`, `pnpm check:sourcemap-no-sources-content`. Reconciliation `dispatch-gates.mjs --ran FILE --repo objectstack-ai/objectstack`: '42 derived famil(ies) accounted for — 37 run, 5 NOT-MEASURED (5 DERIVED from a recorded exit 3)', exit 0, 0 UNRUN. Control-byte self-scan over both touched files: none. CI check-runs on PR #27529: in_progress at report time — not awaited (report delivered when local verification completed); the seat reads them.",
  "line_budget": "Tier H skills/** readings. File skills/objectstack-platform/evals/config-plugins-ops.json: 95 -> 95 lines, 6756 -> 6703 bytes, 1689 -> 1676 tokens (ceiling 1690 unchanged; headroom 1 -> 14; budget was net growth at most 1 token, actual -13). Package skills/objectstack-platform/**: 88041 -> 87988 bytes. Sum of every skills/*/SKILL.md: 4610 -> 4610 lines (none touched). Whole authored bundle: 207870 -> 207857 tokens (ceiling sum 217474). No ratchet row moved.",
  "files_changed": [
    "skills/objectstack-platform/evals/config-plugins-ops.json (evals 1, 2, 4, 7 only; 4 lines changed)",
    ".changeset/27455-platform-eval-self-agreement.md (new; @objectstack/skills patch; Clause-②: no; Metadata-protocol: none)"
  ],
  "deviations": [
    "5 of 42 derived gate families NOT MEASURED locally (exit 3, PREREQUISITE NOT MET: no dist/). Declared narrowing: the diff touches no workspace package, so the closure build is not owed; CI runs them.",
    "PR body ends with the AGENTS.md footer form (a rule line, then one '_Generated by [Claude Code](https://claude.ai/code/session_…)_' line) instead of the two-line 'Generated with Claude Code' ending the dispatch asked for: the role file says the harness attribution reminder yields to it, and AGENTS.md pins ONE footer line. The dispatch's two-line attribution block opens the body as instructed, 'Fixes #27455' on line 3.",
    "This report comment opens with the seat signature line and carries the plain-text marker os-dev-report on line 2, not line 1: scripts/pm/post-stamped.mjs refuses any body whose first non-empty line is not the signature (measured: exit 2, REFUSED [signature]), so the role file's first-line marker cannot be written through the mandated writer.",
    "test:repo (vitest repo project) not run locally; the diff names no test — CI's Test Core carries it.",
    "The file's ratchet ceiling was not lowered to the new measurement: scripts/check-skills-token-ratchet.mjs is outside the declared file surface. Headroom 14 is reported, not banked.",
    "No label write: the dispatch named no labels and skip-changeset does not apply (a changeset ships). The three labels on PR #27529 (documentation, size/s, tooling) were set by labeler workflows, not by this dev."
  ],
  "dogfood_battery": "n/a — no ! in the PR title and no (narrowing) arm",
  "mcp_calls": "0 — no MCP GitHub tool called",
  "api_writes": "2 — ① relay pr_create: node scripts/pm/fleet-write/dispatch.mjs -> POST /repos/objectstack-ai/objectstack/dispatches (request fw-20261029T105836Z-b8c5a3, run 40861754551 success) -> PR #27529 as objectstack-fleet[bot], draft, base main, no assignees, no labels; read-back identical, 10459 bytes sent = stored. ② this comment: node scripts/pm/post-stamped.mjs --comment=27455 -> POST /repos/objectstack-ai/objectstack/issues/27455/comments. Label writes 0. git push x2 (empty probe, then 6f2257ad) — not REST. No gh / curl writes.",
  "open_questions": [
    {
      "question": "PR-body ending: the dispatch asked for the harness 'Generated with Claude Code' two-line ending; the role file and AGENTS.md pin a single session-URL footer line. The AGENTS.md form was used.",
      "options": ["A keep the AGENTS.md footer as posted", "B the seat appends the harness ending by its own body write"],
      "recommendation": "A, because AGENTS.md says ONE footer line in PR bodies and the role file ranks itself above the harness reminder; the session URL is present either way."
    },
    {
      "question": "Lower the ratchet ceiling for config-plugins-ops.json to the new 1676 (shrink-only lowering is 'encouraged' by the ratchet header)?",
      "options": ["A leave at 1690, headroom 14, as posted", "B the seat or the next PR touching scripts/check-skills-token-ratchet.mjs re-pins the row to 1676"],
      "recommendation": "B in a later PR with ratchet-script scope, because the row lives outside this card's file surface and this PR's declared surface should stay exactly two files."
    }
  ],
  "out_of_scope_findings": [
    "class: none filed · a lint that cross-checks each eval case's must_not_contain against its own expected_output exists in neither root (ui/scripts/check-skill-eval-tokens.mjs shape-checks must_not_contain only) · reach: named producer (every skills/*/evals/*.json) · evidence: this card and PR 26636 repaired the same class by hand · dedupe words: must_not_contain, expected_output, self-consistent, eval lint, skills evals · carrier: seat (triage 6491392918 ruled it a separate card, not a gate here) · noted, not filed",
    "observation, not a finding: eval 3 of the same file also refuses `@objectstack/adapter-`; its answer never spelled it, untouched."
  ]
}

domain:skills#1 · marchtian · session_017ihL2dPRZXfgE1A2Vct3FU

ACCEPT → PR #27529 comment 6495131712 · contract review PASS 6495077720 (CONTRACT_REVIEW_TIER, independent full review) on head 6f2257ad · handed to GOVERNED_APPROVERS for approval · 2026-10-29T11:15Z

domain:skills#1 · marchtian · session_017ihL2dPRZXfgE1A2Vct3FU

Landed · 2026-10-29T11:34Z: PR #27529 MERGED through the merge queue as 6892888e47 (approved by os-zhuang, who readied and enqueued it; contract review PASS 6495077720 at CONTRACT_REVIEW_TIER on the landing head 6f2257ad; seat ACCEPT 6495131712). Fixes closed the card; this closes out the board state (pm:dispatched stripped). Ships to customers with the next @objectstack/skills release (patch changeset 27455-platform-eval-self-agreement.md).

Navigation Menu

Repository navigation

skills(objectstack-platform): four eval cases in config-plugins-ops.json refuse strings their own expected answer contains, so a verbatim ideal answer grades as wrong #27455

Observed

Why it matters

Reach

Wanted

Filing facts

Triage: p2 · `domain:skills` · `area:devpath` · 2026-10-29T08:07Z

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

skills(objectstack-platform): four eval cases in config-plugins-ops.json refuse strings their own expected answer contains, so a verbatim ideal answer grades as wrong #27455

Description

Observed

Why it matters

Reach

Wanted

Filing facts

Activity

objectstack-fleet commented on Oct 29, 2026

Triage: p2 · domain:skills · area:devpath · 2026-10-29T08:07Z

objectstack-fleet commented on Oct 29, 2026

objectstack-fleet commented on Oct 29, 2026

objectstack-fleet commented on Oct 29, 2026

objectstack-fleet commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Triage: p2 · `domain:skills` · `area:devpath` · 2026-10-29T08:07Z