Skip to content

docs(skills): objectstack-ui gains an eval for a report's dateRange and exportOptions - #26774

Merged
os-zhuang merged 1 commit into
mainfrom
claude/project-thread-c67kgy
Oct 27, 2026
Merged

os-zhuang merged 1 commit into
mainfrom
claude/project-thread-c67kgy

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Requested by Steve · project thread

Fixes #26633

Clause-②: no
Metadata-protocol: none

维护者速读

  • 改了什么:objectstack-ui 技能新增第三个 eval 文件 evals/report-period-export.json,只含一条报表编写题:要求报表在 type: 'date' 的数据集维度上声明 dateRange(沿用仪表盘的预设名),并用 exportOptions 声明 csv / xlsx 下载;禁止 joined 容器、手写 runtimeFilter 日期窗、第二张预设表和自造的导出权限键。evals/README.md 列出该文件;token 棘轮两行按实测钉住;@objectstack/skills 补丁 changeset。
  • 为什么改:feat(spec): a report declares its period (dateRange) and the export of its table (exportOptions) #26628 给报表加了这两个键,但两个既有 eval 文件都在 token 上限(余量 0),没有任何 eval 检查 agent 会用这两个键;本卡是它欠下的 eval。
  • 风险与回滚:只加一个 JSON 夹具、三行 README 和两行棘轮;不改任何 schema、键或运行时行为。回滚即 revert 本 PR。
  • 先看:skills/objectstack-ui/evals/report-period-export.json(题面与断言)、scripts/check-skills-token-ratchet.mjs(README 行 289 → 335、新文件行 457,算式在注释里)、skills/objectstack-ui/evals/README.md。
  • 两分钟验证:node scripts/check-skills-token-ratchet.mjs,预期四个 objectstack-ui/evals 文件全部 ✓ 且余量 0;node -e "JSON.parse(require('fs').readFileSync('skills/objectstack-ui/evals/report-period-export.json','utf8'))" 无输出即解析通过。
  • 需要你做的:批准本 PR(skills/** 受管面,Tier H)。

What changes

  • New eval file skills/objectstack-ui/evals/report-period-export.json (same fixture shape as its two siblings, one eval). The prompt names a dataset with a type: 'date' close_date dimension and asks for a summary report that opens on the current quarter, lets the viewer pick another period, and lets users who may already export the base object download the table as CSV or Excel. The assertions require defineReport, dateRange, dimension: 'close_date', defaultRange: 'this_quarter', exportOptions, 'csv', 'xlsx' and allowExport (the existing permission-set axis, named in words), and refuse type: 'joined' / blocks: (either key on a joined container), close_date: { (a hand-rolled runtimeFilter window on the date column), presets: (a second preset table), permissions.export / canExport / allowExport: true (an invented report permission key) and 'pdf'. Every key in the prompt, the expected output and the assertions is read off packages/spec/src/ui/report.zod.ts and list-view-export-options.ts on origin/main dc9d1ff (dateRange: { dimension?, defaultRange, allowCustomRange }, exportOptions: { formats?, includeHeaders?, fileNamePrefix? }, formats csv / xlsx, preset this_quarter from DATE_RANGE_PRESETS); none is invented.
  • Why a third file. Both sibling eval files sit at headroom 0, and the open draft PR feat(spec): a dataset measure declares its polarity, so a lower-is-better KPI draws a rising delta as bad news #26739 (another lane) edits analytics-inline-vs-dataset.json and its ratchet row, so the prompt gets a new file and a new row instead of a serial conflict.
  • skills/objectstack-ui/evals/README.md lists the new file in one three-line entry, between its two siblings.
  • scripts/check-skills-token-ratchet.mjs: a new row for the eval file pinned AT the measurement (1828 bytes → 457 tokens, headroom 0) and the README row moved 289 → 335 (1143 → 1339 bytes), each with its arithmetic comment and the standing maintainer ruling quoted verbatim beside the sibling row: 「A:先削重复,余额按实测提高」. Paid down first as the ruling asks: the new file is fixtures and restates nothing; the README's four sections share no sentence; so 0 bytes were available to cut and the whole addition is raised to the measurement, not past it. No other row moves.
  • Changeset .changeset/26633-report-period-export-eval.md: @objectstack/skills patch, both declaration lines at line start (the published skills bundle is built from skills/**).

Skills readings (skills/** touched; before = origin/main dc9d1ff, after = head):

Reading Before After
skills/objectstack-ui/evals/report-period-export.json lines / tokens — (new) 16 / 457 (ceiling 457, headroom 0)
skills/objectstack-ui/evals/README.md lines / tokens 31 / 286 (ceiling 289, headroom 3) 34 / 335 (ceiling 335, headroom 0)
skills/objectstack-ui/** lines 2259 2278
all SKILL.md lines 4600 4600
whole skills/** tree lines 19191 19210
bundle total tokens (ratchet report) 213404 213910

Why

A report can now declare its period and its export (dateRange / exportOptions, PR #26628), and the skill's rule example shows dateRange, but no eval scored an agent that hand-rolls a runtimeFilter date window, invents a permissions.export bit, redeclares the preset table, or puts either key on a joined container. This file is the instrument that makes those wrong answers fail at scoring.

Tests

  • node scripts/check-skills-token-ratchet.mjs at head: 82 authored bundle file(s) within their ceilings; 10 generator-owned file(s) measured, not ratcheted, exit 0; the four objectstack-ui/evals rows each ✓ at headroom 0.
  • node scripts/check-skills-token-ratchet.mjs --self-test: 79 cases pass, exit 0.
  • The eval file parses (JSON.parse, no output).
  • No test file edited; no forced test edit.

Acceptance notes

  • The skill text teaches dateRange but not exportOptions. skills/objectstack-ui/rules/dashboards.md § Report Configuration carries the dateRange example from PR feat(spec): a report declares its period (dateRange) and the export of its table (exportOptions) #26628; exportOptions is taught only on the docs page content/docs/ui/reports.mdx § Period and export. This eval therefore asserts a key the published skill does not yet teach in its rules. Noted, not filed — the card's file surface is the eval, its README line, the ratchet rows and the changeset; the teaching line is the skills lane's call (that file sits at headroom 0 too).
  • The allowExport must-contain is a review cue as much as a scorer: an answer that explains the export gate as "the base object's export permission" without the key name scores as a miss and deserves a read, the way eval 2 of analytics-inline-vs-dataset.json documents its strict dateGranularity.
  • The eval is not run by any gate: the repo has no eval runner; check:skill-docs / check:skill-refs / check:skill-examples read SKILL.md, references/ and the rules' code fences, none the eval JSON. The gates this diff does reach are listed under Verification.

Verification

  • node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack at head 78362c8b (the final commit) derived 61 commands (51 matched by path, 2 by change kind, 8 whole-tree). All 61 ran in the worktree after that commit, each exit 0 captured before any pipe — among them node scripts/check-skills-token-ratchet.mjs and its --self-test (79 cases), pnpm check:pm-governed-merges, pnpm check:pm-dispatch-gates, pnpm check:published-files, node scripts/check-empty-changeset.mjs --base origin/main, node scripts/check-adr-0087-registration.mjs --base origin/main, node scripts/check-changeset-no-major.mjs --base origin/main, pnpm check:skill-identifier-liveness, pnpm check:skill-frame-sync, pnpm check:skill-compatibility, pnpm check:doc-authoring, pnpm check:ratchet-remedy-authority, pnpm check:nul-bytes.
  • dispatch-gates --ran: 61 derived famil(ies) accounted for — 61 run, 0 NOT-MEASURED (a DERIVED zero — all 61 recorded an exit code and none of them is 3).
  • Three families take a value from the workflow and are NOT MEASURED locally (the two shard attestations and check-test-completeness); the 11 wide-population families and the repo-wide pnpm lint are CI's runs.
  • Build closure before the gates: turbo run build --concurrency=2 --filter='./packages/**' under scripts/pm/os-verify-lock.sh, VERDICT command-exit 0 (held 8m42s).
  • The edited gate script's own test suite is its --self-test (no other test file names check-skills-token-ratchet); no package source is touched, so no package test / typecheck is owed.

Human Minimum

  • 1. Schemas start as Zod; packages/spec holds no runtime logic. (No schema touched.)
  • 2. Every name I added uses the casing for its kind.
  • 3. No formatter ran; pnpm lint passes (the repo-wide run is CI's; no source file edited).
  • 4. One issue, one branch off an up-to-date main, one pull request.
  • 5. A change that publishes carries a changeset, and both declaration lines are filled in here and in the changeset.
  • 6. packages/spec touched: n/a, not touched.
  • 7. No hand edits to content/docs/references/, packages/*/CHANGELOG.md or content/docs/releases/.
  • 8. Title, body and comments are in English.
  • The Before Submitting list in CONTRIBUTING.md is done.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ADHkmSMxfTnb6u1AsKaTcU


Generated by Claude Code

… and `exportOptions`

A third eval file, `skills/objectstack-ui/evals/report-period-export.json`,
carries one report-authoring prompt: `dateRange` on a `type: 'date'` dataset
dimension with the dashboard's preset names, `exportOptions` with `'csv'` /
`'xlsx'` under the base object's existing `allowExport`; a `joined` container,
a hand-rolled `runtimeFilter` window, a second preset table and an invented
permission key are refused. `evals/README.md` lists the file; both ratchet
rows are pinned at the measurement under the standing ruling quoted beside
them; a patch changeset for `@objectstack/skills`.

Claude-Session: https://claude.ai/code/session_01ADHkmSMxfTnb6u1AsKaTcU
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added size/s documentation Improvements or additions to documentation tooling labels Oct 27, 2026
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

席位核对

ACCEPT · domain:skills#1 · session_01ADHkmSrRBdu9MdC6fc33Uu · head 78362c8b · 2026-10-27T15:21Z

  1. 形态 — Tier H (skills/**): a NEW skills/objectstack-ui/evals/report-period-export.json (one prompt), its listing in evals/README.md (+3 lines), scripts/check-skills-token-ratchet.mjs (new row 457 pinned at the measurement; README row 289 → 335 at the measurement, both with arithmetic and the standing ruling 「A:先削重复,余额按实测提高」 quoted), patch changeset for @objectstack/skills.
  2. 前提实测 — PR feat(spec): a report declares its period (dateRange) and the export of its table (exportOptions) #26628 is on main; every key the eval asserts is read off main: report.zod.ts (dateRange, exportOptions, the joined refusals at :187–201), ObjectPermissionSchema.allowExport. The new file avoids PR feat(spec): a dataset measure declares its polarity, so a lower-is-better KPI draws a rising delta as bad news #26739's hold on analytics-inline-vs-dataset.json.
  3. 门禁 check-runs — dev ran the 61 derived families on the head, all exit 0 (--ran 61/61, 0 NOT-MEASURED); three workflow-valued families are not locally measurable and are read off CI before landing.
  4. 接受的偏差 — none. The README raise follows the ruling (no duplicate to fold in either file, measured 0 B).
  5. 范围外去向 — the skill's rules/dashboards.md teaches a report's dateRange but not exportOptions; the eval asserts a key the skill does not yet teach. The seat takes the dev's option A (land the eval) and files a follow-up card for the teaching line. Contract review at CONTRACT_REVIEW_TIER: requested from the director seat.
  6. 落地条款 — governed, Tier H: stays draft until a GOVERNED_APPROVERS approval; then it lands through the queue.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

Contract review

Served-tier: CONTRACT_REVIEW_TIER
Head-sha: 78362c8bc7050a74666945fe1a8f9448ea5e4013
Local-runs: none
Scope: skills/objectstack-ui/evals/README.md, skills/objectstack-ui/evals/report-period-export.json + scripts/check-skills-token-ratchet.mjs, .changeset/26633-report-period-export-eval.md, packages/spec/src/ui/report.zod.ts, packages/spec/src/ui/list-view-export-options.ts, packages/spec/src/data/date-range-presets.ts, skills/objectstack-ui/rules/dashboards.md

Director seat (summon #52, ledger objectstack#12708), summoned by the domain:skills#1 seat. Read-only at the head above: the diff, card #26633 (filing gate ①, the eval owed by #26628), the seat's ACCEPT 6445946312, the new eval and README as stored on the platform at this head, the cited spec and skill files on origin/main 785a7c3b, the head's check runs. skills/** is a governed-register row and publishes (the @objectstack/skills bundle) ⇒ the rules face, CONTRACT_REVIEW_TIER; Tier H, lands on a GOVERNED_APPROVERS approval, not on this record. Served at CONTRACT_REVIEW_TIER. The seat records; it does not approve, ready, queue or merge.

① Derived judgments

  • Every key the eval asserts exists on main with the shape the ideal answer gives it. report.zod.ts: non-joined ReportSchema carries dataset, rows, values, runtimeFilter; dateRange: { dimension?: string (a type: 'date'dataset dimension), defaultRange: enum(DATE_RANGE_DEFAULT_RANGES) defaultthis_month, allowCustomRange: boolean default true }; exportOptions: ReportExportOptionsSchema.optional() = { formats?, includeHeaders?, fileNamePrefix? } with the format enum narrowed to csv / xlsx ('json' and 'pdf' refused with their own messages, list-view-export-options.ts:41–48); this_quarter is in date-range-presets.ts:46. Both keys are refused on a joined container (report.zod.ts:743–747, messages at :199–207), and allowExport is answered by the schema's own guidance (:463: "a permission, not a report key … ObjectPermission.allowExport on a permission set"). Nothing in prompt, ideal answer or assertions is invented.
  • The grader is consistent with its own ideal answer. Reading the stored head file: 1 fixture, every must_contain present, no must_not_contain present — 0 contradictions (the pattern skills(objectstack-automation): eval fixtures 5, 7 and 8 of flows-triggers-approvals.json contradict their own assertions — expected_output spells a must_not_contain string or lacks a must_contain one, so the ideal answer fails its own check #26515 just repaired on the automation file is not repeated here). The refusals pin exactly the wrong answers the card names: a joined container (type: 'joined', blocks:), a hand-rolled runtimeFilter window (close_date: {), a second preset table (presets:), an invented export permission (permissions.export, canExport, allowExport: true), and 'pdf'. allowExport as a must-contain beside allowExport: true as a must-not-contain is deliberate and right: the answer must name the existing permission axis and must not declare it on the report.
  • Fixture shape matches the siblings (skill_name / description / evals[] { id, prompt, expected_output, files, assertions { must_contain, must_not_contain } }), so whatever reads the other two files reads this one.
  • A third file, not a raise on a sibling: both siblings sit at headroom 0 and analytics-inline-vs-dataset.json is held by open PR feat(spec): a dataset measure declares its polarity, so a lower-is-better KPI draws a rising delta as bad news #26739; a new file with its own row avoids a serial conflict and leaves the siblings' pins untouched. The card left the choice to the lane; this is the cheaper of its three options and the one that does not re-open a file another PR holds.
  • Ratchet rows by ruling A, at the measurement. New row report-period-export.json 457 ⇐ 1828 bytes stored at the head (ceil(1828/4) = 457); README 289 → 335 ⇐ 1339 bytes stored (ceil(1339/4) = 335, from 1143 on main); both comments carry the arithmetic, the 0-byte pay-down reading and the maintainer's tap verbatim. Headroom 0 on both. Correct under check-skills-token-ratchet.mjs's ceil(utf8 bytes / 4) convention.
  • README entry is the three-line form of its siblings, placed between them. Fine.

② Semver level

@objectstack/skills takes a patch changeset (.changeset/26633-report-period-export-eval.md), both declaration lines present (Clause-②: no, Metadata-protocol: none) and equal to the PR body. The catalog ships in the package's dist, so a changeset is owed (the reading #26566 landed on; #26636's skip-changeset reading of the same surface is the lane's open inconsistency, noted on this seat's ledger, not a finding here). A new eval fixture publishes no new accept set — patch is right.

③ Boundary flags

  • Governed path (skills/**, Tier H): no approval yet; draft; os-zhuang, hotlong requested through the relay. This seat's account is not a GOVERNED_APPROVERS member; this record is not an approval and does not lift Tier H.
  • Check runs on 78362c8b at this record: 21+ success (incl. Lint · repo & tooling gates — home of the token ratchet — Check Changeset, Governed Surface Queue Guard, Lint · scoped · tooling self-tests, the PR-body guards, Test UI, Type Check ×4), 11 expected skips, the rest in_progress (Build Core, Test Core shards, Lint · PM dispatch-gates, Lint · scoped · slot lookup & corpus) — none touches the eval JSON; nothing red. A red among them re-opens this record.
  • The eval asserts exportOptions, which the skill's rules do not yet teach (rules/dashboards.md § Report Configuration shows dateRange at :402, nothing on exportOptions; the key is taught only on content/docs/ui/reports.mdx). The dev flagged it, the seat took option A (land the eval first) and filed skills(objectstack-ui): rules teach a report's dateRange but not its exportOptions #26777 for the teaching line. Right order: the eval is the instrument that will show the gap when it is run; a skill that teaches a key no eval checks was the card's complaint in the other direction. Should-fix sequencing for the lane: land skills(objectstack-ui): rules teach a report's dateRange but not its exportOptions #26777 close behind, since that rules file sits at headroom 0 too and will need its own ratchet reading.
  • No gate runs the eval. check:skill-docs / check:skill-refs / check:skill-examples read SKILL.md, references/ and the rules' code fences, not the eval JSON; the green here proves parse, ratchet and changeset, not that an agent passes the prompt. Named so the green is not over-read.
  • Strict single-quote must_contain strings (dimension: 'close_date', 'csv') will miss a double-quoted but correct answer — the same strictness the sibling file documents on its dateGranularity case; a review cue, not a defect.
  • The two skills-lane exportOptions holders (feat(spec): a dataset measure declares its polarity, so a lower-is-better KPI draws a rising delta as bad news #26739 on the analytics file, this PR on a new file) do not collide on any row.

④ Design judgment

  • evals/report-period-export.json against the spec it tests and the card's scenario (an agent authoring a quarterly pipeline report with export from the skill): the prompt gives the dataset, the date dimension and its granularity, the period behaviour and the "who may export" condition in the user's words; the ideal answer resolves each to the one spec key that carries it and says in words the one thing that is not a report key (allowExport). Every refusal corresponds to a documented spec refusal or to a key that does not exist, so a wrong answer fails for a reason the spec can state back. Holds; nothing to break. One thing it will not catch: an answer that writes dateRange without allowCustomRange is still correct (schema default true), and the assertions rightly do not pin it.
  • evals/README.md: the entry says what the file tests in one breath (non-joined, one type: 'date' dimension, csv / xlsx under allowExport). Holds.

Implemented-by: claude/project-thread-c67kgy
Reviewed-by: session_01ADHkmQr732toX135tyo5zs

VERDICT: PASS


Generated by Claude Code

@os-zhuang
os-zhuang marked this pull request as ready for review October 27, 2026 22:48
@os-zhuang
os-zhuang enabled auto-merge October 27, 2026 22:48
@os-zhuang
os-zhuang added this pull request to the merge queue Oct 27, 2026
Merged via the queue into main with commit b1b153a Oct 27, 2026
40 checks passed
@os-zhuang
os-zhuang deleted the claude/project-thread-c67kgy branch October 27, 2026 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/s tooling

Projects

None yet

3 participants