Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions docs/evals/writing-2026-10-09/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Writing evaluation, 9 October 2026

Six Skills rewrote the same three generated Markdown documents.
OpenCode 1.18.32 used `zai-coding-plan/glm-5.3` for every successful run.
Each Skill ran once. The baseline ran once without Skill instructions.
This run compares writing instructions, not Skill discovery or a stable quality ranking.

[Recording evidence](record.json) contains exact commits, source hashes, timings, and transcript hashes.
[Seed prompt](seed-prompt.txt) and [rewrite prompt](rewrite-prompt.txt) preserve the shared inputs.
Each directory contains the unedited Markdown output and raw OpenCode events.
The demo manifest stores these same documents for the existing reader.

## Results

The no-Skill baseline already removes the seed's obvious padded wording.
All six Skills preserve the exact code block, URLs, and inline identifiers.
The seed and baseline also pass those exact material checks.
Those checks do not establish complete factual preservation or writing quality.

Word counts below include headings and exclude fenced code. Shorter does not mean better.

| Run | Article words | README words | PR words |
| --- | ---: | ---: | ---: |
| seed | 217 | 123 | 131 |
| baseline | 157 | 88 | 109 |
| humanizer | 174 | 77 | 94 |
| stop-slop | 148 | 81 | 103 |
| no-ai-slop | 114 | 88 | 77 |
| anti-slop | 158 | 80 | 102 |
| deslop-writing | 133 | 67 | 84 |
| im-not-a-fly | 132 | 79 | 86 |

## Meaning changes found during agent review

These observations compare outputs with the supplied fictional facts.
They are agent assessments, not a human review or an automated quality score.

| Run | Observation |
| --- | --- |
| Baseline | Keeps the app limitations, package behavior, existing category field, and PR exclusions. Adds a generic closing summary. |
| Humanizer | Changes the article's β€œcan be more useful” into β€œis more useful.” Adds β€œThe rest of the interface is unchanged” to the PR. |
| Stop Slop | Changes β€œcan be more useful” into β€œserves you better.” Keeps the PR's existing-field detail and exclusions. |
| No AI Slop | Keeps the article's qualified comparison. Omits the PR's detail about using the existing category field. |
| Anti Slop | Keeps the article's qualified comparison. Adds β€œThe rest of the interface is unchanged” to the PR. |
| Deslop Writing | Adds an unsupported requirement that links only become a reading list after sorting. Strengthens the final comparison. Omits the existing-field detail and adds an unchanged-interface claim. |
| Brundlefly im-not-a-fly | Keeps the qualified comparison, existing-field detail, and PR exclusions. Adds a generic causal claim about structure in the article's opening. |

The PR brief says the change maintains simplicity. It does not establish that every other interface element is unchanged.
All README outputs retain case sensitivity, original ordering, the empty-category behavior, and the unchanged input array.
They also retain the limits on fetching URLs and storing links.
All PR outputs retain bookmark behavior, reload reset, and the search and saved-preference exclusions.

## Limits

The topic is fictional. The runner provides the facts and all Markdown source files inside each Skill directory.
The runner disables tools, so Skills cannot inspect external sources or execute scripts.
The model receives Skill instructions inline, including bundled Markdown references.
The baseline uses the same task and original documents without those instructions.
The run uses no temperature override or repeat trials.
The run uses published Brundlefly commit `67d1cf8e3543133a4d73b8a857ff8fc988717956`.
It does not exercise the local checkout's uncommitted `write-human` files.
OpenCode Go rejected an earlier seed attempt because its subscription was inactive.
No output from that attempt enters this comparison.
22 changes: 22 additions & 0 deletions docs/evals/writing-2026-10-09/anti-slop/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Margin

Margin is a fictional TypeScript package for managing reading lists. It provides an API for organizing links by category.

## Getting started

Import `filterLinks` and pass your links and the category you want. The function returns matching links in their original order. Passing an empty category returns all links.

```ts
import { filterLinks } from 'margin'

const links = [
{ title: 'CSS notes', url: 'https://example.com/css', category: 'CSS' },
{ title: 'TypeScript notes', url: 'https://example.com/ts', category: 'TypeScript' },
]

const cssLinks = filterLinks(links, 'CSS')
```

## Important considerations

Category matching is case-sensitive. The function does not modify the input array. Margin does not fetch URLs or store links; it is solely a filtering function.
3 changes: 3 additions & 0 deletions docs/evals/writing-2026-10-09/anti-slop/events.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"type":"step_start","timestamp":1791534032576,"sessionID":"ses_ee0402829ffeR2jsMQdLoZ2TWQ","part":{"id":"prt_11fbfe6ac0016QIcEO7CKHEcAx","messageID":"msg_11fbfd9a0001fkOXl80vY7w7g9","sessionID":"ses_ee0402829ffeR2jsMQdLoZ2TWQ","type":"step-start"}}
{"type":"text","timestamp":1791534097152,"sessionID":"ses_ee0402829ffeR2jsMQdLoZ2TWQ","part":{"id":"prt_11fc0cae1001RW21tMecZ15OgP","messageID":"msg_11fbfd9a0001fkOXl80vY7w7g9","sessionID":"ses_ee0402829ffeR2jsMQdLoZ2TWQ","type":"text","text":"{\"reading-list.md\": \"# Keeping a reading list that works for you\\n\\nA reading list can help you stay informed. With a bit of care, a collection of links becomes a resource you keep learning from.\\n\\n## A simple approach\\n\\nA reading list doesn't need to be complicated. Our fictional reading-list app, Margin, stores a title, URL, and category for each link. You can filter by category and bookmark items you want to return to. Everything stays in your browser. There's no account or cloud sync, which keeps things simple.\\n\\n## Building better habits\\n\\nA weekly review helps you keep a growing collection of articles under control. Remove links you no longer want to read. Keep the ones that connect to a project you are working on. The habit pays off over time.\\n\\nThe goal is a reading list that makes the most of your time. A small, focused list can be more useful than an extensive collection you never open.\", \"README.md\": \"# Margin\\n\\nMargin is a fictional TypeScript package for managing reading lists. It provides an API for organizing links by category.\\n\\n## Getting started\\n\\nImport `filterLinks` and pass your links and the category you want. The function returns matching links in their original order. Passing an empty category returns all links.\\n\\n```ts\\nimport { filterLinks } from 'margin'\\n\\nconst links = [\\n { title: 'CSS notes', url: 'https://example.com/css', category: 'CSS' },\\n { title: 'TypeScript notes', url: 'https://example.com/ts', category: 'TypeScript' },\\n]\\n\\nconst cssLinks = filterLinks(links, 'CSS')\\n```\\n\\n## Important considerations\\n\\nCategory matching is case-sensitive. The function does not modify the input array. Margin does not fetch URLs or store links; it is solely a filtering function.\", \"pr.md\": \"# Add category filtering to the reading list\\n\\nThis pull request adds category filtering to our fictional reading-list app, Margin. It uses the existing category field, so users can view only the articles in a chosen category.\\n\\nPreviously, the list displayed every saved link regardless of the selected category. Now, selecting a category displays only matching links. Selecting All displays the complete list.\\n\\nFiltering preserves the original order of links and does not change bookmarks. The selected category resets when the page reloads. Search and saved filter preferences are outside the scope of this change.\\n\\nThe rest of the interface is unchanged.\"}","time":{"start":1791534090977,"end":1791534097135}}}
{"type":"step_finish","timestamp":1791534097152,"sessionID":"ses_ee0402829ffeR2jsMQdLoZ2TWQ","part":{"id":"prt_11fc0e2f2001U7EpJO9qM90M6Y","reason":"stop","messageID":"msg_11fbfd9a0001fkOXl80vY7w7g9","sessionID":"ses_ee0402829ffeR2jsMQdLoZ2TWQ","type":"step-finish","tokens":{"total":7949,"input":3083,"output":530,"reasoning":4336,"cache":{"write":0,"read":0}},"cost":0}}
9 changes: 9 additions & 0 deletions docs/evals/writing-2026-10-09/anti-slop/pr.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Add category filtering to the reading list

This pull request adds category filtering to our fictional reading-list app, Margin. It uses the existing category field, so users can view only the articles in a chosen category.

Previously, the list displayed every saved link regardless of the selected category. Now, selecting a category displays only matching links. Selecting All displays the complete list.

Filtering preserves the original order of links and does not change bookmarks. The selected category resets when the page reloads. Search and saved filter preferences are outside the scope of this change.

The rest of the interface is unchanged.
13 changes: 13 additions & 0 deletions docs/evals/writing-2026-10-09/anti-slop/reading-list.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Keeping a reading list that works for you

A reading list can help you stay informed. With a bit of care, a collection of links becomes a resource you keep learning from.

## A simple approach

A reading list doesn't need to be complicated. Our fictional reading-list app, Margin, stores a title, URL, and category for each link. You can filter by category and bookmark items you want to return to. Everything stays in your browser. There's no account or cloud sync, which keeps things simple.

## Building better habits

A weekly review helps you keep a growing collection of articles under control. Remove links you no longer want to read. Keep the ones that connect to a project you are working on. The habit pays off over time.

The goal is a reading list that makes the most of your time. A small, focused list can be more useful than an extensive collection you never open.
22 changes: 22 additions & 0 deletions docs/evals/writing-2026-10-09/baseline/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Margin

Margin is a fictional TypeScript package for managing reading lists. It provides an API for organizing links by category, so developers can build reading list features.

## Getting started

Import `filterLinks` and pass in your links and the category you want. The function returns matching links in their original order. Passing an empty category returns all links.

```ts
import { filterLinks } from 'margin'

const links = [
{ title: 'CSS notes', url: 'https://example.com/css', category: 'CSS' },
{ title: 'TypeScript notes', url: 'https://example.com/ts', category: 'TypeScript' },
]

const cssLinks = filterLinks(links, 'CSS')
```

## Important considerations

Category matching is case-sensitive. The function does not modify the input array. Margin does not fetch URLs or store links; it is solely a filtering function.
3 changes: 3 additions & 0 deletions docs/evals/writing-2026-10-09/baseline/events.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"type":"step_start","timestamp":1791533726252,"sessionID":"ses_ee044d486ffeNIZpDo6MgsJ0YR","part":{"id":"prt_11fbb3a1c001z6pxEEMmFUxJ4B","messageID":"msg_11fbb2cfb001xwUOa8LSk1j7Cc","sessionID":"ses_ee044d486ffeNIZpDo6MgsJ0YR","type":"step-start"}}
{"type":"text","timestamp":1791533763754,"sessionID":"ses_ee044d486ffeNIZpDo6MgsJ0YR","part":{"id":"prt_11fbbbf3a001ccR1HK8snd1XYI","messageID":"msg_11fbb2cfb001xwUOa8LSk1j7Cc","sessionID":"ses_ee044d486ffeNIZpDo6MgsJ0YR","type":"text","text":"{\n \"reading-list.md\": \"# Keeping a reading list that works for you\\n\\nA reading list helps you stay informed, but only if you can keep up with it. With some care, a collection of links can become a resource you actually learn from.\\n\\n## A simple setup\\n\\nMargin, our fictional reading-list app, keeps things simple. It stores a title, URL, and category for each link. You can filter by category and bookmark items you want to come back to. Everything stays in your browser, so there is no account or cloud sync.\\n\\n## Building better habits\\n\\nA weekly review helps you keep the collection under control. Remove links you no longer want to read. Keep the ones that connect to a project you are working on. That habit pays off over time.\\n\\nThe goal is a reading list that makes good use of your time. A small, focused list can be more useful than an extensive collection you never open.\",\n \"README.md\": \"# Margin\\n\\nMargin is a fictional TypeScript package for managing reading lists. It provides an API for organizing links by category, so developers can build reading list features.\\n\\n## Getting started\\n\\nImport `filterLinks` and pass in your links and the category you want. The function returns matching links in their original order. Passing an empty category returns all links.\\n\\n```ts\\nimport { filterLinks } from 'margin'\\n\\nconst links = [\\n { title: 'CSS notes', url: 'https://example.com/css', category: 'CSS' },\\n { title: 'TypeScript notes', url: 'https://example.com/ts', category: 'TypeScript' },\\n]\\n\\nconst cssLinks = filterLinks(links, 'CSS')\\n```\\n\\n## Important considerations\\n\\nCategory matching is case-sensitive. The function does not modify the input array. Margin does not fetch URLs or store links; it is solely a filtering function.\",\n \"pr.md\": \"# Add category filtering to the reading list\\n\\nThis pull request adds category filtering to Margin, our fictional reading-list app. It uses the existing category field, so users can narrow their saved articles to a single category.\\n\\nPreviously, the list displayed every saved link regardless of the selected category. Now, selecting a category displays only matching links. Selecting All displays the complete list.\\n\\nFiltering preserves the original order of links and does not change bookmarks. The selected category resets when the page reloads. Search and saved filter preferences are outside the scope of this change.\\n\\nThe result is a more focused reading experience, while keeping the existing interface simple.\"\n}","time":{"start":1791533760314,"end":1791533763740}}}
{"type":"step_finish","timestamp":1791533763754,"sessionID":"ses_ee044d486ffeNIZpDo6MgsJ0YR","part":{"id":"prt_11fbbcc9f001taC7GxtX7HNPDy","reason":"stop","messageID":"msg_11fbb2cfb001xwUOa8LSk1j7Cc","sessionID":"ses_ee044d486ffeNIZpDo6MgsJ0YR","type":"step-finish","tokens":{"total":4430,"input":918,"output":548,"reasoning":2964,"cache":{"write":0,"read":0}},"cost":0}}
9 changes: 9 additions & 0 deletions docs/evals/writing-2026-10-09/baseline/pr.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Add category filtering to the reading list

This pull request adds category filtering to Margin, our fictional reading-list app. It uses the existing category field, so users can narrow their saved articles to a single category.

Previously, the list displayed every saved link regardless of the selected category. Now, selecting a category displays only matching links. Selecting All displays the complete list.

Filtering preserves the original order of links and does not change bookmarks. The selected category resets when the page reloads. Search and saved filter preferences are outside the scope of this change.

The result is a more focused reading experience, while keeping the existing interface simple.
13 changes: 13 additions & 0 deletions docs/evals/writing-2026-10-09/baseline/reading-list.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Keeping a reading list that works for you

A reading list helps you stay informed, but only if you can keep up with it. With some care, a collection of links can become a resource you actually learn from.

## A simple setup

Margin, our fictional reading-list app, keeps things simple. It stores a title, URL, and category for each link. You can filter by category and bookmark items you want to come back to. Everything stays in your browser, so there is no account or cloud sync.

## Building better habits

A weekly review helps you keep the collection under control. Remove links you no longer want to read. Keep the ones that connect to a project you are working on. That habit pays off over time.

The goal is a reading list that makes good use of your time. A small, focused list can be more useful than an extensive collection you never open.
22 changes: 22 additions & 0 deletions docs/evals/writing-2026-10-09/deslop-writing/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Margin

Margin is a fictional TypeScript package for organizing reading-list links by category.

## Getting started

Import `filterLinks`, then pass it your links and a category. The function returns matching links in their original order. An empty category returns all links.

```ts
import { filterLinks } from 'margin'

const links = [
{ title: 'CSS notes', url: 'https://example.com/css', category: 'CSS' },
{ title: 'TypeScript notes', url: 'https://example.com/ts', category: 'TypeScript' },
]

const cssLinks = filterLinks(links, 'CSS')
```

## Behavior and limits

Category matching is case-sensitive. The function doesn't modify the input array. Margin doesn't fetch URLs or store links; it only filters.
3 changes: 3 additions & 0 deletions docs/evals/writing-2026-10-09/deslop-writing/events.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"type":"step_start","timestamp":1791534105335,"sessionID":"ses_ee03f0d99ffewFhu7xI89LQ4IC","part":{"id":"prt_11fc102e8001YMv6JLZ7U6KnVh","messageID":"msg_11fc0f3d2001Av0sgJQiEogB7Z","sessionID":"ses_ee03f0d99ffewFhu7xI89LQ4IC","type":"step-start"}}
{"type":"text","timestamp":1791534175212,"sessionID":"ses_ee03f0d99ffewFhu7xI89LQ4IC","part":{"id":"prt_11fc20677001HEM0vjoQHarsOF","messageID":"msg_11fc0f3d2001Av0sgJQiEogB7Z","sessionID":"ses_ee03f0d99ffewFhu7xI89LQ4IC","type":"text","text":"{\"reading-list.md\": \"# Keep a reading list that works\\n\\nA pile of saved links only becomes a reading list once you can sort it. The links you keep should be the ones you'll actually return to, and structure is what makes that possible.\\n\\n## What Margin does\\n\\nMargin, our fictional reading-list app, stores three things for each link: a title, URL, and category. You can filter by category and bookmark items you want to come back to. Everything stays in your browser; there's no account and no cloud sync, which keeps things simple.\\n\\n## A weekly review\\n\\nGo through the list once a week. Remove the links you no longer want to read, and keep the ones tied to a project you're working on. A small, focused list beats a big collection you never open.\", \"README.md\": \"# Margin\\n\\nMargin is a fictional TypeScript package for organizing reading-list links by category.\\n\\n## Getting started\\n\\nImport `filterLinks`, then pass it your links and a category. The function returns matching links in their original order. An empty category returns all links.\\n\\n```ts\\nimport { filterLinks } from 'margin'\\n\\nconst links = [\\n { title: 'CSS notes', url: 'https://example.com/css', category: 'CSS' },\\n { title: 'TypeScript notes', url: 'https://example.com/ts', category: 'TypeScript' },\\n]\\n\\nconst cssLinks = filterLinks(links, 'CSS')\\n```\\n\\n## Behavior and limits\\n\\nCategory matching is case-sensitive. The function doesn't modify the input array. Margin doesn't fetch URLs or store links; it only filters.\", \"pr.md\": \"# Add category filtering to the reading list\\n\\nThis PR adds category filtering to Margin, our fictional reading-list app. The list previously showed every saved link regardless of the selected category. Selecting a category now shows only matching links; selecting All shows the complete list.\\n\\nFiltering preserves the original order of links and doesn't change bookmarks. The selected category resets when the page reloads. Search and saved filter preferences are out of scope for this change, and the rest of the interface is unchanged.\"}","time":{"start":1791534171767,"end":1791534175197}}}
{"type":"step_finish","timestamp":1791534175212,"sessionID":"ses_ee03f0d99ffewFhu7xI89LQ4IC","part":{"id":"prt_11fc213e1001IhQ7t6fFHtu04a","reason":"stop","messageID":"msg_11fc0f3d2001Av0sgJQiEogB7Z","sessionID":"ses_ee03f0d99ffewFhu7xI89LQ4IC","type":"step-finish","tokens":{"total":14880,"input":10170,"output":466,"reasoning":4244,"cache":{"write":0,"read":0}},"cost":0}}
5 changes: 5 additions & 0 deletions docs/evals/writing-2026-10-09/deslop-writing/pr.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Add category filtering to the reading list

This PR adds category filtering to Margin, our fictional reading-list app. The list previously showed every saved link regardless of the selected category. Selecting a category now shows only matching links; selecting All shows the complete list.

Filtering preserves the original order of links and doesn't change bookmarks. The selected category resets when the page reloads. Search and saved filter preferences are out of scope for this change, and the rest of the interface is unchanged.
Loading
Loading