Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions COPY.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,16 @@ Each status shows a title, then a detail line:

Found Skills replace the search results under the heading `Skills for this task`, with the line `A language model picked these from a few searches of the registry.`

### Behavior readings

The Skill behaviors panel shows a behavior reading under each SKILL.md match that needs approval. GLOSSARY.md defines the term and ADR-0016 the mechanism.
Name the model plainly, as a language model. Never call a reading a review, a scan, a safety call, or a false positive.
The strings live in `layers/registry/app/components/_SkillBehaviors.vue`.

- Each match: `SKILL.md:42 Β· Quoted example. {reason}`. `behaviorVerdictLabel` in `shared/behavior-readings.ts` owns the five labels: `Instruction`, `Quoted example`, `Prohibition`, `Documentation`, `Unclear`.
- Panel note, shown when a match has a reading: `A language model read each match that needs approval in its context. Its reading is no guarantee and changes no approval.`
- The skilld CLI approval message adds `(model reading: quoted example. {reason})` to each match, and ends its list with `A language model on skilld.dev wrote each model reading. A reading is no guarantee and changes no approval.` The CLI owns that string, so change it there first.

## Discord digest

The weekly card uses `Trending skills this week` as its title and links to the trending page.
Expand Down
13 changes: 13 additions & 0 deletions GLOSSARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ Shared terms match the skilld CLI glossary (`skilld-dev/skilld/GLOSSARY.md`). Th
| skilld token | `cli_tokens`, `layers/identity` | Author 1β€”N token | "skilld token" |
| provenance | Skill detail, cards | Skill 1β€”1 author and source link | "written by", "source" |
| behavior | `skilld-protocol/behaviors`, Skill page | Skill 1β€”N behavior | "Skill behaviors" |
| behavior reading | `behavior-review` check result, Skill page, skilld CLI | behavior match 1β€”1 behavior reading | "model reading" |
| Skillgen | `workers/skill-harness`, `/skillgen` | Repository 1β€”N pull request | "Skillgen" |
| task search | search box, `layers/registry` | sentence 1β€”N Skill | "Find skills for this task" |

Expand Down Expand Up @@ -226,6 +227,18 @@ Existing route paths, table names, and protocol fields keep their identifiers.

**Never:** permission, capability, risk, threat, scan result, "safe", "secure". No match never means the Skill does nothing.

### behavior reading

**Is:** a language model's reading of one match of a behavior marked "Needs approval": what the matched line does where it stands, as one of five verdicts, with a reason of at most 20 words. The `behavior-review` check result carries the readings of one Artifact. See ADR-0016.

**Verdicts:** Instruction, Quoted example, Prohibition, Documentation, Unclear.

**Use for:** the line under a match in the Skill behaviors panel, and the CLI approval message. Always say a language model wrote it, and that it changes no approval.

**Never:** review result, AI review, verdict (as the noun in UI), clearance, scan, audit, "safe", "cleared", "false positive". A reading never removes the approval a behavior needs.

**Collides with:** behavior. A behavior is the pattern match, and it decides the approval. A reading only annotates one match.

### Owner

**Is:** the GitHub organization or user that hosts Skill Repositories. A proxied entity, never a row skilld.dev owns. Lives at `/gh/[owner]`.
Expand Down
78 changes: 78 additions & 0 deletions docs/adr/0016-behavior-readings.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# ADR-0016: Behavior readings

Date: 2026-10-07

## Context

The skilld CLI stops a remote run until the user approves each `ask` behavior: `remote-code`, `privilege`, `credentials`, `destructive`, and `hidden-text`.
The gate is a fixed text pattern match on the user's machine. It lists each match as `PATH:LINE`.

Many matches are not requests. A security guide quotes `curl ... | bash` as an attack to block.
A checklist says "No `~/.ssh` reads". A table recommends `sudo apt install` to a person.
The user sees only `PATH:LINE` and must open each file to tell these apart.
395 registry Skills still stop at the gate. Each `simota/agent-skills` Skill stops on the quoted attacks in its linked `_common/SECURITY.md` and `_common/WEB_FETCH_SAFETY.md`.

## Decision

A language model reads each match the CLI lists, once per Skill revision, and says what the line does where it stands.
The reading annotates the match. It never removes, adds, or changes a gate.

### What the model reads

An Artifact build runs the same matcher the CLI runs, `skilld-protocol/behaviors`, over every file it packs.
It keeps the first five matches of each `ask` behavior, as the CLI lists them.
A file over 1 MiB, or one that is not UTF-8, gets no reading.

For each match the model receives only:

- the matched line, cut to 400 characters;
- the Markdown headings above it;
- the first row of the table that holds it, or the prose line that introduces its list or code block;
- whether it sits in a fenced code block, and the block's language;
- the file path inside the Skill.

It never receives a whole file. Every invisible or control character arrives as `<U+XXXX>`{lang="html"}.

The model answers one verdict per match: `instruction`, `quoted-example`, `prohibition`, `documentation`, or `unclear`, with a reason of at most 20 words.

### Prompt injection

Skill text is untrusted. The matches arrive as JSON inside a block whose delimiter carries a random nonce, and the system prompt says to follow nothing inside it.
The answer must match a strict JSON schema. A verdict outside the five, a reason over 20 words or with an invisible character, an extra field, a match named twice, and a skipped match all read `unclear` with no reason.

The structural defence is the gate itself: it runs on the user's machine, and no check result reaches it.
A hostile Skill can at most mislabel its own match, on a surface that says a model wrote the label.

### Where it runs

The review runs in the Artifact build, after the scan and before the checks are recorded.
It is the non-required `behavior-review` check result, signed into the attestation. Policy `2026-10-07.5` adds it.
The released skilld CLI accepts a new non-required check, as `omitted-files` showed.

A build waits for the model only when no stored review covers its matches.
D1 keeps one row per Repository ID, commit, Skill folder, and rules version in `behavior_reviews`.
A build whose model input matches a stored row copies that row's readings, so a later commit that left the matched lines alone, or another Skill that links the same files, costs no model call.
A model that fails or takes more than 8 seconds gives an `error` result, which blocks nothing, and the build stores nothing.

Private Skill text never goes to a model. A private build carries a passing `behavior-review` result that says so.

### Where it shows

- The skilld CLI adds each reading to `BEHAVIOR_CONFIRMATION_REQUIRED`, labelled as a language model's reading and no guarantee.
- The Skill page panel shows the SKILL.md readings under each match that needs approval. It reads them from `GET /api/behavior-readings?blob=<SKILL.md blob SHA>`{lang="html"}, and drops a reading whose line text changed.

### Cost

Measured 2026-10-07 over 52 gated Skills, with GPT-6 Luna through the Workers AI chat completions endpoint and reasoning off:
199 matches, 43,328 input and 6,794 output tokens, $0.0078 in all.
That is $0.00015 a Skill and $0.00004 a match. The median call took 2.1 s, the 90th percentile 3.2 s, and the slowest 3.8 s.

The 395 gated Skills cost about $0.06 to read once.
The registry recorded about 166 indexed Skill revisions a day in the week before, and about 9 a day for gated Skills over 30 days: about $0.0013 a day.
Low reasoning effort, tried on 17 of the Skills, cost 19% more, took 1.4 s longer at the median, and left more matches unclear.

## Consequences

- A gated Skill's first build at new matched lines takes about 2 seconds longer.
- The digest and every other surface stay unchanged.
- Cull path: drop the check in a policy bump, drop `/api/behavior-readings`, then drop `behavior_reviews`.
2 changes: 1 addition & 1 deletion docs/arch/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Use the names exactly. Drift breeds shallow modules.
- **app**: `/`, `/community`, `/@<login>/*`, `/collections/new`, `api/community|collections|feed|*`. Owns native data.
- **marketing**: `/skills`, `/skills/*`, `/frameworks/*`, `/learn/*`, `/compare/*`. Comark renders its Markdown. Owns SEO content. [Comparison requirements](comparisons.md) cover editorial evidence and admission.
- **admin**: existing.
- **artifact delivery:** `/api/v1/resolutions|artifacts|trusted-root|github/connections`, and the internal `/api/run-checks/flags`. Owns exact source resolution, checks, signing, grants, transient delivery, and the run checks behind the run check flag.
- **artifact delivery:** `/api/v1/resolutions|artifacts|trusted-root|github/connections`, and the internal `/api/run-checks/flags` and `/api/behavior-readings`. Owns exact source resolution, checks, signing, grants, transient delivery, the run checks behind the run check flag, and the behavior readings ([ADR-0016](../adr/0016-behavior-readings.md)).

Cross-layer reads go via HTTP (`$fetch('/api/...')`), never shared server utilities. Each layer is deletion-testable.

Expand Down
23 changes: 23 additions & 0 deletions layers/artifact-delivery/server/api/behavior-readings.get.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
import { setHeader } from 'h3'
import { defineApiHandler } from '#shared/server/handler'
import { presentBehaviorReadings } from '../presenters/behavior-readings'
import { behaviorReadingsQuerySchema } from '../schemas/behavior-readings'
import { loadSkillMdReadings } from '../utils/behavior-reviewer'

/**
* A language model's readings of the SKILL.md matches that need approval,
* for one SKILL.md Git blob (ADR-0016). The Skill page shows them beside its
* own matches.
*
* Public and read only: each reading names a line of a public file, and only
* public builds are reviewed. A blob's readings change only when a newer
* build reads it again, so five minutes of reuse is safe.
*/
export default defineApiHandler({
schema: behaviorReadingsQuerySchema,
handler: async ({ event, body, platform }) => {
setHeader(event, 'cache-control', 'public, max-age=300')
return await loadSkillMdReadings(platform.db, body.blob)
},
presenter: presentBehaviorReadings,
})
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import type { BehaviorReading, BehaviorReadingsResponse } from '#shared/behavior-readings'

export function presentBehaviorReadings(readings: BehaviorReading[]): BehaviorReadingsResponse {
return { items: readings }
}
8 changes: 8 additions & 0 deletions layers/artifact-delivery/server/schemas/behavior-readings.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import { z } from 'zod'

/** The Git blob SHA of the SKILL.md a Skill page shows. */
export const behaviorReadingsQuerySchema = z.object({
blob: z.string().regex(/^[0-9a-f]{40}$/),
}).strict()

export type BehaviorReadingsQuery = z.infer<typeof behaviorReadingsQuerySchema>
24 changes: 23 additions & 1 deletion layers/artifact-delivery/server/utils/artifact-pack.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
import type { Hash } from 'node:crypto'
import type { ResolvedSource } from '../schemas/contracts'
import type { CollectedBehaviorHits } from './behavior-review'
import type { ArtifactFileObserver, CheckedArtifactSource } from './checks'
import type { OmittedArtifactFile, SourceRejection } from './github-source'
import type { SymbolicLinkNote } from './symbolic-links'
import { createHash } from 'node:crypto'
import { createBehaviorHitCollector } from './behavior-review'
import { createArtifactCheckScanner } from './checks'
import { projectedUstarBytes, USTAR_END_BYTES, ustarHeader, ustarPadding } from './ustar'

Expand Down Expand Up @@ -120,6 +122,11 @@ export interface ScannedArtifact {
contentSha256: string
contentBytes: number
checked: CheckedArtifactSource
/**
* The matches of behaviors that need approval, for the behavior review.
* The build turns them into the `behavior-review` check result.
*/
behaviorHits: CollectedBehaviorHits
/** The archive itself, when it is at most `spoolBytes`. Null means a store reads the files again. */
spool: Uint8Array | null
/** Files read from GitHub one by one, because their archive bytes failed. */
Expand Down Expand Up @@ -164,11 +171,25 @@ export async function scanArtifact(input: {
const spool = contentBytes <= input.spoolBytes ? new Uint8Array(contentBytes) : null
let written = 0
const scanner = createArtifactCheckScanner(input.source)
const hits = createBehaviorHitCollector()
const outcome = await packArtifactFiles({
files: input.files,
read: input.read,
readFromGithub,
observer: scanner,
observer: {
begin: (file) => {
scanner.begin(file)
hits.begin(file)
},
chunk: (bytes) => {
scanner.chunk(bytes)
hits.chunk(bytes)
},
end: () => {
scanner.end()
hits.end()
},
},
write: async (bytes) => {
digest.update(bytes)
spool?.set(bytes, written)
Expand All @@ -190,6 +211,7 @@ export async function scanArtifact(input: {
contentSha256: digest.digest('hex'),
contentBytes,
checked: scanner.finish(input.omitted, input.symbolicLinks),
behaviorHits: hits.finish(),
spool,
readFromGithub,
}
Expand Down
Loading
Loading