From 29a9ba7d33eb07074bd53b386b2d5862a54c0473 Mon Sep 17 00:00:00 2001 From: Chris Cashwell Date: Fri, 25 Sep 2026 20:46:52 -0400 Subject: [PATCH 1/6] feat(skills): sync pashov-audit-pipeline with upstream solidity-auditor v4 Upstream (github.com/pashov/skills, 2026-09-23) moved solidity-auditor from v3 to v4: loop mode, scan memory, and a shell-assembled report. - Loop mode: `--loop N` runs N passes of the twelve agents in one scan; every pass after the first is handed what earlier passes found as "ground already walked", and the scan ends in one combined report. - Scan memory: `--memory` (automatic when passes > 1) keeps a ledger at `.solidity-auditor/memory.tsv` in the audited repo. Findings print as KNOWN (n scans) or NEW; records not raised again are listed under "Known from earlier scans" and marked not re-checked. - Shell-assembled report: each pass writes gated findings to a run file while it still has context, and `references/assemble.sh` is the only producer of full-report.md. The orchestrator never composes or re-words the report, so it can never overstate its coverage. - Findings are worded in Simplified Technical English (report-language.md, appended to every bundle); confidence threshold moves from 80 to 75 so promoted leads clear it; deploy scripts are in scope; agents are READ-ONLY inside the audited repo; the upgrade warning fires only when the local VERSION is lower than upstream. Vendoring layout: upstream's SKILL.md is carried verbatim as `references/orchestration.md` so the turn-by-turn procedure (memory read, prune, bundle build, run files, assembly) stays a file-by-file diff. The local SKILL.md is now a wrapper holding only the EVM Cortex layer: agent mapping, context package, Foundry pre-flight writing under the scan directory, a structured `**Severity**` line in each finding block (pasted through by the assembler unchanged), and a Turn 6 severity/PoC annex whose rows are extracted from the assembled report by awk rather than typed. PoCs live under `.solidity-auditor/runs/{stamp}/poc/` and run with FOUNDRY_TEST overridden, never in the project's test/. Deviations from upstream, all documented in SKILL.md: the severity/PoC addendum appended to judging.md (its fix-verification threshold now 75), and onchain/offchain spelling in references/ prose plus the one disclaimer string assemble.sh prints. Everything else under references/ is byte-identical to upstream after that normalization. Verified: assemble.sh runs end to end on a synthetic three-finding scan, the severity line survives into full-report.md, and the annex extractor returns rows equal to the assembler's finding count. Co-Authored-By: Claude Fable 5.1 Signed-off-by: Chris Cashwell --- CLAUDE.md | 2 +- UPGRADING.md | 2 + skills/pashov-audit-pipeline/SKILL.md | 315 ++++++------ skills/pashov-audit-pipeline/VERSION | 2 +- .../references/agent-prompts.md | 119 +++++ .../references/assemble.sh | 353 +++++++++++++ .../references/dedup-and-assembly.md | 419 ++++++++++++++++ .../references/hacking-agents/shared-rules.md | 20 +- .../references/judging.md | 8 +- .../references/orchestration.md | 462 ++++++++++++++++++ .../references/report-formatting.md | 393 +++++++++++---- .../references/report-language.md | 151 ++++++ 12 files changed, 1984 insertions(+), 262 deletions(-) create mode 100644 skills/pashov-audit-pipeline/references/agent-prompts.md create mode 100755 skills/pashov-audit-pipeline/references/assemble.sh create mode 100644 skills/pashov-audit-pipeline/references/dedup-and-assembly.md create mode 100644 skills/pashov-audit-pipeline/references/orchestration.md create mode 100644 skills/pashov-audit-pipeline/references/report-language.md diff --git a/CLAUDE.md b/CLAUDE.md index 405c87d..73b40ca 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -113,7 +113,7 @@ Full analysis with PoC verification for Medium+ findings. Mix of opus and sonnet Complete audit with invariant fuzzing, formal properties, multi-iteration depth analysis, and skeptic review for High/Critical findings. ### Pashov 12-Agent Pipeline -For comprehensive security review, use the `pashov-audit-pipeline` skill which runs 12 specialized attacker agents in parallel. Nine work a single lens — math precision, access control, economic security, execution trace, invariant, periphery, first principles, asymmetry, boundary — and three are gap-hunters (numerical, trust, flow) that report only bugs living at the seam between lenses. Findings are deduplicated, run through four judging gates, confidence-scored, and severity-classified with PoCs for Critical/High. +For comprehensive security review, use the `pashov-audit-pipeline` skill which runs 12 specialized attacker agents in parallel. Nine work a single lens — math precision, access control, economic security, execution trace, invariant, periphery, first principles, asymmetry, boundary — and three are gap-hunters (numerical, trust, flow) that report only bugs living at the seam between lenses. Findings are deduplicated, run through four judging gates, confidence-scored, and assembled by shell into one report, with an EVM Cortex severity annex and PoCs for Critical/High. Supports loop mode (`--loop N` passes per scan, each told what earlier passes found) and a findings ledger across scans (`--memory`). ### 0xSimao Accounting-First Pipeline For accounting-heavy protocols (vaults, lending, staking, yield, AMMs, bridges), use the `simao-audit-pipeline` skill. It reverse-engineers 0xSimao's 869 published findings: the orchestrator first writes a **money map** — assets, tracked totals, an asymmetry table, invariants, lifecycles, and actor cohorts — then hands it to 12 parallel single-specialty lenses (accounting-desync, share-exchange-rate, temporal-cohort, liquidation-solvency, cross-chain-state, rounding-precision, ordering-mev, dos-griefing, access-trust, integration-assumptions, edge-states, flow-completeness). His signature class: a tracked total that desyncs from reality, letting early actors over-withdraw and leaving the last user out insolvent. Findings are deduplicated through four hard gates, judged, and severity-classified with Foundry PoCs for High findings. Complements `pashov-audit-pipeline` — run both and treat overlap as signal. diff --git a/UPGRADING.md b/UPGRADING.md index 0221604..4a41b9c 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -73,6 +73,8 @@ These three skills now ship `references/`, `scripts/`, and `templates/` subdirec **Added `simao-audit-pipeline`.** An accounting-first audit skill vendored from [0xSimao AI](https://github.com/0xsimao/0xsimao-ai) (VERSION 1.0.0), reverse-engineered from 0xSimao's 869 published findings. The orchestrator builds a money map (assets, tracked totals, asymmetry table, invariants, lifecycles, cohorts), then runs 12 parallel single-specialty lenses over it, deduplicates through four hard gates, and emits severity-classified findings with Foundry PoCs for Highs. It ships a `references/` subdirectory (method, severity calibration, report formatting, 12 attack lenses + shared rules) carried over from upstream intact except for the repo's `onchain` spelling normalization; the EVM Cortex adaptations (agent mapping, Foundry pre-flight, PoC routing) live in `SKILL.md` only. Skills total 95. +**`pashov-audit-pipeline` synced to solidity-auditor v4.** Upstream (2026-09-23) added loop mode (`--loop N` passes per scan, each pass told what earlier passes found, one combined report), a findings ledger across scans (`--memory`, on automatically when passes > 1, stored at `.solidity-auditor/memory.tsv` in the audited repo), and a shell-assembled report: each pass writes its gated findings to a run file and `references/assemble.sh` is the only thing that produces `full-report.md`, so the report can never overstate its coverage. Findings are worded in Simplified Technical English (`report-language.md`, appended to every agent bundle), the confidence threshold moved from 80 to 75, deploy scripts are now in scope, agents are READ-ONLY in the audited repo, and the upgrade warning fires only when the local `VERSION` is lower. Upstream's `SKILL.md` is vendored verbatim as `references/orchestration.md` so future syncs stay a file-by-file diff; the EVM Cortex layer (agent mapping, context package, Foundry pre-flight, a `**Severity**` line in each finding block, and a Turn 6 severity/PoC annex built by shell from the assembled report) lives in `SKILL.md`. PoCs are written under `.solidity-auditor/runs/{stamp}/poc/` and run with `FOUNDRY_TEST` overridden, never into the project's `test/`. Audited repos should gitignore `.solidity-auditor/`. + ### v1.0.0 (2026-04-10) **Initial release as EVM Cortex** — Ethereum protocol engineering squad. diff --git a/skills/pashov-audit-pipeline/SKILL.md b/skills/pashov-audit-pipeline/SKILL.md index 0288f9a..5b0ea5a 100644 --- a/skills/pashov-audit-pipeline/SKILL.md +++ b/skills/pashov-audit-pipeline/SKILL.md @@ -1,36 +1,53 @@ --- name: pashov-audit-pipeline -description: Use when performing a comprehensive smart contract security audit. Implements the Pashov Audit Group's parallelized 12-agent attacker-framing methodology — nine single-specialty lenses (math precision, access control, economic security, execution trace, invariant, periphery, first principles, asymmetry, boundary) plus three gap-hunters that find bugs living at the seams between lenses. Produces deduplicated, confidence-scored, severity-classified findings with PoC verification. +description: Use when performing a comprehensive smart contract security audit. Implements the Pashov Audit Group's parallelized 12-agent attacker-framing methodology (solidity-auditor v4) — nine single-specialty lenses (math precision, access control, economic security, execution trace, invariant, periphery, first principles, asymmetry, boundary) plus three gap-hunters that find bugs living at the seams between lenses. Supports loop mode (N passes per scan, each told what earlier passes found) and a findings ledger that remembers results across scans. Produces a shell-assembled, deduplicated, confidence-scored report plus an EVM Cortex severity and PoC annex. Trigger on "pashov audit", "run the auditor in loop mode", "run 3 passes". --- # Pashov Audit Pipeline You are the orchestrator of a parallelized smart contract security audit. -Twelve specialized agents attack the same codebase at once, then their output is deduplicated and gated into a single report. Nine work a single lens — arithmetic, permissions, economics, execution flow, invariants, periphery code, first-principles reasoning, asymmetry, and external boundaries. Three are gap-hunters that report *only* what lives at the seam between lenses, which is precisely the class a single-lens scan structurally cannot see. +Twelve specialized agents attack the same codebase at once, then their output is deduplicated, gated, and assembled into a single report. Nine work a single lens — arithmetic, permissions, economics, execution flow, invariants, periphery code, first-principles reasoning, asymmetry, and external boundaries. Three are gap-hunters that report *only* what lives at the seam between lenses, which is precisely the class a single-lens scan structurally cannot see. -Vendored from the Pashov Audit Group's open-source approach (`github.com/pashov/skills`), skill `solidity-auditor`, **VERSION 3** (see the `VERSION` file alongside this one). Everything under `references/` is upstream content carried over intact except for two documented EVM Cortex additions — the severity/PoC addendum in `references/judging.md` and the deviations section in `references/report-formatting.md`. Check for a newer upstream revision before a high-stakes audit: +Vendored from the Pashov Audit Group's open-source approach (`github.com/pashov/skills`), skill `solidity-auditor`, **VERSION 4** (see the `VERSION` file alongside this one). Everything under `references/` is upstream content carried over intact, with three documented EVM Cortex deviations: + +1. `references/orchestration.md` is upstream's `SKILL.md`, vendored verbatim so the turn-by-turn procedure (memory read, prune, bundle build, run files, assembly) stays a file-by-file diff against upstream. Where any reference file says "SKILL.md Turn N", it means that file. +2. `references/judging.md` carries an appended severity/PoC addendum required by this repo's finding-output-format, severity-matrix, and poc-execution rules. +3. `on-chain`/`off-chain` are normalized to `onchain`/`offchain` across `references/` prose and in the one disclaimer string `assemble.sh` prints, per this repo's style rule. + +Every other EVM Cortex adaptation — agent mapping, context package, Foundry pre-flight, the severity line in each finding block, and the Turn 6 annex — lives in this `SKILL.md` only, so an upstream re-sync replaces `references/` cleanly. Check for a newer upstream revision before a high-stakes audit: ```bash curl -sf https://raw.githubusercontent.com/pashov/skills/main/solidity-auditor/VERSION ``` -If that returns a number greater than 3, this skill is behind upstream. +If that returns a number greater than 4, this skill is behind upstream. + +## What v4 changed + +- **Loop mode.** `--loop [N]` runs N passes of the twelve agents in one scan. Every pass after the first is handed what the earlier passes found as "ground already walked", so it hunts new ground. One combined report at the end. When the runner asks for loop mode without a number, the default is 3 passes; the reference gives measured times (about 15 minutes per pass on a 2,200-line codebase). +- **Scan memory.** `--memory` (on automatically when passes > 1) keeps a ledger at `.solidity-auditor/memory.tsv` in the audited repo. Findings are tagged `KNOWN (n scans)` or `NEW`; records the scan did not raise again are listed under "Known from earlier scans" and explicitly marked not re-checked. +- **Shell-assembled report.** Each pass writes its gated findings to `.solidity-auditor/runs/{stamp}/run-K.md` while it still has context to spare; `references/assemble.sh` is the only producer of `full-report.md`. The orchestrator never composes, re-words, or summarizes the report. A real 3-pass scan that produced 71 findings once printed 14 of them and claimed full coverage; the design exists to kill that defect. +- **Simplified Technical English.** `references/report-language.md` is appended to every agent bundle and governs every title and Description: one sentence, 25 words or fewer, active voice, names who acts and what they get. +- **Threshold 75, not 80.** A promoted lead lands at exactly 75 and must clear the line. It is set once in `judging.md`; the assembler reads it from there. +- **Scope.** Build and dependency directories are excluded; deploy scripts (`script/`, `deploy/`, `*.s.sol`) are **in scope** because they set constructor arguments and hand over ownership; explicitly named files are always scanned wherever they live. +- **Agents are READ-ONLY** inside the audited repository. No PoC files, no scratch notes, not even ones deleted afterwards. PoCs are written under the scan directory (Turn 6). +- **Upgrade warning** fires only when the local `VERSION` is lower than upstream, not merely different. ## The stance: agents are attackers, not reviewers -This is what changed most in v3 and it is the whole point of the pipeline. Every agent is framed as an attacker with unlimited capital and flash loans, not as a reviewer working a checklist. Three consequences worth stating up front, because they invert the instinct: +Every agent is framed as an attacker with unlimited capital and flash loans, not as a reviewer working a checklist. Three consequences worth stating up front, because they invert the instinct: - **When an agent finds a bug it deepens the attack — it never argues itself out of one.** Chain it, find more victims, lower the precondition cost. Refutation belongs to the judging phase, not the hunting phase. - **A finding is not real until traced with concrete values.** No proof means LEAD, not FINDING. Leads are not failures; they are honest calibration and they get emitted. -- **Catalog scanning is not the product.** Pattern catalogs live in the sibling skills (`reentrancy-patterns`, `flash-loan-attacks`, `oracle-manipulation`, `signature-vulnerabilities`, `economic-attack-vectors`, `denial-of-service`) — load those for reference. A pure catalog sweep was the previous generation of this pipeline; it produced volume without depth, which is why upstream dropped the dedicated vector-scan agent in v3. +- **Catalog scanning is not the product.** Pattern catalogs live in the sibling skills (`reentrancy-patterns`, `flash-loan-attacks`, `oracle-manipulation`, `signature-vulnerabilities`, `economic-attack-vectors`, `denial-of-service`) — load those for reference. A pure catalog sweep was an earlier generation of this pipeline; it produced volume without depth, which is why upstream dropped the dedicated vector-scan agent in v3. ### When to Use - Full security audit of a protocol before mainnet deployment -- Re-audit after significant code changes or new feature additions +- Re-audit after significant code changes or new feature additions — run with `--memory` so the report says what is new - Pre-merge security review of high-risk PRs touching core accounting or token logic -- Competitive audit participation where thoroughness and finding volume matter +- Competitive audit participation where thoroughness and finding volume matter — use loop mode ### When NOT to Use @@ -39,238 +56,198 @@ This is what changed most in v3 and it is the whole point of the pipeline. Every - Gas-only review — use `gas-optimizer` - Code quality review without security focus — use `code-reviewer` - Pre-audit reconnaissance and readiness assessment — run `xray-pre-audit` first, then feed its output in as the context package +- Accounting-heavy protocols where the question is "does the tracked total match reality" — run `simao-audit-pipeline` as well and treat overlap as signal --- -## Mode Selection +## How to run this skill -**Exclude pattern:** skip directories `interfaces/`, `lib/`, `mocks/`, `test/`, `script/` and files matching `*.t.sol`, `*.s.sol`, `*Test*.sol`, `*Mock*.sol`. +Follow `references/orchestration.md` turn by turn — Mode Selection, Turn 1 through Turn 5, and its Banner — with the EVM Cortex substitutions and insertions below. The reference files it delegates to sit in the same directory: `agent-prompts.md` (Turn 3a prompts), `dedup-and-assembly.md` (Turn 4 and Turn 5 procedure), `report-formatting.md` (finding-block shape), `report-language.md` (wording), `judging.md` (gates, confidence, lead promotion, severity addendum), `senior-auditor-sop.md` and `hacking-agents/` (bundle content), and `assemble.sh` (the assembler). -- **Default** (no arguments): scan all `.sol` files using the exclude pattern. Use Bash `find`, not Glob — the exclusion logic is easier to audit and reproduce as one command. -- **`$filename ...`**: scan the specified file(s) only. - -**Flags:** - -- `--file-output` (off by default): also write the report to a markdown file, at the path in `references/report-formatting.md`. Never write a report file unless explicitly passed. - ---- +Two upstream steps are pinned here because the runtime differs: -## Orchestration +- **`{resolved_path}`** is this skill's own `references/` directory. Do not glob for `shared-rules.md` — the installer places a copy under `~/.claude/skills/` and a repo checkout may hold another, and a glob can pick the wrong one. +- **Version check** (Turn 1 e): compare as numbers and warn only when local is lower. Print `⚠️ Upstream solidity-auditor is at version N, this skill is vendored at 4. See https://github.com/pashov/skills`. A failed fetch is skipped silently — a network failure is not an audit finding. -### Turn 1 — Discover +### Turn 1b — Model and pass count -Print the banner, then make these tool calls in parallel in one message: +Ask both questions in one `AskUserQuestion` call exactly as the reference describes. The runner's model choice `{agent_model}` applies to agents 1–9. **The three gap-hunters (agents 10–12) run on `opus` regardless of the answer.** Cross-lens reasoning is where model tier matters most; a weaker gap-hunter collapses into restating single-lens findings, which dedup then discards as duplicates — the most expensive way to save money in this pipeline. Say so in one line when the runner picks a lower tier. -1. Bash `find` for in-scope `.sol` files per mode selection -2. `ToolSearch select:Agent` -3. Read the local `VERSION` file in this skill's directory -4. Bash `curl -sf https://raw.githubusercontent.com/pashov/skills/main/solidity-auditor/VERSION` -5. Bash `mktemp -d ./.audit-XXXXXX` → store as `{bundle_dir}` +If the audited repo's `.gitignore` does not list `.solidity-auditor/`, print a one-line reminder that the runs directory and ledger should be ignored. Do not edit `.gitignore` yourself. -`{resolved_path}` is this skill's `references/` directory. +### Turn 2b — Context package (EVM Cortex insertion, once per scan) -If the remote VERSION fetch succeeds and differs from local, print: `⚠️ Upstream solidity-auditor is at version N, this skill is vendored at 3. See https://github.com/pashov/skills`. If the fetch fails, skip silently — a network failure is not an audit finding. +After `source.md` is built and before the bundles are catted, assemble, when available: the protocol README, known issues (to avoid duplicate reports), documented design decisions, deployment context (target chains, upgrade strategy), external dependencies, and prior audit reports with resolution status. Write it to `{bundle_dir}/context.md` and append it to every bundle **after `report-language.md` and before `known-findings.md`**. Keep it under 300 lines; the agents' attention belongs to the source. -### Turn 1b — Model selection +If `xray-pre-audit` has been run, its `x-ray/x-ray.md` is the best available context package — it already carries the threat model, invariant list, and entry-point classification. -Ask which model tier the twelve agents should run at, via `AskUserQuestion`, defaulting to the orchestrator's own family. Store as `{agent_model}`. +### Turn 2c — Foundry pre-flight (EVM Cortex insertion, once per scan) -Prefer opus for the three gap-hunters regardless of the answer. Cross-lens reasoning is where model tier matters most, and a weaker gap-hunter collapses into restating single-lens findings — which Phase 4 then discards as duplicates. That is the most expensive way to save money in this pipeline. +Run in the audited repo, writing outputs only under the scan directory: -### Turn 2 — Prepare bundles - -Read `{resolved_path}/report-formatting.md` and `{resolved_path}/judging.md` in parallel. - -Then build all bundles in a single Bash command using `cat` (not shell variables or heredocs): +```bash +forge build --deny-warnings +forge test --summary +slither . --filter-paths "test|script|node_modules|lib" --json .solidity-auditor/runs/{stamp}/slither-report.json +forge tree > .solidity-auditor/runs/{stamp}/dependency-tree.txt +``` -1. `{bundle_dir}/source.md` — ALL in-scope `.sol` files, each under a `### path` header inside a fenced `solidity` block. -2. One bundle per agent: `source.md` + SOP + that agent's specialty + shared rules. +`forge build` and `forge test` write only to `out/` and `cache/`, which the scan excludes. A build failure is not a stopper — note it in the context package and continue; the agents read source, not artifacts. If the project is not a Foundry project, skip the `forge` steps and say so. -| Bundle | Appended files (relative to `{resolved_path}`) | -|--------|-----------------------------------------------| -| `agent-1-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/math-precision-agent.md` + `hacking-agents/shared-rules.md` | -| `agent-2-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/access-control-agent.md` + `hacking-agents/shared-rules.md` | -| `agent-3-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/economic-security-agent.md` + `hacking-agents/shared-rules.md` | -| `agent-4-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/execution-trace-agent.md` + `hacking-agents/shared-rules.md` | -| `agent-5-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/invariant-agent.md` + `hacking-agents/shared-rules.md` | -| `agent-6-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/periphery-agent.md` + `hacking-agents/shared-rules.md` | -| `agent-7-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/first-principles-agent.md` + `hacking-agents/shared-rules.md` | -| `agent-8-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/asymmetry-agent.md` + `hacking-agents/shared-rules.md` | -| `agent-9-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/boundary-agent.md` + `hacking-agents/shared-rules.md` | -| `agent-10-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/numerical-gap-agent.md` + `hacking-agents/shared-rules.md` | -| `agent-11-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/trust-gap-agent.md` + `hacking-agents/shared-rules.md` | -| `agent-12-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/flow-gap-agent.md` + `hacking-agents/shared-rules.md` | +### Turn 3a — Agent mapping -Print line counts for `source.md` and every bundle. An undersized bundle means a broken `cat` and must be caught before agents launch, not after twelve agents return empty. +Use the two prompt templates in `references/agent-prompts.md` verbatim, substituting `{bundle_dir}`, the agent number, and the bundle's real line count. The READ-ONLY paragraph is unconditional; the "Known findings" paragraph appears only when memory is on and `known-findings.md` was appended. Spawn all twelve as parallel background Agent calls with these `subagent_type` values: -**Never inline source code into an Agent prompt.** The prompt points at the bundle file and the agent reads it. Inlining multiplies token cost by twelve and truncates on large codebases. +| # | Lens | `subagent_type` | Model | +|---|------|-----------------|-------| +| 1 | Math & Precision | `depth-token-flow` | opus | +| 2 | Access Control | `access-control-reviewer` | sonnet | +| 3 | Economic Security | `mev-analyst` | sonnet | +| 4 | Execution Trace | `depth-state-trace` | opus | +| 5 | Invariant | `invariant-analyst` | sonnet | +| 6 | Periphery | `depth-external` | opus | +| 7 | First Principles | `sleuth` | opus | +| 8 | Asymmetry | `code-reviewer` | opus | +| 9 | Boundary | `depth-edge-case` | sonnet | +| 10 | Numerical Gap | `depth-token-flow` | opus | +| 11 | Trust Gap | `mev-analyst` | opus | +| 12 | Flow Gap | `sleuth` | opus | -Each agent gets its own lens only. Handing all twelve specialties to every agent defeats the design — the lenses are supposed to be independent so their overlap is evidence rather than an echo. +The `subagent_type` selects a base persona and tool set; **the specialty file in the bundle is what determines the lens.** Reused types (`depth-token-flow`, `mev-analyst`, `sleuth`) run as independent instances with different bundles and share no context. The Model column is the default when Turn 1b set no `{agent_model}`; when it did, agents 1–9 take the runner's choice and 10–12 stay on opus. -### Turn 2b — Context package +### Verifying the agents did the work -Every agent also gets, when available: the protocol README, known issues (to avoid duplicate reports), documented design decisions, deployment context (target chains, upgrade strategy), external dependencies, and prior audit reports with resolution status. +`shared-rules.md` binds every agent to three mental tools from `senior-auditor-sop.md`, each with a trigger that requires a literal marker in the agent's output: `[Feynman: ]` when it opens a new function, `[Socratic: — why?]` when it stops on an unclear line, `[Inversion: ]` when a path reads as clean. -If `xray-pre-audit` has been run, its `x-ray/x-ray.md` output is the best available context package — it already contains the threat model, invariant list, and entry-point classification. +After each agent returns, grep its output for those markers. An agent that returns findings with no markers did not reason — it scanned. Note the shortfall as a workflow violation and weight that agent's findings accordingly. Do not respawn it: in loop mode the next pass covers the same lens for free, and on a 1-pass scan a retry costs an unbounded wait for one twelfth of the coverage. -### Turn 2c — Pre-flight +### Turn 4 — Severity line (EVM Cortex insertion, every pass) -```bash -forge build --deny-warnings # must compile clean -forge test --summary # establish a baseline -slither . --filter-paths "test|script|node_modules" --json slither-report.json -forge tree > dependency-tree.txt -``` +Follow `references/dedup-and-assembly.md` Turn 4 step by step. Between step 3 (lead promotion) and step 4 (memory tag), classify every gated **FINDING** per the severity addendum in `judging.md` — impact × likelihood from the global severity matrix, assigned independently of confidence. LEADs are not classified. -### Turn 3a — Spawn all twelve agents +In step 5a, write the severity into the finding block's body as one structured line between the Description and the Fix: -One message, twelve parallel background Agent calls (`run_in_background=true`), `model={agent_model}`. Single phase, no later spawns. You will be notified as each completes — do not poll or sleep. +````markdown +**Description** + -Agents 1–9 use the single-specialty prompt; agents 10–12 use the gap-hunter prompt. +**Severity** High · Impact High · Likelihood Likely -**Single-specialty prompt (agents 1–9):** +**Fix** +... +```` -``` -You are an attacker. Your specialty, mindset, source, and output rules -are in your bundle. Read it fully before producing findings. +The assembler pastes the body through unchanged, so the line reaches the report without any change to `assemble.sh`. It is a structured field, not prose: `report-language.md` rule 10 (no `critical`, `severe` in sentences) governs sentences and does not reach it. Use exactly one of `Critical`, `High`, `Medium`, `Low`, `Informational`, and keep the `**Severity** ` prefix and ` · ` separators exactly — Turn 6 extracts the value by shell. -Read first: -- {bundle_dir}/agent-N-bundle.md (XXXX lines) — source + SOP + specialty + shared rules. +### Turn 5 — Assemble, print, clean -The bundle contains all in-scope source. Do NOT re-read in-scope files -for the initial scan. Use Read/Grep only for cross-file searches or -out-of-scope context (interfaces/, lib/, mocks/, test/). +Follow `references/dedup-and-assembly.md` Turn 5 unchanged. Do not re-word, re-order, add to, or summarize `full-report.md`. The `--file-output` copy is named `{project-name}-pashov-ai-audit-report-{stamp}.md`. -What a finding looks like: -- file, function -- root cause — the one-sentence code-level defect -- minimal fix — the smallest change that eliminates the defect -- proof — concrete numbers, a trace, or quoted code +### Turn 6 — Severity and PoC annex (EVM Cortex, once per scan) -Without concrete proof, it's a LEAD, not a finding. Leads are honest -about what you couldn't verify — they're not failures, they're -calibration. Emit them. +Runs after Turn 5, at any pass count. It reads `full-report.md` and never writes to it. -Don't skim. Don't trust your first read. Trust your discomfort. +1. **Extract the severity table by shell, never by hand.** One `awk` over the assembled file; the `#` column is the finding's number in `full-report.md`: -Output format: see shared-rules.md inside your bundle. +```bash +REPORT=.solidity-auditor/runs/{stamp}/full-report.md +awk -v OFS='\t' ' +function rank(s){ return (s=="Critical")?0:(s=="High")?1:(s=="Medium")?2:(s=="Low")?3:(s=="Informational")?4:5 } +function flush(){ if (want) { n++; print rank(sev), conf+0, "| " n " | " sev " | [" conf "] | " title " | `" loc "` |"; want=0 } } +/^\[[0-9]+\] \*\*[0-9]+\. / { flush(); match($0,/^\[[0-9]+\]/); conf=substr($0,RSTART+1,RLENGTH-2) + t=$0; sub(/^\[[0-9]+\] \*\*[0-9]+\. /,"",t); sub(/\*\*[ \t]*$/,"",t); title=t; want=1; sev="Unclassified"; loc=""; next } +want && loc=="" && /^`/ { match($0,/^`[^`]*`/); loc=substr($0,RSTART+1,RLENGTH-2); next } +want && /^\*\*Severity\*\* / { s=$0; sub(/^\*\*Severity\*\* /,"",s); sub(/ ·.*$/,"",s); sev=s; next } +/^Findings List/ { flush() } +END { flush() } +' "$REPORT" | sort -t$'\t' -k1,1n -k2,2nr | cut -f3 > .solidity-auditor/runs/{stamp}/severity-rows.md +wc -l < .solidity-auditor/runs/{stamp}/severity-rows.md ``` -**Gap-hunter prompt (agents 10–12):** identical, except the finding shape adds `seam — which two or three lenses combine`, the proof must demonstrate the seam, and the closing line points at the gap-hunter-specific output fields in the specialty file. - -### Turn 3b — Wait - -Proceed only once all twelve have notified completion. Let them run to natural completion; do not start dedup early and do not poll. +The row count must equal `F` from Turn 5 step 3. If it does not, the annex says so in words and lists what it could read — it never claims to cover more than it does. A finding whose block lacks a severity line prints as `Unclassified`; leave it so and say why, rather than editing the assembled report. -### Turn 4 — Deduplicate, judge, report +2. **Write `.solidity-auditor/runs/{stamp}/severity-annex.md`:** -Single pass: dedup, gate, and produce the final report in one turn. Do not print an intermediate dedup list. +````markdown +# Severity annex — -1. **Dedup** per the four hard gates below. -2. **Gate** each deduped finding through the four gates in `judging.md`, in order, no skipping and no revisiting after a verdict. -3. **Promote or reject leads** per `judging.md`. -4. **Classify severity** and attach PoCs per the addendum in `judging.md`. -5. **Format and print** per `report-formatting.md`. With `--file-output`, also write the file. -6. **Auto-clean:** `rm -rf {bundle_dir}`. It is transient build state, not an artifact. For debugging, copy it elsewhere before re-running. +_EVM Cortex layer over `full-report.md` (same stamp). Severity is impact × likelihood per the global severity matrix and is independent of confidence. `#` is the finding's number in the report. Rows: N of F findings._ ---- - -## Deduplication Gates +| # | Severity | Confidence | Title | Location | +|---|---|---|---|---| + -Group findings by `group_key` (`Contract | function | bug-class`). Exact match first, then merge synonymous `bug_class` values within the same `(Contract, function)`. Keep the best item per group, number sequentially, annotate `[agents: N]`. +## Proof of concept -Four gates govern this phase. They exist because the failure mode of naive merging is silently deleting real bugs — twelve agents converging on one function is *information*, not redundancy. +_Critical and High findings require a working Foundry PoC before they are reported at that severity. Medium findings need a PoC or a step-by-step reproduction._ -**Gate A — Function isolation (HARD).** NEVER merge across different `function:` values. Dedup only within `(Contract, function)`. A different function is a different bug, always. +| # | Severity | PoC | Status | +|---|---|---|---| +| 3 | Critical | `.solidity-auditor/runs/{stamp}/poc/Exploit_3.t.sol` | passes · fork block 19_000_000 | +| 7 | High | — | pending — routed to poc-writer | +```` -**Gate B — Wide description.** A merged group whose constituents describe distinct mechanisms — different `fix:`, different code-level cause, or different attack path — MUST list every mechanism. One function can host several coexisting bugs at the same `group_key` and all of them must appear. +3. **Route PoCs.** For every Critical and High finding, spawn `security-verifier` or `poc-writer` with the finding block and the source it names. PoC tests are written **only** under `.solidity-auditor/runs/{stamp}/poc/` — never into the project's `test/` — and run with the test directory overridden so the project tree stays untouched: -**Gate C — Function-level second pass.** After `group_key` dedup, run a second pass at `(Contract, function)` ignoring `bug_class` entirely. Agents often tag coexisting bugs with different `bug_class` values while referencing multiple mechanisms in the body text. For every `(Contract, function)` with multiple final findings, scan the description, path, proof, and fix of every constituent for distinct mechanisms crossing `bug_class` boundaries. Every mechanism appearing in any constituent body must survive into at least one final finding. This pass stays *within* `(Contract, function)` — never across, per Gate A. - -**Gate D — Fix preservation (HARD).** Before writing a merged `fix:` for a `(Contract, function)` with multiple findings: +```bash +FOUNDRY_TEST=.solidity-auditor/runs/{stamp}/poc forge test --match-path '.solidity-auditor/runs/{stamp}/poc/*' -vvv +``` -1. Collect every raw `fix:` from every agent that flagged the tuple. -2. Group them by ADD-lines (the `+` lines, or the equivalent require/assignment). -3. Two fixes are distinct if their ADD-lines differ in the called function or expression (`require(msg.value == amount)` vs `require(zrc20 != _ETH_ADDRESS_)`), the check direction (validate / restrict / ban), or the checked parameter. -4. Two or more distinct fixes are presented as Option A, Option B, … — one block each, **verbatim** from the agent's text, no paraphrase. -5. Label each intuitively: validate / restrict / allow-and-handle / ban-path. +Pin the fork block in every fork-based PoC. A Critical or High finding whose PoC fails is downgraded in the annex with a one-line reason; the severity line in the run file is left as written — the run file is the pass's record, the annex is this turn's verdict. -Before printing, count the distinct fixes in the raw output for that `(Contract, function)`. Two or more distinct but only one shown is a violation — add the alternatives. +4. **Fix verification** for findings at or above the threshold, per the `judging.md` addendum: trace the fix against the attack path, run the side-effect checklist, and pattern-check the rest of the codebase for the same defect. -**Completeness check (HARD).** Before printing the report, enumerate every unique `(Contract, function, bug-class)` in any raw FINDING or LEAD across all twelve agents. Every unique `(Contract, function)` must have at least one item in the final report; zero means a silent drop, so fix it. Multiple `bug_class` values within one `(Contract, function)` may collapse into a single wide-description item, but the `(Contract, function)` itself must survive. Print this line before the report: +5. **Print** a five-number summary and the annex path, nothing else: ``` -Completeness: N unique (Contract, function) in raw, N covered in final. +Severity: Critical N · High N · Medium N · Low N · Informational N — annex: .solidity-auditor/runs/{stamp}/severity-annex.md ``` -**Composite chains.** If finding A's output feeds finding B's precondition AND the combined impact exceeds either alone, add `Chain: [A] + [B]` at `confidence = min(A, B)`. Most audits produce zero to two. - ---- - -## Verifying the agents did the work - -`shared-rules.md` binds every agent to three mental tools from `senior-auditor-sop.md`, each with a trigger that requires a literal marker in the agent's output: `[Feynman: ]` when it opens a new function, `[Socratic: — why?]` when it stops on an unclear line, `[Inversion: ]` when a path reads as clean. - -After each agent returns, grep its output for those markers. An agent that returns findings with no markers did not reason — it scanned. Note the shortfall as a workflow violation and weight that agent's findings accordingly; consider re-running it. +With `--file-output`, also copy the annex to `{project-name}-pashov-ai-audit-severity-{stamp}.md` beside the report copy. --- -## EVM Cortex Agent Mapping +## Deduplication gates (summary) -| # | Lens | `subagent_type` | Model | -|---|------|-----------------|-------| -| 1 | Math & Precision | `depth-token-flow` | opus | -| 2 | Access Control | `access-control-reviewer` | sonnet | -| 3 | Economic Security | `mev-analyst` | sonnet | -| 4 | Execution Trace | `depth-state-trace` | opus | -| 5 | Invariant | `invariant-analyst` | sonnet | -| 6 | Periphery | `depth-external` | opus | -| 7 | First Principles | `sleuth` | opus | -| 8 | Asymmetry | `code-reviewer` | opus | -| 9 | Boundary | `depth-edge-case` | sonnet | -| 10 | Numerical Gap | `depth-token-flow` | opus | -| 11 | Trust Gap | `mev-analyst` | opus | -| 12 | Flow Gap | `sleuth` | opus | +The procedure is `references/dedup-and-assembly.md` Turn 4 step 1 and is followed from there, not from here. The gates are hard, and the failure mode they exist for is silently deleting real bugs — twelve agents converging on one function is *information*, not redundancy: -The `subagent_type` selects a base persona and tool set; **the specialty file in the bundle is what determines the lens.** Reused types (`depth-token-flow`, `mev-analyst`, `sleuth`) run as independent instances with different bundles and share no context. +- **Canonicalise the bug-class label** (new in v4) — one label per (Contract, function) before grouping: the repository's own label from `known-findings.md` wins, then the majority label, then the shortest label that names the defect. Without it one bug becomes three ledger records and is never recognised again. +- **Function isolation** — never merge across `function:` values. A different function is a different bug, always. +- **Wide description** — a merged group with distinct mechanisms lists every mechanism. +- **Function-level second pass** — at (Contract, function) ignoring `bug_class`, every mechanism in any constituent body survives into a final finding. +- **Fix preservation** — distinct fixes are shown verbatim as Option A, Option B, … with intuitive labels. +- **Completeness** — every unique (Contract, function) in raw output has at least one item in the run file; print `Completeness: N unique (Contract, function) in raw, N covered in final.` -PoC construction for Critical and High findings routes to `security-verifier` or `poc-writer` after Turn 4. +Composite chains: `Chain: [A] + [B]` at `confidence = min(A, B)` when A's output feeds B's precondition and the combined impact exceeds either alone. Most audits produce zero to two. --- ## Pre-Audit Checklist -- [ ] All in-scope files identified via the exclude pattern +- [ ] All in-scope files identified with the exact `find` command in `orchestration.md` (deploy scripts in, build and dependency dirs out) - [ ] `xray-pre-audit` run, or an equivalent context package assembled -- [ ] Bundle directory created; `source.md` plus twelve agent bundles built -- [ ] Line counts printed for `source.md` and every bundle, and none is undersized -- [ ] Context package assembled (README, known issues, design docs, prior audits) -- [ ] `forge build --deny-warnings` passes clean -- [ ] `forge test` passes with no failures -- [ ] Slither baseline report generated -- [ ] Known issues documented to avoid duplicate findings -- [ ] Local `VERSION` checked against upstream +- [ ] Local `VERSION` (4) checked against upstream +- [ ] `{stamp}` computed once; `.solidity-auditor/runs/{stamp}/scope.tsv` opened with `name`, `mode`, `files` +- [ ] Pass count and model settled in one `AskUserQuestion`; `passes_planned` written +- [ ] Memory on → ledger validated (`#solidity-auditor-memory v1`, 6 columns) or the scan stopped +- [ ] `source.md` built once; twelve bundles built per pass, each ending with `report-language.md` (+ `context.md`, + `known-findings.md` when present); line counts printed and none undersized +- [ ] Foundry pre-flight run; outputs under the scan directory only +- [ ] `.solidity-auditor/` gitignored in the audited repo (reminder printed if not) ## Post-Audit Checklist -- [ ] All twelve agents returned findings +- [ ] All twelve agents returned; any loss recorded in the pass summary line, the run-file header, and `pass_K_agents` - [ ] Mental-tool marker counts verified per agent -- [ ] Findings grouped by `group_key`; function isolation respected (Gate A) -- [ ] Wide-description gate applied to every merged group (Gate B) -- [ ] Function-level second pass run across `bug_class` boundaries (Gate C) -- [ ] Distinct fixes preserved verbatim as Option A/B/… (Gate D) -- [ ] `Completeness: N / N` line printed and reconciled -- [ ] Composite chains identified -- [ ] Every finding run through all four judging gates in order, one pass, no revisiting -- [ ] Confidence scored from 100 with documented deductions +- [ ] Bug-class labels canonicalised; function isolation, wide description, second pass, fix preservation, and the completeness line applied per pass +- [ ] Every finding run through the four judging gates in order, one pass, no revisiting - [ ] LEADs promoted or rejected with justification; no deployer-intent reasoning used -- [ ] Severity assigned per the matrix in `judging.md` -- [ ] Foundry PoC written for every Critical and High finding -- [ ] Fix verification completed for confidence ≥ 80 findings -- [ ] Fix pattern-check run across the rest of the codebase -- [ ] Report ordered by severity then confidence; sub-80 findings carry no fix block -- [ ] Agent attribution table completed -- [ ] Bundle directory deleted +- [ ] Every gated FINDING carries a `**Severity**` line in its run-file block +- [ ] Every run file uses the exact `` / `` markers; `assemble.sh` reported no structure break +- [ ] `full-report.md` printed word for word (20 findings or fewer) or as the counted top-3 slice (more than 20); never re-worded +- [ ] Memory on → `memory.tsv` written atomically via `.tmp`; `mem_after` and `mem_sha` recorded +- [ ] Annex row count equals `F`; Foundry PoC passing for every Critical and High, under `.solidity-auditor/runs/{stamp}/poc/` +- [ ] Fix verification and codebase pattern-check completed for findings at or above 75 +- [ ] Bundle directory deleted; nothing written outside `.solidity-auditor/` except `--file-output` copies --- diff --git a/skills/pashov-audit-pipeline/VERSION b/skills/pashov-audit-pipeline/VERSION index 00750ed..b8626c4 100644 --- a/skills/pashov-audit-pipeline/VERSION +++ b/skills/pashov-audit-pipeline/VERSION @@ -1 +1 @@ -3 +4 diff --git a/skills/pashov-audit-pipeline/references/agent-prompts.md b/skills/pashov-audit-pipeline/references/agent-prompts.md new file mode 100644 index 0000000..453a356 --- /dev/null +++ b/skills/pashov-audit-pipeline/references/agent-prompts.md @@ -0,0 +1,119 @@ +# Agent prompt templates — Turn 3a + +The two prompts the orchestrator gives to the 12 agents. Agents 1–9 get the +single-specialty prompt; agents 10–12 get the gap-hunter prompt. + +Both are verbatim text with values substituted in. Substitute `{bundle_dir}`, +the agent number `N`, and the bundle's real line count. Change nothing else. + +The orchestrator reads this file in **Turn 2**, in the same parallel message +that reads `report-formatting.md` and `judging.md`. + +--- + +## Single-specialty prompt + +**Turn 3a-i — Single-specialty prompt template (agents 1–9, substitute real values):** + +``` +You are an attacker. Your specialty, mindset, source, and output rules +are in your bundle. Read it fully before producing findings. + +Read first: +- {bundle_dir}/agent-N-bundle.md (XXXX lines) — source + SOP + specialty + shared rules. + +The bundle contains all in-scope source. Do NOT re-read in-scope files +for the initial scan. Use Read/Grep only for cross-file searches or +out-of-scope context (interfaces/, mocks/, test/, and dependency +sources under node_modules/ or lib/ — read a base contract when the +behaviour it inherits matters). + +You are READ-ONLY inside the audited repository. Never create, edit or +delete a file there — not a Foundry PoC, not a test, not a scratch note, +not even one you intend to delete afterwards. An audit that changes the +code it is measuring is not an audit. Write proof-of-concept code in your +own scratchpad, or quote it in your finding as text. + +What a finding looks like: +- file, function +- root cause — the one-sentence code-level defect +- minimal fix — the smallest change that eliminates the defect +- proof — concrete numbers, a trace, or quoted code + +Without concrete proof, it's a LEAD, not a finding. Leads are honest +about what you couldn't verify — they're not failures, they're +calibration. Emit them. + +Don't skim. Don't trust your first read. Trust your discomfort. + +Write every description in Simplified Technical English — the rules are +in your bundle, in "Report language". One sentence, 25 words or fewer, +active voice, no metaphor, and it names who acts and what they get. +Your bug_class label, the identifiers and any code you quote are data: +write those exactly as the source and the output rules require. + +Your bundle ends with "Known findings — ground already walked". Obey it: +spend your effort on new ground, report every bug you find in full — +the listed ones included — and reuse its bug-class labels for the same +class of bug in the same function. + +Output format: see shared-rules.md inside your bundle. +``` + +The "Known findings" paragraph is included **only when memory is on and `known-findings.md` was appended**. On a plain scan the prompt is byte-identical to the one it has always been — a paragraph about a section that is not there would send agents hunting for it. + +The READ-ONLY paragraph is **unconditional** — every agent, every mode, every pass. It is here because a real scan proved it necessary: an agent built Foundry proof-of-concept files inside the audited repository and deleted them afterwards. It left the tree clean and the stored SHA honest, and it was still wrong. A later editor must not make it conditional, and must not soften it into a preference. + +## Gap-hunter prompt + +**Turn 3a-ii — Gap-hunter prompt template (agents 10–12, substitute real values):** + +``` +You are an attacker. Your gap-hunter specialty, mindset, source, and +output rules are in your bundle. Read it fully before producing findings. + +Read first: +- {bundle_dir}/agent-N-bundle.md (XXXX lines) — source + SOP + gap-hunter specialty + shared rules. + +The bundle contains all in-scope source. Do NOT re-read in-scope files +for the initial scan. Use Read/Grep only for cross-file searches or +out-of-scope context (interfaces/, mocks/, test/, and dependency +sources under node_modules/ or lib/ — read a base contract when the +behaviour it inherits matters). + +You are READ-ONLY inside the audited repository. Never create, edit or +delete a file there — not a Foundry PoC, not a test, not a scratch note, +not even one you intend to delete afterwards. An audit that changes the +code it is measuring is not an audit. Write proof-of-concept code in your +own scratchpad, or quote it in your finding as text. + +What a finding looks like: +- file, function +- seam — which two or three lenses combine +- root cause — the one-sentence code-level defect that lives at the seam +- minimal fix — the smallest change that eliminates the defect +- proof — concrete numbers, a trace, or quoted code showing the seam + +Without concrete proof of the seam, it's a LEAD, not a finding. +Leads are honest about what you couldn't verify — they're not failures, +they're calibration. Emit them. + +Don't skim. Don't trust your first read. Trust your discomfort. + +Write every description in Simplified Technical English — the rules are +in your bundle, in "Report language". One sentence, 25 words or fewer, +active voice, no metaphor, and it names who acts and what they get. +Your bug_class label, the identifiers and any code you quote are data: +write those exactly as the source and the output rules require. + +Your bundle ends with "Known findings — ground already walked". Obey it: +spend your effort on new ground, report every bug you find in full — +the listed ones included — and reuse its bug-class labels for the same +class of bug in the same function. + +Output format: see shared-rules.md inside your bundle (gap-hunter-specific +output fields are in your specialty file). +``` + +The same paragraph, under the same condition as Turn 3a-i: memory on and the file appended, or the paragraph is left out. + diff --git a/skills/pashov-audit-pipeline/references/assemble.sh b/skills/pashov-audit-pipeline/references/assemble.sh new file mode 100755 index 0000000..38b3222 --- /dev/null +++ b/skills/pashov-audit-pipeline/references/assemble.sh @@ -0,0 +1,353 @@ +#!/usr/bin/env bash +# THE ASSEMBLER. It writes the whole solidity-auditor report. SKILL.md Turn 5 runs it. +# +# This is the only producer of a report in this skill. The orchestrator does not compose one, +# and must not be re-taught to: an orchestrator under context pressure is the thing that +# failed. Every finding this scan gated is in a run file; this script prints all of them or +# says out loud that it could not. +# +# assemble.sh --dir .solidity-auditor/runs/{stamp} +# +# Reads ONE directory and writes {dir}/full-report.md — the complete report, banner line down +# to the disclaimer, ready to print word for word. No model judgment runs in here. +# +# reads dir/run-K.md the run files (ticket 01 shape), one per pass, model-written +# dir/scope.tsv key value, append-only, LAST LINE PER KEY WINS +# dir/memory-before.tsv the pruned photocopy — its PRESENCE is not the memory flag, +# scope.tsv's `mem_before` key is +# dir/source-names.tsv normalised as spelled in the source +# writes dir/full-report.md +# +# The model types the markers, so it can mistype one. The assembler CHECKS them and prints +# the disagreement into the Scope table. A broken run file becomes a visible line in the +# report, never a silent loss. +# +# Written to .part and moved into place on exit 0, so a half-report never exists. +set -euo pipefail + +dir= +while [ $# -gt 0 ]; do + case "$1" in + --dir) dir=$2; shift 2 ;; + *) echo "assemble.sh: unknown argument $1" >&2; exit 2 ;; + esac +done +[ -d "$dir" ] || { echo "assemble.sh: no such directory $dir" >&2; exit 2; } + +out="$dir/full-report.md" +work="$(mktemp -d)" +trap 'rm -rf "$work"' EXIT + +THRESHOLD=75 # judging.md sets it. Named there and read here; never a second opinion. + +# ---------------------------------------------------------------- scope.tsv, last wins +# An absent key gives an absent row. That is what keeps a plain scan's table at four rows. +# A missing scope.tsv is every key missing at once — the same path, no special case. It must +# not be a non-zero exit either: `set -e` would turn one absent label into no report at all. +scope() { + [ -f "$dir/scope.tsv" ] || return 0 + awk -F'\t' -v k="$1" '$1==k { v=$2 } END { print v }' "$dir/scope.tsv" +} + +name=$(scope name); [ -n "$name" ] || name='**name missing**' +mode=$(scope mode); [ -n "$mode" ] || mode='**Mode missing**' +passes_planned=$(scope passes_planned) +mem_before=$(scope mem_before) +mem_after=$(scope mem_after) +mem_sha=$(scope mem_sha) + +memory_on=0; [ -n "$mem_before" ] && memory_on=1 +if [ "$memory_on" = 1 ] && [ ! -f "$dir/memory-before.tsv" ]; then + echo "assemble.sh: memory is on but $dir/memory-before.tsv is missing — Turn 5 did not copy it" >&2 + memory_on=0 +fi + +# run files in PASS order. A plain glob puts run-10 before run-2, which is a real 10-pass bug. +runs=() +while IFS= read -r f; do runs+=("$f"); done < <( + ls "$dir"/run-*.md 2>/dev/null | sed 's/.*run-\([0-9]*\)\.md/\1 &/' | sort -n | cut -d' ' -f2) +N=${#runs[@]} + +# ---------------------------------------------------------------- index every block +# key conf kind agents file pass start end bodystart bodyend title loc leadbody +# +# A FINDING and a LEAD are NOT the same shape on disk, and one geometry read over both is what +# produced `- **- **Title**` rows and a "Body missing" line under leads that had a body: +# +# FINDING LEAD +# +0 +0 +# +1 +1 +# +2 [95] **1. T** +2 - **T** — `loc` · TAG — smells — why not verified +# +3 +3 +# +4 `loc` · Conf +4 +# +5 +# +6 body … +# +# Read with the finding geometry, a lead put its WHOLE formatted line in `title` (which the +# emitter then wrapped in `- **…**` a second time), found no backticks at +4 so `loc` came out +# empty, and had no body region at all. Each kind now gets its own reader. +# +# **No field is ever emitted empty** — `-` stands in. `read` below splits on IFS=$'\t', and a +# tab is IFS *whitespace*, so bash collapses `\t\t` into one delimiter and every later field +# shifts left by one. That is how a lead's memory tag ended up printed as its location. +: > "$work/index.tsv" +for f in ${runs[@]+"${runs[@]}"}; do + awk -v F="$f" ' + /^[[:space:]]*$/, "", line) + n=split(line, a, /[[:space:]]+/) + for (i=2; i<=n; i++) { eq=index(a[i],"="); if (substr(a[i],1,eq-1)=="pass") pass=substr(a[i],eq+1) } } + /^[[:space:]]*$/, "", line) # terminator off BEFORE splitting + n = split(line, a, /[[:space:]]+/) + for (i = 2; i <= n; i++) { + eq = index(a[i], "="); nm = substr(a[i], 1, eq-1); v = substr(a[i], eq+1) + if (nm == "key") key = v; else if (nm == "conf") conf = v + else if (nm == "kind") kind = v; else if (nm == "agents") agents = v + } + if (conf == "") conf = "-" # a LEAD is not scored + start = NR; title = ""; loc = ""; lbody = "" + next + } + kind != "LEAD" && start && NR == start + 2 { t = $0; sub(/^\[[0-9]+\] /, "", t); sub(/^\*\*/, "", t); sub(/\*\*$/, "", t); title = t } + kind != "LEAD" && start && NR == start + 4 { l = $0; if (match(l, /`[^`]*`/)) loc = substr(l, RSTART+1, RLENGTH-2) } + + # A lead is one line. Split it here, once, so the emitter formats leads and findings with + # the same fields instead of re-printing a line that is already formatted. + # `match` + RSTART/RLENGTH, never index()+a literal width: the separator is an em dash, + # 3 bytes and 1 character, and the two disagree across awks and locales. + kind == "LEAD" && start && NR == start + 2 && /^- \*\*/ { + l = $0 + sub(/^- \*\*/, "", l) + if (match(l, /\*\* — /)) { title = substr(l, 1, RSTART-1); l = substr(l, RSTART+RLENGTH) } + else { title = l; l = "" } + if (match(l, /^`[^`]*`/)) { loc = substr(l, RSTART+1, RLENGTH-2); l = substr(l, RSTART+RLENGTH) } + # What is left is " · NEW — body" or " — body". Either way the body starts after the + # first " — ". The inline tag is DROPPED: the assembler derives the memory tag itself + # from memory-before.tsv, and two sources for one fact is how they come to disagree. + if (match(l, / — /)) l = substr(l, RSTART+RLENGTH); else l = "" + lbody = l + } + /^/ && start { + if (title == "") title = "-" + if (loc == "") loc = "-" + if (lbody == "") lbody = "-" + print key "\t" conf "\t" kind "\t" agents "\t" F "\t" pass "\t" start "\t" NR "\t" (start+6) "\t" (NR-1) "\t" title "\t" loc "\t" lbody + start = 0; kind = "" + } + ' "$f" >> "$work/index.tsv" +done + +# ---------------------------------------------------------------- the structure check +# The model typed these markers. When the counts disagree, a finding was raised and cannot be +# read — say so IN the report. The report may print less than the scan found; it may never +# claim to print more. +# +# No run file at all is the same rule at its limit: pass 1 produced nothing, so the scan +# reviewed nothing. That is a REPORT, not an exit — SKILL.md Turn 3b routes a real scan down +# here, and a shell error in place of a report is the one outcome this skill must never have. +# The row is necessary because the Passes row is suppressed on a 1-pass scan, so without it +# nothing in the report would say the pass died. +broken= +if [ "$N" = 0 ]; then + broken="⚠️ No pass produced a run file. This scan reviewed nothing." + echo "assemble.sh: NO RUN FILES — $broken" >&2 +else + opens=$(cat "${runs[@]}" | grep -c '^' || :) + indexed=$(wc -l < "$work/index.tsv" | tr -d ' ') + if [ "$opens" != "$closes" ] || [ "$opens" != "$indexed" ]; then + broken="⚠️ $opens findings marked, $indexed readable. $(( opens - indexed )) could not be read and are missing below. See the run files." + echo "assemble.sh: STRUCTURE BROKEN — $broken" >&2 + fi +fi + +# ---------------------------------------------------------------- dedup across runs +# One winner per key: strongest kind, then highest confidence, then the LATER pass — a later +# pass read the ledger every earlier pass wrote, so it is the better-informed write-up. +# k counts every run that raised the key, in either section. +sort -t $'\t' -k1,1 "$work/index.tsv" | awk -F'\t' ' + function rank(k) { return (k == "FINDING") ? 1 : 0 } + { + if (!(($5 SUBSEP $1) in seenrun)) { seenrun[$5 SUBSEP $1]=1; k[$1]++ } + c = ($2 == "-") ? -1 : $2 + 0 + if (!($1 in best) || rank($3) > br[$1] || (rank($3) == br[$1] && c > bc[$1]) \ + || (rank($3) == br[$1] && c == bc[$1] && $6 + 0 >= bp[$1])) { + best[$1] = $0; br[$1] = rank($3); bc[$1] = c; bp[$1] = $6 + 0 + } + } + END { for (key in best) print k[key] "\t" best[key] } +' > "$work/merged.tsv" + +# ---------------------------------------------------------------- the memory tag +# Derived here from memory-before.tsv, not carried in the marker. One source: the report and +# the ledger cannot disagree because both read the same photocopy. +# KNOWN (n scans) — the STORED count plus one; this scan has just found it again. +if [ "$memory_on" = 1 ]; then + awk -F'\t' -v OFS='\t' ' + FILENAME==ARGV[1] { if (FNR>1) sc[$1]=$3; next } + { print $0, ($2 in sc) ? "KNOWN (" sc[$2]+1 " scans)" : "NEW" } + ' "$dir/memory-before.tsv" "$work/merged.tsv" > "$work/merged.tagged.tsv" +else + awk -F'\t' -v OFS='\t' '{ print $0, "" }' "$work/merged.tsv" > "$work/merged.tagged.tsv" +fi + +# columns now: 1 k · 2 key · 3 conf · 4 kind · 5 agents · 6 file · 7 pass · 8 start · 9 end +# 10 bodystart · 11 bodyend · 12 title · 13 loc · 14 leadbody · 15 memtag +# 12, 13 and 14 are never empty — `-` stands in, because IFS=$'\t' collapses adjacent tabs. +sort -t $'\t' -k4,4 -k3,3nr -k2,2 "$work/merged.tagged.tsv" > "$work/sorted.tsv" +awk -F'\t' '$4=="FINDING"' "$work/sorted.tsv" > "$work/findings.tsv" +awk -F'\t' '$4=="LEAD"' "$work/sorted.tsv" > "$work/leads.tsv" + +nfind=$(wc -l < "$work/findings.tsv" | tr -d ' ') +nlead=$(wc -l < "$work/leads.tsv" | tr -d ' ') + +# ---------------------------------------------------------------- emit +R="$work/report.md" +: > "$R" +say() { printf '%s\n' "$1" >> "$R"; } +rule() { say ""; say "---"; say ""; } +row() { printf '| %s | %s |\n' "$1" "$2" >> "$R"; } + +say "# 🔐 Security Review — $name" +rule +say "## Scope" +say "" +row "" "" +row "---" "---" +row "**Mode**" "$mode" + +# Files reviewed — 3 per line, in source.md order, joined with `
`. One `files` key holds +# them space separated; shell does the wrapping. +files=$(scope files | tr ' ' '\n' | awk 'NF { printf "`%s`", $0; if (++n % 3 == 0) printf "
"; else printf " · " }' \ + | sed -e 's/ · $//' -e 's/
$//') +row "**Files reviewed**" "$files" +row "**Confidence threshold (1-100)**" "$THRESHOLD" + +# The Passes cell is COMPOSED from facts, never written as prose by a model. +if [ -n "$passes_planned" ] && [ "$passes_planned" -gt 1 ] 2>/dev/null; then + short="" + for k in $(seq 1 "$passes_planned"); do + a=$(scope "pass_${k}_agents"); fail=$(scope "pass_${k}_failed") + if [ -n "$fail" ]; then short="${short:+$short, }pass $k failed" + elif [ -n "$a" ] && [ "$a" != "12/12" ]; then short="${short:+$short, }pass $k ran $a agents"; fi + done + if [ "$N" -lt "$passes_planned" ]; then cell="$N of $passes_planned"; else cell="$passes_planned"; fi + [ -n "$short" ] && cell="$cell ($short)" + row "**Passes**" "$cell" +fi +[ "$memory_on" = 1 ] && row "**Memory**" "$mem_before records before this scan · $mem_after after · \`$mem_sha\`" +[ -n "$broken" ] && row "**Run files**" "$broken" +rule + +say "## Findings" +say "" + +emit_body() { # file bodystart bodyend — copied through, never re-worded + awk -v s="$2" -v e="$3" 'NR>=s && NR<=e' "$1" \ + | awk '{ lines[++n] = $0 } END { + b=1; while (b<=n && lines[b]=="") b++ + while (n>=b && lines[n]=="") n-- + for (i=b; i<=n; i++) print lines[i] }' +} + +i=0 +while IFS=$'\t' read -r k key conf kind agents file pass start end bstart bend title loc lbody memtag; do + i=$((i+1)) + [ "$title" = "-" ] && title="**Title missing** — the block's title line could not be read" + if [ "$loc" = "-" ]; then meta="**location missing**"; else meta="\`$loc\`"; fi + say "[$conf] **$i. $title**" + say "" + meta="$meta · Confidence: $conf" + [ "$N" -gt 1 ] && meta="$meta · seen in $k/$N runs" + [ -n "$memtag" ] && meta="$meta · $memtag" + say "$meta" + say "" + body=$(emit_body "$file" "$bstart" "$bend") + if [ -z "$body" ]; then + say "**Body missing** — pass $pass raised this finding and wrote no description." + else + printf '%s\n' "$body" >> "$R" + fi + rule +done < "$work/findings.tsv" + +if [ "$nfind" = 0 ]; then + # An empty table with a header row and no rows under it is not a report of nothing, it is a + # report that looks broken. Say it in words instead. + say "_None — this scan raised no findings._" + rule +else +say "Findings List" +say "" +say "| # | Confidence | Title |" +say "|---|---|---|" +i=0; sep=0 +while IFS=$'\t' read -r k key conf kind agents file pass start end bstart bend title loc lbody memtag; do + i=$((i+1)) + if [ "$sep" = 0 ] && [ "$conf" -lt "$THRESHOLD" ]; then + say "| | | **Below Confidence Threshold** |"; sep=1 + fi + say "| $i | [$conf] | $title |" +done < "$work/findings.tsv" +rule +fi + +say "## Leads" +say "" +say "_Vulnerability trails with concrete code smells where the full exploit path could not be completed in one analysis pass. These are not false positives — they are high-signal leads for manual review. Not scored._" +say "" +[ "$nlead" = 0 ] && say "_None._" +while IFS=$'\t' read -r k key conf kind agents file pass start end bstart bend title loc lbody memtag; do + seg="" + [ "$N" -gt 1 ] && seg="$seg · seen in $k/$N runs" + [ -n "$memtag" ] && seg="$seg · $memtag" + # The body was parsed out of the lead's own line by the indexer. Do NOT read it back out of + # the run file here: a lead has no body region, so emit_body returned nothing and every lead + # got the "Body missing" line whether or not it had one. + [ "$title" = "-" ] && title="**Title missing** — the lead's line could not be read" + if [ "$loc" = "-" ]; then locmd="**location missing**"; else locmd="\`$loc\`"; fi + [ "$lbody" = "-" ] && lbody="**Body missing** — pass $pass raised this lead and wrote no description." + say "- **$title** — $locmd$seg — $lbody" +done < "$work/leads.tsv" + +# ---------------------------------------------------------------- Known from earlier scans +# The keys this scan raised come from the index — the run files are the record, so no separate +# run-keys.tsv or scan-rows.tsv has to be carried in. +if [ "$memory_on" = 1 ]; then + held=$(( $(wc -l < "$dir/memory-before.tsv") - 1 )) + if [ "$held" -gt 0 ]; then + tail -n +2 "$dir/memory-before.tsv" | cut -f1 | sort -u > "$work/keys-before.txt" + cut -f1 "$work/index.tsv" | sort -u > "$work/keys-scan.txt" + comm -23 "$work/keys-before.txt" "$work/keys-scan.txt" > "$work/keys-left.txt" + rule + say "## Known from earlier scans" + say "" + say "_Recorded by earlier scans of this repo, not raised again by this one. Not re-checked — a record here may be fixed, or may still be live and missed. \`.solidity-auditor/memory.tsv\`._" + say "" + if [ -s "$work/keys-left.txt" ]; then + say "| Scans | Kind | Location | Title |" + say "|---|---|---|---|" + awk -F'\t' ' + FILENAME==ARGV[1] { nm[$1]=$2; next } + FILENAME==ARGV[2] { if (FNR>1) { sc[$1]=$3; ti[$1]=$5; ki[$1]=$6 } ; next } + { + split($1, p, "|") + c = (p[1] in nm) ? nm[p[1]] : p[1] # print-as-stored when the segment is not in the map + f = (p[2] in nm) ? nm[p[2]] : p[2] + print "| " sc[$1] " | " ki[$1] " | `" c "." f "` | " ti[$1] " |" + } + ' "$dir/source-names.tsv" "$dir/memory-before.tsv" "$work/keys-left.txt" >> "$R" + else + say "_None — every record in the ledger was raised again by this scan._" + fi + fi +fi + +rule +say "> ⚠️ This review was performed by an AI assistant. AI analysis can never verify the complete absence of vulnerabilities and no guarantee of security is given. Team security reviews, bug bounty programs, and onchain monitoring are strongly recommended. For a consultation regarding your projects' security, visit [https://www.pashov.com](https://www.pashov.com)" + +cp "$R" "$out.part" +mv "$out.part" "$out" +echo "assembled $out — $nfind findings, $nlead leads, $N run files" diff --git a/skills/pashov-audit-pipeline/references/dedup-and-assembly.md b/skills/pashov-audit-pipeline/references/dedup-and-assembly.md new file mode 100644 index 0000000..8bab8b2 --- /dev/null +++ b/skills/pashov-audit-pipeline/references/dedup-and-assembly.md @@ -0,0 +1,419 @@ +# Dedup, record and assembly — Turn 4 and Turn 5 + +This file holds the two turns that run **after the 12 agents return**: Turn 4 +deduplicates, gates, tags and records each pass, and Turn 5 assembles and prints +the one report. `SKILL.md` keeps the name of each turn and the condition that +decides whether it runs; the procedure is here. + +The orchestrator reads this file **once, in Turn 3b**, while it waits for the +agents. It stays in context for the later passes of a loop. + +Both turns use `{stamp}`, `{bundle_dir}`, `{passes}`, `{resolved_path}` and +`{project-name}` exactly as `SKILL.md` set them. + +--- + +## Turn 4 + +**Turn 4 — Deduplicate, validate & record.** Runs **every pass**, and it is the end of the loop body: deduplicate this pass's agent results, gate-evaluate, tag, record the pass, write the ledger. **It prints no report** — there is exactly one report per scan and Turn 5 prints it. Do NOT print an intermediate dedup list. + +> **This turn used to print the report and clean up, and both moved.** The report moved to Turn 5 so that a 1-pass scan and an N-pass scan take the **same** path through this turn — the "1 pass is byte-identical to today" constraint is easier to hold with one report writer than with a last pass that doubles as one. The auto-clean moved to Turn 5 because, left here, pass 1 deletes what pass 2 needs: `source.md`, the ledger photocopy and the accumulating rows file. A later editor must not move either back. + +Then, after the loop body has run `{passes}` times (or stopped early), go to Turn 5 once. + +1. **Dedup.** Parse every FINDING and LEAD from the 12 agents. Group by `group_key` (Contract | function | bug-class). Exact-match first; merge synonymous bug_class within same (Contract, function). Keep best per group, number sequentially, annotate `[agents: N]`. + + **MANDATORY — Canonicalise the bug-class label (HARD GATE).** Before grouping, for every (Contract, function) whose agents used **more than one** bug-class label, choose **one** label and rewrite every one of those findings to carry it. Then group. + + Choose in this order, and stop at the first that applies: + 1. A label already listed for that same contract and function in `{bundle_dir}/known-findings.md` — the repository's own word wins, always. + 2. Otherwise the label that most agents used. + 3. Otherwise the shortest label that names the **defect**, not its consequence. `zero-abort-address` over `abort-address-zero-burns-funds`. + + Merging under one label is only correct when the labels name the **same class of bug**. Two genuinely different bugs in one function keep two labels and stay two records — function isolation is about functions, and this rule is about words. + + **Why this is a gate and not a nicety.** The per-function vocabulary that keeps a key stable only constrains what an **earlier scan** wrote. A class discovered mid-scan is constrained by nothing, and a real scan proved what follows: three agents in one pass named one bug `zero-abort-address`, `unregistered-abort-address` and `abort-address-zero-burns-funds`, and the over-approval class drew three labels across three passes. Unfixed, that is three ledger records for one bug, three lines in every future `known-findings.md`, and one bug that is never recognised again. This step is the only thing standing between the fork and the ledger. Canonicalise here and the next pass inherits the canonical label, so the fork closes after one scan. + + **MANDATORY — Wide-description (group_key).** Merged group with distinct mechanisms (different `fix:`, code-level cause, or attack path) MUST list every mechanism. No dropping. Same function can have multiple coexisting bugs at the same group_key — all MUST appear. + + **MANDATORY — Function-level second pass (after group_key dedup).** Run at (Contract, function), ignoring bug_class. Agents often label coexisting bugs with different bug_class tags but reference multiple mechanisms in the body. For every (Contract, function) with multiple final findings: scan body (description, path, proof, fix) of every constituent for distinct mechanisms across bug_class boundaries. Every mechanism in any constituent body MUST appear in ≥1 final finding. + + **MANDATORY — Function isolation (HARD).** NEVER merge across different `function:` fields. Dedup only within (Contract, function). Different function = different bug. Second pass above stays WITHIN (Contract, function), never across. + + **MANDATORY — Fix preservation (HARD GATE).** Before writing merged `fix:` on a multi-finding (Contract, function): + 1. Collect every raw `fix:` from agents flagging the tuple. + 2. Group by ADD-lines (`+` lines, or equivalent require/assignment). + 3. Distinct if ADD-lines differ in: called function/expression (e.g., `require(msg.value == amount)` vs `require(zrc20 != _ETH_ADDRESS_)`), check direction (validate/restrict/ban), or checked parameter. + 4. ≥2 distinct → present as Option A, B, … — one block per distinct fix, verbatim from agent text (no paraphrase). + 5. Label intuitively: validate / restrict / allow-and-handle / ban-path. + + **Output format when 2+ distinct fixes exist:** + + ``` + **Fix (Option A —