From 46fe8ba55ed62d5eff2248f08ff95115f258a1f0 Mon Sep 17 00:00:00 2001 From: unohee Date: Wed, 11 Nov 2026 21:53:41 +0900 Subject: [PATCH 1/2] =?UTF-8?q?auto-claude=201.1.0:=20never=20rest=20?= =?UTF-8?q?=E2=80=94=20build=20the=20next=20milestone=20stacked=20while=20?= =?UTF-8?q?a=20PR=20waits=20on=20CI,=20wake=20at=201=20min,=20refuse=20in-?= =?UTF-8?q?turn=20waits?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - startBase: with a PR shipping, the next milestone starts on its branch (up to MAX_AHEAD=2 ahead), its start commit recorded; never while a deploy is broken; fix goals keep their own base - stacked ship items wait their turn; when the one ahead lands or is held, git rebase --onto drops its commits; a branch that does not rebase gets a fix goal that does it by hand - final check waits for every unfinished ship, stacked ones included - the diagnose prompt waits until the milestone being built is done (no prompt mid-milestone) - watchdog wakes an idle milestone at 1, 3, 6 min, blocks at 12; planning reminder after 1 min - guard refuses long sleeps (>=60 s or computed), gh run watch, gh pr checks --watch, Monitor, CronCreate, ScheduleWakeup during a run --- README.md | 2 +- auto-claude/.claude-plugin/plugin.json | 2 +- auto-claude/README.md | 7 +- auto-claude/hooks/auto.test.ts | 112 ++++++++++++++++++++----- auto-claude/hooks/guard.ts | 31 +++++++ auto-claude/hooks/loop.ts | 11 ++- auto-claude/hooks/pure.test.ts | 51 ++++++++--- auto-claude/hooks/register.ts | 91 +++++++++++++++----- auto-claude/hooks/ship.ts | 33 +++++++- auto-claude/types/index.d.ts | 10 ++- 10 files changed, 283 insertions(+), 67 deletions(-) diff --git a/README.md b/README.md index 3e3a646..8c1df99 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ Claude Code mods (plugins with a hooks module), served as a local folder marketp | mod | version | what it does | | --- | --- | --- | | [harness](harness/) | 0.3.4 | The integrated harness: the user's Claude rules enforced at the tool call, session tooling and workflow automation (formerly house-rules) | -| [auto-claude](auto-claude/) | 1.0.0 | Unattended run of a repository through to live: plans the long-term goal, milestones and deploy; its own loop builds and judges each milestone in a worktree; ships each (push, PR, required CI, merge), deploys and smoke-checks it; a watchdog repairs or routes around stalls; replans until the final check passes. Absorbed goal-mode in 1.0.0 | +| [auto-claude](auto-claude/) | 1.1.0 | Unattended run of a repository through to live: plans the long-term goal, milestones and deploy; its own loop builds and judges each milestone in a worktree; ships each (push, PR, required CI, merge), deploys and smoke-checks it; a watchdog repairs or routes around stalls; replans until the final check passes. Absorbed goal-mode in 1.0.0; since 1.1.0 the session never rests: the next milestone is built, stacked, while a PR waits on CI | | [model-tier](model-tier/) | 0.11.0 | Routes each main-loop turn to fable, opus, sonnet or haiku and a reasoning effort by task complexity, with a `set_tier` tool the model calls to correct either | | [refactor](refactor/) | 0.1.3 | Enforces the architecture a repo declares in `.claude/architecture.json` at Edit/Write (layer direction, forbidden imports), notes code smells with the refactoring.guru technique to apply, `/arch audit` for violations and import cycles | diff --git a/auto-claude/.claude-plugin/plugin.json b/auto-claude/.claude-plugin/plugin.json index 2535251..c2def8b 100644 --- a/auto-claude/.claude-plugin/plugin.json +++ b/auto-claude/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "auto-claude", - "version": "1.0.0", + "version": "1.1.0", "description": "Finishes a repository's work unattended and live: Claude plans the long-term end state, its milestones and how it deploys; the built-in loop builds each milestone in its own worktree and judges it; auto-claude ships it (push, PR, CI, merge when green), deploys and smoke-checks it, and a watchdog keeps the run moving until the final check passes. Absorbs goal-mode's loop (1.0.0).", "author": { "name": "unohee" diff --git a/auto-claude/README.md b/auto-claude/README.md index 3af77c3..b083aa7 100644 --- a/auto-claude/README.md +++ b/auto-claude/README.md @@ -44,6 +44,7 @@ goal-mode merged into auto-claude in 1.0.0 (see "From goal-mode" below). Its loo - Right before landing, the milestone's check runs once more in a **fresh checkout**: a detached worktree at `worktrees/auto-verify`, made from the branch head fetched from origin (github) or from the rebased commit (local). A check that passes only in the builder's worktree becomes a fix goal, like a red CI. - When CI fails or the PR conflicts, auto-claude appends a fix goal built on the milestone's branch (`:: base: goal/NN-…`), and the loop builds that next. After 3 failures the milestone is held. - **Deploy.** After each merge, as the plan says (see "Deploying"). + - **The session never rests through CI.** While a PR waits on CI, the next milestone starts at once, **stacked** on that PR's branch: the commit it starts from is recorded. Up to 2 milestones are built ahead this way. They ship in order. When the one before lands (squashed) or is held, the stacked branch runs `git rebase --onto origin/ `, which drops its parent's commits, and is then pushed like any other. A stacked branch that does not rebase cleanly gets a fix goal that does the rebase by hand. Nothing is stacked while the live deploy is broken: its repair goes first. A fix goal (`:: base:`) always starts on its own base. 4. **Finish or replan.** When the queue is done, auto-claude runs the final check on a fresh checkout of the merged default branch. - If it passes, auto-claude deploys once more if a deploy is still due. Then it writes a `## Result` section and stops. - If it fails, Claude appends new milestones from the check's output (round 2, and so on, up to `maxRounds`). @@ -103,8 +104,10 @@ A run never asks the person to look at `status` or to run a command. When someth | what stops | what happens | | --- | --- | -| A milestone is open, but the session has had no turn for 3 min: the model ended its turn to "wait", a prompt was dropped, an interrupt, or a stop the judge could not read | The milestone's prompt is sent again, after 3, 6 and 12 min of quiet. After the third, the milestone is blocked and the next one starts. A stop being judged, or a retry pause, is not quiet. Turns are tracked with `turn.start` / `turn.complete`. | -| Planning ends without `auto_plan_ready` | A reminder 2 min after that turn ended, listing what the plan files lack (15 min at most otherwise). After 3, the run stops. | +| A milestone is open, but the session has had no turn for 1 min: the model ended its turn to "wait", a prompt was dropped, an interrupt, or a stop the judge could not read | The milestone's prompt is sent again, after 1, 3 and 6 min of quiet. At 12 min, the milestone is blocked and the next one starts. A stop being judged, or a retry pause, is not quiet. Turns are tracked with `turn.start` / `turn.complete`. | +| The model waits inside a turn | Refused while a run is on: `sleep` of 60 s or more (or a computed duration), `gh run watch`, `gh pr checks --watch`, `Monitor`, `CronCreate` and `ScheduleWakeup`. auto-claude watches CI, merges and deploys itself, and prompts the session when there is work. Shorter sleeps (a check waiting for a server) pass. | +| A shipping error to diagnose while the session builds another milestone | The diagnosis prompt waits until that milestone is done, so the stop judge never judges the wrong work. | +| Planning ends without `auto_plan_ready` | A reminder 1 min after that turn ended, listing what the plan files lack (15 min at most otherwise). After 3, the run stops. | | An SSH remote refuses the push (`Permission denied (publickey)`) | The push goes to the https URL of the same repository. | | The same shipping error, check after check | At 5 checks, the session model gets the error and what was tried, and fixes what it can (gh auth, remotes, `gh repo set-default`). At 10, the milestone is held and the run goes on. | | A PR closed under the run | It is opened again once, then held. | diff --git a/auto-claude/hooks/auto.test.ts b/auto-claude/hooks/auto.test.ts index 94bab54..05ab96b 100644 --- a/auto-claude/hooks/auto.test.ts +++ b/auto-claude/hooks/auto.test.ts @@ -72,6 +72,9 @@ function world(on: On, store: Record = {}) { deployExit: 0, smokeExit: 0, missing: new Set(), + // Every `git rebase`, and whether a stacked branch's rebase --onto conflicts. + rebases: [] as string[], + rebaseFails: false, clock: mock.clock(on, { now: 1_000 }), } const removed = (path: string) => [...w.gone].some(g => path === g || path.startsWith(`${g}/`)) @@ -172,7 +175,9 @@ function world(on: On, store: Record = {}) { if (a.startsWith('gh run list')) return ok(w.runList) if (e.argv[0] === 'test' && e.argv[1] === '-d') return ok('', w.gone.has(String(e.argv[2])) ? 1 : 0) if (e.argv[0] === 'test' && e.argv[1] === '-e') return ok('', w.missing.has(String(e.argv[2])) ? 1 : 0) - if (a.includes('rev-parse --verify')) return ok('', w.branchGone ? 1 : 0) + if (a.includes('rev-parse --verify')) return w.branchGone ? ok('', 1) : ok(`tip-of-${String(e.argv.at(-1)).replace(/\^\{commit\}$/, '')}\n`) + if (a.includes(' rebase ')) w.rebases.push(a) + if (a.includes(' rebase --onto') && w.rebaseFails) return ok('', 1, 'CONFLICT (content): Merge conflict in calc.py') if (a.includes('worktree add') && !a.includes('--detach')) { w.adds.push(a) w.gone.delete(String(e.argv[5])) @@ -439,7 +444,7 @@ describe('a run, local CI', () => { expect((await cmd($, 'status')).text).toContain('no plan was accepted after 3 reminders') }) - test('a planning turn that ends without a plan is reminded 2 min later, not 15 (P4)', async ($, on) => { + test('a planning turn that ends without a plan is reminded 1 min later, not 15 (P4)', async ($, on) => { const w = world(on) await $.session.start({ cwd: '/repo', surface: 'terminal', isInteractive: true }) await cmd($, 'start') @@ -571,6 +576,63 @@ describe('a run, github CI', () => { expect(w.merged).toHaveLength(2) }) + test('the session keeps building while a PR waits on CI: the next milestone is stacked on its branch, then rebased off it and shipped once it lands', { timeoutMs: 30_000 }, async ($, on) => { + const w = world(on) + w.github = true + w.checks = '[{"bucket":"pending"}]' + await planned($, w) + await meet($, w) + expect(w.merged).toHaveLength(0) + // Milestone 2 started at once, on milestone 1's branch, with the commit it builds on kept. + expect(w.adds.at(-1)).toBe(`git -C /repo worktree add ${WT2} -b ${BR2} ${BR1}`) + expect(w.prompts.at(-1)).toContain('Milestone #2') + // Met while milestone 1 still waits: it waits its turn, unpushed. + await meet($, w) + await w.clock.advance(TICK) + expect(w.pushes.some(p => p.includes(BR2))).toBe(false) + expect((await cmd($, 'status')).text).toContain('- stacked: sub() returns a-b') + // Milestone 1 lands; milestone 2 drops its commits, then is pushed and landed the same way. + w.checks = '[{"bucket":"pass"}]' + await w.clock.advance(10_000) + expect(w.merged).toHaveLength(2) + expect(w.rebases).toContain(`git -C ${WT2} rebase --onto origin/main tip-of-${BR1}`) + expect(w.pushes.some(p => p.includes(`${BR2}:${BR2}`))).toBe(true) + for (let i = 0; i < 3; i++) await w.clock.advance(TICK) + expect(w.notifies.at(-1)).toContain('finished: the final check passed after 1 round(s). 2 merged, 0 held') + }) + + test('a stacked milestone that does not rebase once its parent lands gets a fix goal that rebases it by hand', { timeoutMs: 30_000 }, async ($, on) => { + const w = world(on) + w.github = true + w.checks = '[{"bucket":"pending"}]' + await planned($, w) + await meet($, w) + await meet($, w) + w.rebaseFails = true + w.checks = '[{"bucket":"pass"}]' + await w.clock.advance(TICK) + expect(w.merged).toHaveLength(1) + expect(w.runs).toContain(`git -C ${WT2} rebase --abort`) + const fix = parseQueue(w.files.get(GOALS) ?? '').find(i => i.base !== undefined) + expect(fix?.base).toBe(BR2) + expect(fix?.condition).toContain(`git rebase --onto origin/main tip-of-${BR1}`) + // The fix goal is built next, on milestone 2's branch. + expect(w.adds.at(-1)).toContain(`-b goal/03-`) + expect(w.adds.at(-1)).toMatch(new RegExp(`${BR2}$`)) + }) + + test('nothing is stacked while the deploy is broken: its repair goes first', { timeoutMs: 30_000 }, async ($, on) => { + const w = world(on) + w.github = true + w.deployExit = 1 + await planned($, w, PLAN_DEPLOY) + await meet($, w) + // Milestone 1 landed and its deploy failed: the repair line, from main, comes before milestone 2. + expect(w.merged).toHaveLength(1) + expect(w.adds.at(-1)).toMatch(/origin\/main$/) + expect(w.prompts.at(-1)).toMatch(/deploy/i) + }) + test('only required checks decide: a red check that is not required does not block the merge (P1)', async ($, on) => { const w = world(on) w.github = true @@ -646,17 +708,17 @@ describe('a run, github CI', () => { }) describe('the watchdog (P2, P4, P5)', () => { - test('a milestone whose session sits idle is prompted again at 3, 6 and 12 min, then blocked, and the run goes on', { timeoutMs: 30_000 }, async ($, on) => { + test('a milestone whose session sits idle is prompted again at 1, 3 and 6 min, then blocked, and the run goes on', { timeoutMs: 30_000 }, async ($, on) => { const w = world(on) await planned($, w) const wakes = () => w.prompts.filter(p => p.includes('the session sat idle') && p.includes('Milestone #1')) // The model ended its turn without a stop the loop could judge. await turnStart($) await turnEnd($) - await w.clock.advance(2 * MIN) + await w.clock.advance(TICK) expect(wakes()).toHaveLength(0) - // Ticks are 30 s apart: the first one past 3 min of quiet wakes it. - await w.clock.advance(MIN + TICK + 1_200) + // Ticks are 30 s apart: the first one past 1 min of quiet wakes it. + await w.clock.advance(2 * TICK) expect(wakes()).toHaveLength(1) expect(wakes()[0]).toContain('Milestone #1: add() returns a+b') expect(wakes()[0]).toContain('wake 1/3') @@ -726,20 +788,25 @@ describe('the watchdog (P2, P4, P5)', () => { w.gitGone = true await planned($, w) await meet($, w) + // Milestone 2 is built meanwhile, stacked on milestone 1's branch. + expect(w.adds.at(-1)).toBe(`git -C /repo worktree add ${WT2} -b ${BR2} ${BR1}`) + const asked = () => w.prompts.filter(p => p.includes('[auto-claude] Shipping is stuck')) for (let i = 0; i < 5; i++) await w.clock.advance(TICK) - const asked = w.prompts.filter(p => p.includes('[auto-claude] Shipping is stuck')) - expect(asked).toHaveLength(1) - expect(asked[0]).toContain('ENOENT') - expect(asked[0]).toContain('Do not merge or deploy') + // The session is building milestone 2: the diagnosis waits for it. + expect(asked()).toHaveLength(0) + await turnStart($) + await meet($, w) + await w.clock.advance(TICK) + expect(asked()).toHaveLength(1) + expect(asked()[0]).toContain('ENOENT') + expect(asked()[0]).toContain('Do not merge or deploy') // Ten checks are not enough on their own: the error must also have lasted 15 min. - for (let i = 0; i < 6; i++) await w.clock.advance(TICK) + for (let i = 0; i < 4; i++) await w.clock.advance(TICK) expect(w.notifies.some(n => n.includes('held: add() returns a+b'))).toBe(false) - for (let i = 0; i < 20; i++) await w.clock.advance(TICK) + for (let i = 0; i < 26; i++) await w.clock.advance(TICK) expect(w.notifies.some(n => n.startsWith('auto-claude (repo): held: add() returns a+b') && n.includes('going on'))).toBe(true) - // The run went on to milestone 2, which hits the same wall: the run stops instead of holding every milestone. - expect(w.adds.at(-1)).toContain(WT2) - await turnStart($) - await meet($, w) + // Milestone 2 goes next, rebased off milestone 1's commits, and hits the same wall: the run stops instead of holding every milestone. + expect(w.rebases).toContain(`git -C ${WT2} rebase --onto origin/main tip-of-${BR1}`) for (let i = 0; i < 32; i++) await w.clock.advance(TICK) expect(w.notifies.at(-1)).toContain('stopped: 2 milestones in a row were held for one cause (push)') expect((await cmd($, 'status')).text).toContain('phase stopped') @@ -765,15 +832,14 @@ describe('the watchdog (P2, P4, P5)', () => { expect(w.adds.at(-1)).toContain(WT2) }) - test('an idle run is told to the person with what is being done, never asked to check status', { timeoutMs: 30_000 }, async ($, on) => { + test('an idle session never rests long and is never asked to check status: woken, then its milestone blocked', { timeoutMs: 30_000 }, async ($, on) => { const w = world(on) await planned($, w) - // The session does nothing with its milestone: woken at 3 and 6 min, told at 18. - for (let i = 0; i < 40; i++) await w.clock.advance(TICK) - const idle = w.notifies.filter(n => n.includes('idle ')) - expect(idle).toHaveLength(1) - expect(idle[0]).toContain('the session has done nothing for') - expect(idle[0]).toContain('goes on by itself') + // The session does nothing with its milestone: woken at 1, 3 and 6 min, blocked at 12, the next one started. + for (let i = 0; i < 26; i++) await w.clock.advance(TICK) + expect(w.prompts.filter(p => p.includes('the session sat idle'))).toHaveLength(3) + expect(w.adds.at(-1)).toContain(WT2) + expect(w.prompts.at(-1)).toContain('Milestone #2') for (const n of w.notifies) expect(n).not.toContain('/auto-claude status') }) }) diff --git a/auto-claude/hooks/guard.ts b/auto-claude/hooks/guard.ts index 4e22e27..573f70c 100644 --- a/auto-claude/hooks/guard.ts +++ b/auto-claude/hooks/guard.ts @@ -95,6 +95,37 @@ export function deploysSomething(command: string): boolean { return command.split(/&&|\|\||[;|&\n`()]|\$\(/).some(segment => DEPLOY.test(segment) && !readOnly(segment)) } +// Waiting inside a turn: a run never rests, and what it waits for (CI, a +// merge, a deploy) the driver watches itself. A sleep of at least +// WAIT_MAX_S, or watching CI until it ends, is refused; short sleeps (a +// check that waits for a server to come up) pass. Judged on tokens per piece. +export const WAIT_MAX_S = 60 +const UNIT_S: Record = { '': 1, s: 1, m: 60, h: 3600, d: 86400 } + +function sleepSeconds(tokens: readonly string[]): number { + let total = 0 + for (const t of tokens) { + const m = t.match(/^(\d+(?:\.\d+)?)([smhd]?)$/) + if (m === null) return Number.POSITIVE_INFINITY + total += Number(m[1]) * UNIT_S[m[2] ?? '']! + } + return total +} + +export function waitsLong(raw: string): boolean { + return stripProse(raw) + .split(/&&|\|\||[;|&\n`()]|\$\(/) + .some(segment => { + const tokens = segment.trim().split(/\s+/).filter(t => t !== '' && !/^[A-Za-z_][A-Za-z0-9_]*=/.test(t)) + const [tool, ...rest] = tokens + // No literal argument left in its piece: a computed duration, `sleep $((60*5))`. + if (tool === 'sleep') return rest.length === 0 || sleepSeconds(rest) >= WAIT_MAX_S + if (tool !== 'gh') return false + if (rest[0] === 'run' && rest[1] === 'watch') return true + return rest[0] === 'pr' && rest[1] === 'checks' && rest.includes('--watch') + }) +} + function isSafeRmTarget(target: string, worktree: string): boolean { const bare = target.replace(/^['"]|['"]$/g, '') if (bare.includes('..') || bare.startsWith('~') || bare.includes('$') || bare === '*' || bare === '.' || bare === '/') { diff --git a/auto-claude/hooks/loop.ts b/auto-claude/hooks/loop.ts index 7ac21a5..fbeb4fc 100644 --- a/auto-claude/hooks/loop.ts +++ b/auto-claude/hooks/loop.ts @@ -13,7 +13,7 @@ import type { EngineInterface } from 'claude-code' import type { Goal, LoopCurrent, LoopRecord, LoopView } from '../types' import { clip, evaluate } from './evaluator' import type { StopFields } from './evaluator' -import { checkBash, checkPath, uncommitted } from './guard' +import { checkBash, checkPath, uncommitted, WAIT_MAX_S, waitsLong } from './guard' import { markItem, nextPending, parseQueue, slugify } from './queue' export const DEFER_TOOL = 'defer_goal' @@ -314,12 +314,18 @@ export async function deferCall($: EngineInterface, reasonRaw: unknown): Promise // in it (`view`). The person cannot answer, a plan-mode turn would hold the // run, and a session goal of Claude Code's own (/goal) would block the loop's // stops (measured in de-artifact: the run idled to its 24 h cap). +// A run never rests inside a turn either: auto-claude watches CI, merges and +// deploys itself, and wakes the session when it has work for it. +const WAIT_TOOLS: ReadonlySet = new Set(['CronCreate', 'ScheduleWakeup', 'Monitor']) +const WAIT_REFUSAL = `auto-claude run: do not wait or schedule a wake-up. auto-claude watches CI, merges and deploys by itself and prompts this session when there is work; keep working on the milestone, or end the turn if it is done (sleeps under ${WAIT_MAX_S} s are fine).` + export function guardRefusal(e: Record, view: LoopView | null, runOn: boolean): string | undefined { const tool = String(e.tool) if (runOn || view !== null) { if (tool === 'AskUserQuestion') return `auto-claude run: nobody can answer. Decide with a stated assumption (the recommended or safest option), or call ${DEFER_TOOL_ID} for an external blocker.` if (tool === 'EnterPlanMode') return 'auto-claude run: plan mode would hold the run; plan inline and keep working.' if (tool === 'ProposeGoal' || tool === 'mcp__goal-mode__set_goal') return 'auto-claude run: the loop sets each milestone as the goal itself; a session goal would block it.' + if (WAIT_TOOLS.has(tool) || (tool === 'Bash' && typeof e.command === 'string' && waitsLong(e.command))) return WAIT_REFUSAL } if (view === null) return undefined if (tool === 'Bash' && typeof e.command === 'string') return checkBash(e.command, view) @@ -332,7 +338,8 @@ export const LOOP_POLICY = `# auto-claude milestone — ACTIVE You are building one milestone of an unattended auto-claude run. Nobody is watching this session. - Do not ask questions (AskUserQuestion is refused) and do not enter plan mode. Decide with a stated assumption and record it in the commit message. - A "CONFIDENCE HALT" becomes a call to ${DEFER_TOOL_ID} with the reason, and only for external blockers (credentials, a human decision, missing access) after at least ${MIN_CHECKS_TO_DEFER} failed checks. Otherwise keep working. -- Work only inside the milestone's worktree; commit after each meaningful step. Push only your goal/* branch, never main or with force. auto-claude pushes, merges and deploys for you.` +- Work only inside the milestone's worktree; commit after each meaningful step. Push only your goal/* branch, never main or with force. auto-claude pushes, merges and deploys for you. +- Never wait inside a turn (no long sleep, CI watch, Monitor, cron or scheduled wake-up). While an earlier milestone's CI runs you are already building the next one; when this one is done, end the turn and the next prompt comes by itself.` export function kickoffText(current: LoopCurrent, seq: number): string { return [ diff --git a/auto-claude/hooks/pure.test.ts b/auto-claude/hooks/pure.test.ts index 73826e2..a21ebd2 100644 --- a/auto-claude/hooks/pure.test.ts +++ b/auto-claude/hooks/pure.test.ts @@ -2,11 +2,12 @@ import { describe, expect, test } from 'claude-code/testing' import { countSarif, parseDeploy, scriptsOf, specOfPlan } from './deploy' import { parseVerdict, transcriptTail } from './evaluator' -import { checkBash, checkPath, deploysSomething, uncommitted } from './guard' -import { maxBranchSeq } from './loop' +import { checkBash, checkPath, deploysSomething, uncommitted, waitsLong } from './guard' +import { guardRefusal, maxBranchSeq } from './loop' +import type { ShipState } from '../types' import { deployFixLine, deployOf, finalCheckOf, fixLine, kickoffText, POLICY, validatePlan } from './plan' import { markItem, nextPending, parseQueue, slugify, summarize } from './queue' -import { capReached, causeOf, ciAction, httpsOf, idleDue, inFlight, isWaiting, mergeable, noRequired, onlyCancelled, parseChecks, prNumber, progressSig, readLog, repoSlug, shipRemedy, wakeDue } from './ship' +import { capReached, causeOf, ciAction, httpsOf, idleDue, inFlight, isWaiting, mergeable, noRequired, onlyCancelled, parseChecks, prNumber, progressSig, readLog, repoSlug, shipRemedy, startBase, unfinished, MAX_AHEAD, wakeDue } from './ship' const PLAN = '## Goal\nadd and sub work\n\nFinal check: `python3 -m unittest -q`\nDeploy: `bash scripts/deploy.sh`\nSmoke check: `curl -fsS http://localhost:8080/health`\nCodeQL: python\n\n## Milestones\n1. add\n2. sub\n\n## Decisions\n' const QUEUE = '- [ ] add() returns a+b :: check: python3 -m unittest -q test_add\n- [ ] sub() returns a-b :: check: python3 -m unittest -q test_sub\n' @@ -198,16 +199,33 @@ describe('ship', () => { expect(causeOf('its worktree /x and branch goal/01 are gone')).toBe('worktree') }) - test('a milestone open in a quiet session is woken at 3, 6 and 12 min of quiet; quiet is not waiting (P4, P5)', () => { - expect(wakeDue(2 * MIN, 0)).toBe(false) - expect(wakeDue(3 * MIN, 0)).toBe(true) - expect(wakeDue(5 * MIN, 1)).toBe(false) - expect(wakeDue(6 * MIN, 1)).toBe(true) - expect(wakeDue(12 * MIN, 2)).toBe(true) + test('a milestone open in a quiet session is woken at 1, 3 and 6 min of quiet, blocked at 12; quiet is not waiting (P4, P5)', () => { + expect(wakeDue(MIN - 1, 0)).toBe(false) + expect(wakeDue(MIN, 0)).toBe(true) + expect(wakeDue(2 * MIN, 1)).toBe(false) + expect(wakeDue(3 * MIN, 1)).toBe(true) + expect(wakeDue(6 * MIN, 2)).toBe(true) + expect(wakeDue(11 * MIN, 3)).toBe(false) + expect(wakeDue(12 * MIN, 3)).toBe(true) const building = { started: 1, finished: 0, ships: [] } expect(isWaiting(building, 0)).toBe(true) - expect(isWaiting(building, 0, 2 * MIN)).toBe(true) - expect(isWaiting(building, 0, 3 * MIN)).toBe(false) + expect(isWaiting(building, 0, 30_000)).toBe(true) + expect(isWaiting(building, 0, MIN)).toBe(false) + }) + + test('the next milestone starts on the default branch, a fix on its own base, else stacked on the last one shipping, at most MAX_AHEAD deep', () => { + const ship = (headRef: string, state: ShipState) => ({ goal: headRef, headRef, branch: headRef, state, red: 0, since: 0 }) + expect(startBase({ ships: [] }, {}, 'origin/main')).toEqual({ base: 'origin/main', stacked: false }) + expect(startBase({ ships: [ship('a', 'merged'), ship('b', 'held')] }, {}, 'origin/main')).toEqual({ base: 'origin/main', stacked: false }) + expect(startBase({ ships: [ship('a', 'ci')] }, { base: 'goal/01-a' }, 'origin/main')).toEqual({ base: 'goal/01-a', stacked: false }) + expect(startBase({ ships: [ship('a', 'ci')] }, {}, 'origin/main')).toEqual({ base: 'a', stacked: true }) + expect(startBase({ ships: [ship('a', 'ci'), ship('b', 'stacked')] }, {}, 'origin/main')).toEqual({ base: 'b', stacked: true }) + expect(MAX_AHEAD).toBe(2) + expect(startBase({ ships: [ship('a', 'ci'), ship('b', 'stacked'), ship('c', 'stacked')] }, {}, 'origin/main')).toBeUndefined() + // A broken deploy: its repair goes first, nothing piles on. + expect(startBase({ ships: [ship('a', 'ci')], deployFails: 1 }, {}, 'origin/main')).toBeUndefined() + expect(unfinished([ship('a', 'merged'), ship('b', 'fixing'), ship('c', 'stacked')]).map(s => s.headRef)).toEqual(['b', 'c']) + expect(inFlight([ship('c', 'stacked')])).toBeUndefined() }) test('progressSig ignores an error note; idleDue is 15 min, then 30 min apart, at most 3', () => { @@ -305,6 +323,17 @@ describe('queue', () => { const CTX = { repoRoot: '/repo', worktree: '/repo/worktrees/goal-01-x', extraDeny: ['\\bplace_order\\b'] } describe('guard', () => { + test('a run never waits inside a turn: long sleeps, CI watches, cron and wake-ups are refused; short sleeps pass', () => { + for (const c of ['sleep 300', 'sleep 1m', 'sleep 30 30', 'make test && sleep 120', 'gh run watch 123', 'gh pr checks 7 --watch', 'X=1 sleep 90', 'sleep $((60*5))', 'echo $(sleep 600)']) expect(waitsLong(c)).toBe(true) + for (const c of ['sleep 5', 'sleep 0.5', 'npm test', 'gh pr checks 7', 'git commit -m "sleep 600 then gh run watch"', 'echo sleep 600']) expect(waitsLong(c)).toBe(false) + for (const tool of ['CronCreate', 'ScheduleWakeup', 'Monitor']) { + expect(guardRefusal({ tool }, null, true)).toContain('do not wait or schedule a wake-up') + expect(guardRefusal({ tool }, null, false)).toBeUndefined() + } + expect(guardRefusal({ tool: 'Bash', command: 'sleep 900' }, null, true)).toContain('auto-claude watches CI') + expect(guardRefusal({ tool: 'Bash', command: 'sleep 3' }, null, true)).toBeUndefined() + }) + const cases: Array<[string, boolean]> = [ ['git push origin main', false], ['git push origin HEAD:master', false], diff --git a/auto-claude/hooks/register.ts b/auto-claude/hooks/register.ts index 3a1e431..ef7dfe9 100644 --- a/auto-claude/hooks/register.ts +++ b/auto-claude/hooks/register.ts @@ -9,7 +9,8 @@ import { deployMerged } from './deploy-run' import { bootstrapRepo } from './bootstrap' import { ensureLines, FILES, fetchRemote, LOCAL, out, readText, refOf, repoRoot, run, tell } from './io' import { appendLog, blockCurrent, DEFER_TOOL, DEFER_TOOL_ID, deferCall, guardRefusal, LOOP_EXCLUDES, LOOP_POLICY, loopView, MIN_CHECKS_TO_DEFER, onStop, readLoop, resumeLoop, setOnFinish, startNext, stopLoop, watch, writeLoop } from './loop' -import { capReached, causeOf, ciAction, diagnose, httpsOf, idleDue, inFlight, isWaiting, LOOP_WAKES, MAX_IDLE_NUDGES, MAX_RED, mergeable, noRequired, onlyCancelled, parseChecks, prNumber, progressSig, readLog, repoSlug, shipRemedy, summary, wakeDue } from './ship' +import { capReached, causeOf, ciAction, diagnose, httpsOf, idleDue, inFlight, isWaiting, LOOP_WAKES, MAX_IDLE_NUDGES, MAX_RED, mergeable, noRequired, onlyCancelled, parseChecks, prNumber, progressSig, readLog, repoSlug, shipRemedy, startBase, summary, unfinished, wakeDue } from './ship' +import { nextPending } from './queue' import type { StallFacts } from './ship' import type { Caps } from './ship' @@ -23,7 +24,9 @@ import type { Caps } from './ship' // model is denied merging and deploying. A watchdog keeps the run moving: // what it can fix it fixes, what needs the model it prompts the model with, // and what stays stuck it holds and goes past. goal-mode, which used to own -// the loop, merged into this plugin in 1.0.0. +// the loop, merged into this plugin in 1.0.0. The session never rests while +// work is queued: during a PR's CI the next milestones are built on its +// branch (stacked), then rebased onto the default branch once it lands. const COMMAND = 'auto-claude' // The ready tool's full name, as the tool.call matcher sees it. @@ -36,7 +39,7 @@ const DENY = ['\\bgh\\s+pr\\s+merge\\b', '\\bgh\\s+release\\b', '\\bgit\\s+push\ const STALE_MS = 10 * 60_000 // A run still planning is reminded this long after the model's turn ended // without accepting a plan, else after PLAN_NUDGE_MS; this many times, then stopped. -const PLAN_QUIET_MS = 2 * 60_000 +const PLAN_QUIET_MS = 60_000 const PLAN_NUDGE_MS = 15 * 60_000 const MAX_NUDGES = 3 // While a PR's CI is pending it is looked at this often, between the ticks. @@ -122,14 +125,18 @@ function clearSessionGoal($: EngineInterface): void { }) } -// Starts the next milestone when nothing is being built or shipped: the loop -// is called directly, so a start the driver wants cannot be refused. -async function startWanted($: EngineInterface, r: AutoRecord): Promise { +// Starts the next milestone when nothing is being built: the loop is called +// directly, so a start the driver wants cannot be refused. A stacked start +// records the commit it builds on, for the rebase once its parent lands. +async function startWanted($: EngineInterface, r: AutoRecord, base: string, stacked: boolean): Promise { // Every start the log shows has finished: a goal met during this tick but - // not read yet must ship before its successor starts from the default branch. + // not read yet must become a ship item before its successor picks a base. if (r.started > r.finished) return if ((await readLoop($, r.root))?.current !== undefined) return - await startNext($, r.root, refOf(r), r.remote) + const tip = stacked ? (await run($, ['git', '-C', r.root, 'rev-parse', '--verify', '--quiet', `${base}^{commit}`], r.root, 15_000)).stdout.trim() : '' + if (stacked && tip === '') return + const current = await startNext($, r.root, base, r.remote) + if (current !== undefined && stacked) await writeRecord($, { ...r, onto: { ...r.onto, [current.branch]: tip } }) } // Another session owns the run. Once its ticks stop, the run is stuck there: @@ -441,8 +448,17 @@ async function planTick($: EngineInterface, r: AutoRecord): Promise { // start in this one tick. async function shipSteps($: EngineInterface, r: AutoRecord): Promise { for (let steps = 0; steps < MAX_STEPS && r.on; steps++) { - const item = inFlight(r.ships) - if (item === undefined || item.state === 'fixing') break + let item = inFlight(r.ships) + if (item === undefined) { + // The milestone ahead landed or was held: the first stacked one goes next. + const waiting = r.ships.find(s => s.state === 'stacked') + if (waiting === undefined) break + const moved = await unstack($, r, waiting) + r = { ...moved.r, ships: moved.r.ships.map(s => (s.headRef === waiting.headRef ? moved.item : s)) } + if (moved.item.state !== 'push') break + item = moved.item + } + if (item.state === 'fixing') break const step = await advance($, r, item) if (!step.r.on) return step.r const same = step.item.note !== undefined && step.item.note === item.note && step.item.state === item.state @@ -456,6 +472,29 @@ async function shipSteps($: EngineInterface, r: AutoRecord): Promise return r } +// A stacked milestone whose turn has come: its branch is rebased onto the +// default branch, dropping the commits of the milestone it was built on +// (landed as one squashed commit, or held and never landed). One that does +// not rebase gets a fix goal that does it by hand. +async function unstack($: EngineInterface, r: AutoRecord, item: ShipItem): Promise<{ r: AutoRecord; item: ShipItem }> { + const where = await worktreeOf($, r, item) + if ('gone' in where) return hold($, r, item, where.gone) + const wt = where.wt + await fetchRemote($, r, wt) + const onto = item.onto + const args = onto !== undefined ? ['rebase', '--onto', refOf(r), onto] : ['rebase', refOf(r)] + const rebased = await run($, ['git', '-C', wt, ...args], wt) + const now = await $.clock.now() + const { onto: _done, ...rest } = item + if (rebased.exitCode === 0) return { r, item: { ...rest, state: 'push', since: now } } + await run($, ['git', '-C', wt, 'rebase', '--abort'], wt) + const how = onto !== undefined ? `git rebase --onto ${refOf(r)} ${onto}` : `git rebase ${refOf(r)}` + const failure = `it was built on a milestone that has since landed or been held, and does not rebase onto ${r.defaultBranch}; run ${how} and resolve the conflicts: ${out(rebased).slice(0, 120)}` + const line = fixLine(item.goal, undefined, failure, item.check, item.branch) + await appendQueue($, r.root, line) + return { r: { ...r, fixes: { ...r.fixes, [parseQueue(line)[0]!.condition]: item.headRef } }, item: { ...rest, state: 'fixing', red: item.red + 1, note: failure, since: now } } +} + // An error that keeps coming back is treated, not just retried. async function treat($: EngineInterface, r: AutoRecord, item: ShipItem, now: number): Promise<{ r: AutoRecord; item: ShipItem }> { const remedy = shipRemedy(item, now) @@ -465,24 +504,25 @@ async function treat($: EngineInterface, r: AutoRecord, item: ShipItem, now: num } if (remedy === 'diagnose') { const asked = { ...item, tried: [...(item.tried ?? []), 'diagnose'] } - await askModel($, r, asked) - return { r, item: asked } + return { r, item: (await askModel($, r, asked)) ? asked : item } } if (remedy === 'hold') return hold($, r, item, `stuck at ${item.state} for ${item.repeats} checks: ${item.note ?? ''}`) return { r, item } } -// Nothing shipping: the next milestone (a fix goal included), or the end. +// The next milestone (a fix goal included), or the end. A milestone is +// started even while another ships: stacked on its branch, so the session +// keeps building through the CI (ship.ts startBase). async function nextOrFinish($: EngineInterface, r: AutoRecord, caps: Caps): Promise { - const busy = inFlight(r.ships) - if ((busy !== undefined && busy.state !== 'fixing') || r.phase !== 'run') return - const pending = parseQueue((await readText($, FILES(r.root).queue)) ?? '').some(i => i.mark === ' ') - if (pending) { - await startWanted($, r) + if (r.phase !== 'run') return + const next = nextPending(parseQueue((await readText($, FILES(r.root).queue)) ?? '')) + if (next !== undefined) { + const at = startBase(r, next, refOf(r)) + if (at !== undefined) await startWanted($, r, at.base, at.stacked) return } const building = (await readLoop($, r.root))?.current !== undefined - if (busy === undefined && !building && r.started <= r.finished) await finalCheck($, { ...r, phase: 'final' }, caps) + if (unfinished(r.ships).length === 0 && !building && r.started <= r.finished) await finalCheck($, { ...r, phase: 'final' }, caps) } async function tick($: EngineInterface, caps: Caps, pollMs: number): Promise { @@ -587,13 +627,19 @@ async function metToShip($: EngineInterface, r: AutoRecord, goal: string, branch // A fix landed: its branch carries the PR's commits plus the fix. if (parent !== undefined) return { ...r, ships: r.ships.map((s): ShipItem => (s.headRef === parent ? { ...s, branch, worktree: wt, state: 'push', repeats: 0 } : s)) } const check: string | undefined = parseQueue((await readText($, FILES(r.root).queue)) ?? '').find(i => i.condition === goal)?.check - return { ...r, ships: [...r.ships, { goal, headRef: branch, branch, worktree: wt, check, state: 'push', red: 0, since: await $.clock.now() }] } + // Built on a branch still shipping, or behind milestones still shipping: it + // waits its turn, and is rebased onto the default branch before its push. + const onto: string | undefined = r.onto?.[branch] + const { [branch]: _taken, ...rest } = r.onto ?? {} + const state = onto !== undefined || unfinished(r.ships).length > 0 ? 'stacked' : 'push' + return { ...r, onto: rest, ships: [...r.ships, { goal, headRef: branch, branch, worktree: wt, check, state, red: 0, since: await $.clock.now(), ...(onto !== undefined ? { onto } : {}) }] } } // A shipping error that keeps coming back goes to the session model with what // was tried: it can fix what the driver cannot (gh auth, a remote, a branch). -async function askModel($: EngineInterface, r: AutoRecord, item: ShipItem): Promise { - if ((await readLoop($, r.root))?.current !== undefined) return +async function askModel($: EngineInterface, r: AutoRecord, item: ShipItem): Promise { + // A milestone being built has the session: asked once it is done instead. + if ((await readLoop($, r.root))?.current !== undefined) return false const text = [ `[auto-claude] Shipping is stuck: "${item.goal.slice(0, 100)}"${item.pr !== undefined ? ` (PR #${item.pr})` : ''} has failed at ${item.state} ${item.repeats ?? 0} times in a row with:`, '```', @@ -603,6 +649,7 @@ async function askModel($: EngineInterface, r: AutoRecord, item: ShipItem): Prom 'Find the cause and fix what you can from here (gh auth status, git remote -v, the branch on the remote, gh repo set-default, the PR itself). Do not merge or deploy: auto-claude retries on its own each tick, and holds this milestone and goes on if it still fails. Record what you found under ## Decisions in .claude/auto-plan.md, then end the turn.', ].join('\n') $.prompt.submit({ text }).catch(() => {}) + return true } // The watchdog: a milestone whose session sits idle is prompted again (3, 6, diff --git a/auto-claude/hooks/ship.ts b/auto-claude/hooks/ship.ts index f816608..899f8ca 100644 --- a/auto-claude/hooks/ship.ts +++ b/auto-claude/hooks/ship.ts @@ -1,4 +1,4 @@ -import type { AutoRecord, ShipItem } from '../types' +import type { AutoRecord, ShipItem, ShipState } from '../types' // The loop's JSONL log as events, from after the last line already read. // The loop keeps only its last 2000 lines (loop.ts LOG_MAX_LINES), so once the @@ -280,12 +280,37 @@ export function httpsOf(slug: string): string { // A session with a milestone open and no turn for this long is not building it // (a dropped prompt, an interrupt, a turn that ended to "wait"): the watchdog // prompts it again, backing off, LOOP_WAKES times, then blocks the milestone. -export const LOOP_QUIET_MS = 3 * 60_000 +// The first wake comes soon: a run never leaves the session resting. +export const LOOP_QUIET_MS = 60_000 export const LOOP_WAKES = 3 +const WAKE_AT_MS = [1, 3, 6, 12].map(min => min * 60_000) -// The wait before wake number `wakes + 1`: 3, 6, then 12 min of quiet. +// The wait before wake number `wakes + 1`: 1, 3, then 6 min of quiet; the +// milestone is blocked after 12. export function wakeDue(quietMs: number, wakes: number): boolean { - return quietMs >= LOOP_QUIET_MS * 2 ** Math.min(wakes, 4) + return quietMs >= WAKE_AT_MS[Math.min(wakes, WAKE_AT_MS.length - 1)]! +} + +// Milestones built ahead of the PR that is shipping: while its CI runs, the +// session builds the next ones on its branch instead of sitting idle. +export const MAX_AHEAD = 2 +const UNFINISHED: ReadonlySet = new Set(['stacked', 'push', 'ci', 'fixing']) + +// Ship items not merged or held yet, in the order they ship. +export function unfinished(ships: readonly ShipItem[]): ShipItem[] { + return ships.filter(s => UNFINISHED.has(s.state)) +} + +// Where the next milestone starts. Nothing shipping, or a line with its own +// base (a fix goal): there. Else it is stacked on the branch of the last +// milestone still shipping, at most MAX_AHEAD deep, and never while the live +// deploy is broken (its repair goes first). undefined: not yet. +export function startBase(r: Pick, next: { base?: string }, defaultRef: string): { base: string; stacked: boolean } | undefined { + const open = unfinished(r.ships) + if (next.base !== undefined) return { base: next.base, stacked: false } + if (open.length === 0) return { base: defaultRef, stacked: false } + if ((r.deployFails ?? 0) > 0 || open.length > MAX_AHEAD) return undefined + return { base: open[open.length - 1]!.branch, stacked: true } } // Waiting on something that has its own clock is not idle: the loop building a diff --git a/auto-claude/types/index.d.ts b/auto-claude/types/index.d.ts index 7b64d0f..5f39b48 100644 --- a/auto-claude/types/index.d.ts +++ b/auto-claude/types/index.d.ts @@ -1,4 +1,6 @@ -export type ShipState = 'push' | 'ci' | 'fixing' | 'merged' | 'held' +// stacked: built on the branch of a milestone still shipping; it waits for +// that one to land (or be held), is rebased onto the default branch, then pushed. +export type ShipState = 'stacked' | 'push' | 'ci' | 'fixing' | 'merged' | 'held' export type ShipItem = { // The goal-mode queue line this milestone came from. @@ -23,6 +25,9 @@ export type ShipItem = { tried?: string[] // When a CI that would not finish was cancelled and run again. rerunAt?: number + // A stacked milestone: the commit it was started from (its parent's tip + // then), so `git rebase --onto ` drops the parent's commits. + onto?: string } export type AutoPhase = 'plan' | 'run' | 'final' | 'done' | 'stopped' @@ -60,6 +65,9 @@ export type AutoRecord = { ships: ShipItem[] // Fix goals by their queue condition, each naming the headRef it repairs. fixes: Record + // Milestones started ahead, on a branch still shipping: the commit each + // branch started from, by branch name. + onto?: Record merges: number held: number queueEmpty: boolean From 40b505bb735193be891d3e88200a7e417b165a21 Mon Sep 17 00:00:00 2001 From: unohee Date: Wed, 11 Nov 2026 21:59:50 +0900 Subject: [PATCH 2/2] auto-claude 1.1.0: repair a broken deploy before stacked milestones ship, lease-push rebased branches, ask before holding - while a deploy repair is under way nothing new is stacked and nothing stacked ships; the repair ships first and clears on merge, hold or block - a stacked branch rebased by the driver is pushed with --force-with-lease - a repeating ship error is held only after the model was asked and had 10 min (held unasked after 60 min) --- auto-claude/README.md | 4 ++-- auto-claude/hooks/auto.test.ts | 44 ++++++++++++++++++++++++++++++---- auto-claude/hooks/pure.test.ts | 10 ++++++-- auto-claude/hooks/register.ts | 29 +++++++++++++--------- auto-claude/hooks/ship.ts | 21 ++++++++++------ auto-claude/types/index.d.ts | 11 +++++++++ 6 files changed, 93 insertions(+), 26 deletions(-) diff --git a/auto-claude/README.md b/auto-claude/README.md index b083aa7..a0cb51c 100644 --- a/auto-claude/README.md +++ b/auto-claude/README.md @@ -44,7 +44,7 @@ goal-mode merged into auto-claude in 1.0.0 (see "From goal-mode" below). Its loo - Right before landing, the milestone's check runs once more in a **fresh checkout**: a detached worktree at `worktrees/auto-verify`, made from the branch head fetched from origin (github) or from the rebased commit (local). A check that passes only in the builder's worktree becomes a fix goal, like a red CI. - When CI fails or the PR conflicts, auto-claude appends a fix goal built on the milestone's branch (`:: base: goal/NN-…`), and the loop builds that next. After 3 failures the milestone is held. - **Deploy.** After each merge, as the plan says (see "Deploying"). - - **The session never rests through CI.** While a PR waits on CI, the next milestone starts at once, **stacked** on that PR's branch: the commit it starts from is recorded. Up to 2 milestones are built ahead this way. They ship in order. When the one before lands (squashed) or is held, the stacked branch runs `git rebase --onto origin/ `, which drops its parent's commits, and is then pushed like any other. A stacked branch that does not rebase cleanly gets a fix goal that does the rebase by hand. Nothing is stacked while the live deploy is broken: its repair goes first. A fix goal (`:: base:`) always starts on its own base. + - **The session never rests through CI.** While a PR waits on CI, the next milestone starts at once, **stacked** on that PR's branch: the commit it starts from is recorded. Up to 2 milestones are built ahead this way. They ship in order. When the one before lands (squashed) or is held, the stacked branch runs `git rebase --onto origin/ `, which drops its parent's commits, and is then pushed like any other. A rebased branch is pushed with `--force-with-lease`, since the session may have pushed it before. A stacked branch that does not rebase cleanly gets a fix goal that does the rebase by hand. While a broken deploy is being repaired, nothing new is stacked and nothing stacked ships: the repair, built from the default branch, lands and deploys first. A fix goal (`:: base:`) always starts on its own base. 4. **Finish or replan.** When the queue is done, auto-claude runs the final check on a fresh checkout of the merged default branch. - If it passes, auto-claude deploys once more if a deploy is still due. Then it writes a `## Result` section and stops. - If it fails, Claude appends new milestones from the check's output (round 2, and so on, up to `maxRounds`). @@ -106,7 +106,7 @@ A run never asks the person to look at `status` or to run a command. When someth | --- | --- | | A milestone is open, but the session has had no turn for 1 min: the model ended its turn to "wait", a prompt was dropped, an interrupt, or a stop the judge could not read | The milestone's prompt is sent again, after 1, 3 and 6 min of quiet. At 12 min, the milestone is blocked and the next one starts. A stop being judged, or a retry pause, is not quiet. Turns are tracked with `turn.start` / `turn.complete`. | | The model waits inside a turn | Refused while a run is on: `sleep` of 60 s or more (or a computed duration), `gh run watch`, `gh pr checks --watch`, `Monitor`, `CronCreate` and `ScheduleWakeup`. auto-claude watches CI, merges and deploys itself, and prompts the session when there is work. Shorter sleeps (a check waiting for a server) pass. | -| A shipping error to diagnose while the session builds another milestone | The diagnosis prompt waits until that milestone is done, so the stop judge never judges the wrong work. | +| A shipping error to diagnose while the session builds another milestone | The diagnosis prompt waits until that milestone is done, so the stop judge never judges the wrong work. Nothing is held before the model was asked and had 10 min to act (after an hour unasked, the hold goes ahead). | | Planning ends without `auto_plan_ready` | A reminder 1 min after that turn ended, listing what the plan files lack (15 min at most otherwise). After 3, the run stops. | | An SSH remote refuses the push (`Permission denied (publickey)`) | The push goes to the https URL of the same repository. | | The same shipping error, check after check | At 5 checks, the session model gets the error and what was tried, and fixes what it can (gh auth, remotes, `gh repo set-default`). At 10, the milestone is held and the run goes on. | diff --git a/auto-claude/hooks/auto.test.ts b/auto-claude/hooks/auto.test.ts index 05ab96b..b5f2f17 100644 --- a/auto-claude/hooks/auto.test.ts +++ b/auto-claude/hooks/auto.test.ts @@ -596,7 +596,8 @@ describe('a run, github CI', () => { await w.clock.advance(10_000) expect(w.merged).toHaveLength(2) expect(w.rebases).toContain(`git -C ${WT2} rebase --onto origin/main tip-of-${BR1}`) - expect(w.pushes.some(p => p.includes(`${BR2}:${BR2}`))).toBe(true) + // Rebased after the session may have pushed it: pushed with a lease. + expect(w.pushes.find(p => p.includes(`${BR2}:${BR2}`))).toContain('--force-with-lease') for (let i = 0; i < 3; i++) await w.clock.advance(TICK) expect(w.notifies.at(-1)).toContain('finished: the final check passed after 1 round(s). 2 merged, 0 held') }) @@ -621,16 +622,51 @@ describe('a run, github CI', () => { expect(w.adds.at(-1)).toMatch(new RegExp(`${BR2}$`)) }) - test('nothing is stacked while the deploy is broken: its repair goes first', { timeoutMs: 30_000 }, async ($, on) => { + test('a broken deploy is repaired before a stacked milestone merges on top of it', { timeoutMs: 30_000 }, async ($, on) => { const w = world(on) w.github = true - w.deployExit = 1 + w.checks = '[{"bucket":"pending"}]' await planned($, w, PLAN_DEPLOY) await meet($, w) - // Milestone 1 landed and its deploy failed: the repair line, from main, comes before milestone 2. + // Milestone 2 is built and stacked while milestone 1 waits on CI. + await meet($, w) + w.deployExit = 1 + w.checks = '[{"bucket":"pass"}]' + await w.clock.advance(10_000) + // Milestone 1 landed, its deploy failed: milestone 2 stays stacked, the repair is built from main. expect(w.merged).toHaveLength(1) + expect(deployRuns(w).filter(r => r.includes('deploy.sh'))).toHaveLength(1) + expect((await cmd($, 'status')).text).toContain('- stacked: sub() returns a-b') expect(w.adds.at(-1)).toMatch(/origin\/main$/) expect(w.prompts.at(-1)).toMatch(/deploy/i) + // The repair lands first and deploys; only then does milestone 2 go. + w.deployExit = 0 + await meet($, w) + expect(w.merged).toHaveLength(3) + const order = w.pushes.filter(p => p.includes(' push -u ')).map(p => p.split(' ').find(t => t.includes(':goal/'))?.split(':')[1]) + expect(order.indexOf(BR2)).toBeGreaterThan(order.findIndex(b => b?.startsWith('goal/03-'))) + expect((await cmd($, 'status')).text).toContain('3 merged, 0 held') + }) + + test('a shipping error is not held before the model was asked: the question waits while it builds another milestone', { timeoutMs: 30_000 }, async ($, on) => { + const w = world(on) + w.github = true + w.gitGone = true + await planned($, w) + await meet($, w) + // Milestone 2 is built for 20 min while milestone 1's push keeps failing. + await turnStart($) + for (let i = 0; i < 40; i++) await w.clock.advance(TICK) + expect(w.notifies.some(n => n.includes('held: add() returns a+b'))).toBe(false) + expect(w.prompts.filter(p => p.includes('[auto-claude] Shipping is stuck'))).toHaveLength(0) + await meet($, w) + await w.clock.advance(TICK) + expect(w.prompts.filter(p => p.includes('[auto-claude] Shipping is stuck'))).toHaveLength(1) + // It gets 10 min to act before the hold. + for (let i = 0; i < 16; i++) await w.clock.advance(TICK) + expect(w.notifies.some(n => n.includes('held: add() returns a+b'))).toBe(false) + for (let i = 0; i < 6; i++) await w.clock.advance(TICK) + expect(w.notifies.some(n => n.includes('held: add() returns a+b'))).toBe(true) }) test('only required checks decide: a red check that is not required does not block the merge (P1)', async ($, on) => { diff --git a/auto-claude/hooks/pure.test.ts b/auto-claude/hooks/pure.test.ts index a21ebd2..8df5ea9 100644 --- a/auto-claude/hooks/pure.test.ts +++ b/auto-claude/hooks/pure.test.ts @@ -175,6 +175,11 @@ describe('ship', () => { // Ten fast CI looks are under two minutes: a short outage is not held. expect(shipRemedy({ note: 'x', repeats: 10, tried: ['diagnose'], noteSince: 0 }, 2 * MIN)).toBeUndefined() expect(shipRemedy({ note: 'x', repeats: 10, tried: ['diagnose'], noteSince: 0 }, 15 * MIN)).toBe('hold') + // Never held before the model was asked (it was busy building): asked first, then given 10 min; held unasked only after an hour. + expect(shipRemedy({ note: 'x', repeats: 40, noteSince: 0 }, 30 * MIN)).toBe('diagnose') + expect(shipRemedy({ note: 'x', repeats: 40, noteSince: 0, askedAt: 30 * MIN }, 35 * MIN)).toBeUndefined() + expect(shipRemedy({ note: 'x', repeats: 40, noteSince: 0, askedAt: 30 * MIN }, 40 * MIN)).toBe('hold') + expect(shipRemedy({ note: 'x', repeats: 120, noteSince: 0 }, 60 * MIN)).toBe('hold') expect(shipRemedy({ repeats: 30 }, 0)).toBeUndefined() // The runs the driver cancelled to re-run them are not a red CI. expect(onlyCancelled('[{"bucket":"cancel"},{"bucket":"pass"}]')).toBe(true) @@ -222,8 +227,9 @@ describe('ship', () => { expect(startBase({ ships: [ship('a', 'ci'), ship('b', 'stacked')] }, {}, 'origin/main')).toEqual({ base: 'b', stacked: true }) expect(MAX_AHEAD).toBe(2) expect(startBase({ ships: [ship('a', 'ci'), ship('b', 'stacked'), ship('c', 'stacked')] }, {}, 'origin/main')).toBeUndefined() - // A broken deploy: its repair goes first, nothing piles on. - expect(startBase({ ships: [ship('a', 'ci')], deployFails: 1 }, {}, 'origin/main')).toBeUndefined() + // A deploy under repair: the repair goes first, nothing piles on. + expect(startBase({ ships: [ship('a', 'stacked')], repair: 'repair it' }, {}, 'origin/main')).toBeUndefined() + expect(startBase({ ships: [ship('a', 'stacked')], repair: 'repair it' }, { base: 'origin/main' }, 'origin/main')).toEqual({ base: 'origin/main', stacked: false }) expect(unfinished([ship('a', 'merged'), ship('b', 'fixing'), ship('c', 'stacked')]).map(s => s.headRef)).toEqual(['b', 'c']) expect(inFlight([ship('c', 'stacked')])).toBeUndefined() }) diff --git a/auto-claude/hooks/register.ts b/auto-claude/hooks/register.ts index ef7dfe9..e009acd 100644 --- a/auto-claude/hooks/register.ts +++ b/auto-claude/hooks/register.ts @@ -193,7 +193,8 @@ async function hold($: EngineInterface, r: AutoRecord, item: ShipItem, note: str const cause = causeOf(note) const streak = r.holdCause === cause ? (r.holdStreak ?? 0) + 1 : 1 await tell($, r, `held: ${item.goal.slice(0, 60)}${item.pr !== undefined ? ` (PR #${item.pr})` : ''}: ${note.slice(0, 160)}; going on with the next milestone`) - return { r: { ...r, held: r.held + 1, holdCause: cause, holdStreak: streak }, item: { ...item, state: 'held', note } } + // A held repair frees the stacked milestones; their deploys count toward the stop. + return { r: { ...r, held: r.held + 1, holdCause: cause, holdStreak: streak, ...(item.repair === true ? { repair: undefined } : {}) }, item: { ...item, state: 'held', note } } } // GitHub's runs of a PR's branch, cancelled and run again once when its CI @@ -241,7 +242,7 @@ async function advance($: EngineInterface, r: AutoRecord, item: ShipItem): Promi if (r.ciMode === 'local') return { r, item: update({ state: 'ci' }) } // An SSH remote that refused (no key in an unattended shell) is pushed over https instead. const target = (item.tried ?? []).includes('https-push') && slug !== undefined ? httpsOf(slug) : r.remote - const pushed = await run($, ['git', '-C', wt, 'push', '-u', target, `${item.branch}:${item.headRef}`, ...(item.branch !== item.headRef ? ['--force-with-lease'] : [])], wt) + const pushed = await run($, ['git', '-C', wt, 'push', '-u', target, `${item.branch}:${item.headRef}`, ...(item.branch !== item.headRef || item.rebased === true ? ['--force-with-lease'] : [])], wt) if (pushed.exitCode !== 0) return { r, item: update({ note: `push failed: ${out(pushed)}` }) } if (item.pr !== undefined) return { r, item: update({ state: 'ci', since: now, note: undefined, rerunAt: undefined }) } const made = await run($, ['gh', 'pr', 'create', ...repo, '--head', item.headRef, '--base', r.defaultBranch, '--title', item.goal.slice(0, 120), '--body', `Milestone of an auto-claude run.\n\nCheck: \`${item.check ?? '-'}\`\n\nPlan: .claude/auto-plan.md (local to the run).`], wt) @@ -323,7 +324,7 @@ async function advance($: EngineInterface, r: AutoRecord, item: ShipItem): Promi await run($, ['git', '-C', r.root, 'worktree', 'remove', path], r.root) } const landed = item.pr !== undefined ? `PR #${item.pr}` : `into ${r.defaultBranch}` - let next: AutoRecord = { ...r, merges: r.merges + 1, holdStreak: 0, holdCause: undefined } + let next: AutoRecord = { ...r, merges: r.merges + 1, holdStreak: 0, holdCause: undefined, ...(item.repair === true ? { repair: undefined } : {}) } await tell($, next, `merged ${landed}: ${item.goal.slice(0, 80)}`) const merged = update({ state: 'merged', note: undefined }) if (r.deploy !== undefined) next = await deployAfter($, next, r.deploy, landed) @@ -361,9 +362,10 @@ async function deployFailed($: EngineInterface, r: AutoRecord, spec: DeploySpec, const fails = (r.deployFails ?? 0) + 1 const next: AutoRecord = { ...r, deployFails: fails, deployDue: true } if (fails >= MAX_DEPLOY_FAILS) return stopRun($, next, `${why}; ${fails} deploys failed in a row, so nothing more is merged on top of what is live`) - await appendQueue($, r.root, deployFixLine(spec.command, spec.smoke, why, refOf(r))) + const line = deployFixLine(spec.command, spec.smoke, why, refOf(r)) + await appendQueue($, r.root, line) await tell($, next, `${why}; a milestone that repairs the deploy goes first (deploy failure ${fails}/${MAX_DEPLOY_FAILS})`) - return next + return { ...next, repair: parseQueue(line)[0]!.condition } } async function finalCheck($: EngineInterface, r: AutoRecord, caps: Caps): Promise { @@ -450,8 +452,9 @@ async function shipSteps($: EngineInterface, r: AutoRecord): Promise for (let steps = 0; steps < MAX_STEPS && r.on; steps++) { let item = inFlight(r.ships) if (item === undefined) { - // The milestone ahead landed or was held: the first stacked one goes next. - const waiting = r.ships.find(s => s.state === 'stacked') + // The milestone ahead landed or was held: the first stacked one goes next, + // unless the deploy is being repaired (nothing merges on top of it). + const waiting = r.repair === undefined ? r.ships.find(s => s.state === 'stacked') : undefined if (waiting === undefined) break const moved = await unstack($, r, waiting) r = { ...moved.r, ships: moved.r.ships.map(s => (s.headRef === waiting.headRef ? moved.item : s)) } @@ -486,7 +489,7 @@ async function unstack($: EngineInterface, r: AutoRecord, item: ShipItem): Promi const rebased = await run($, ['git', '-C', wt, ...args], wt) const now = await $.clock.now() const { onto: _done, ...rest } = item - if (rebased.exitCode === 0) return { r, item: { ...rest, state: 'push', since: now } } + if (rebased.exitCode === 0) return { r, item: { ...rest, state: 'push', since: now, rebased: true } } await run($, ['git', '-C', wt, 'rebase', '--abort'], wt) const how = onto !== undefined ? `git rebase --onto ${refOf(r)} ${onto}` : `git rebase ${refOf(r)}` const failure = `it was built on a milestone that has since landed or been held, and does not rebase onto ${r.defaultBranch}; run ${how} and resolve the conflicts: ${out(rebased).slice(0, 120)}` @@ -503,7 +506,7 @@ async function treat($: EngineInterface, r: AutoRecord, item: ShipItem, now: num return { r, item: { ...item, tried: [...(item.tried ?? []), 'https-push'] } } } if (remedy === 'diagnose') { - const asked = { ...item, tried: [...(item.tried ?? []), 'diagnose'] } + const asked = { ...item, tried: [...(item.tried ?? []), 'diagnose'], askedAt: now } return { r, item: (await askModel($, r, asked)) ? asked : item } } if (remedy === 'hold') return hold($, r, item, `stuck at ${item.state} for ${item.repeats} checks: ${item.note ?? ''}`) @@ -610,6 +613,8 @@ async function readEvents($: EngineInterface, r: AutoRecord): Promise (s.headRef === parent ? { ...s, state: 'held', note: 'its fix goal was blocked' } : s)) } await tell($, r, `${e.event}: ${(e.goal ?? '').slice(0, 80)}${e.reason !== undefined ? ` (${e.reason.slice(0, 100)})` : ''}; going on`) @@ -631,8 +636,10 @@ async function metToShip($: EngineInterface, r: AutoRecord, goal: string, branch // waits its turn, and is rebased onto the default branch before its push. const onto: string | undefined = r.onto?.[branch] const { [branch]: _taken, ...rest } = r.onto ?? {} - const state = onto !== undefined || unfinished(r.ships).length > 0 ? 'stacked' : 'push' - return { ...r, onto: rest, ships: [...r.ships, { goal, headRef: branch, branch, worktree: wt, check, state, red: 0, since: await $.clock.now(), ...(onto !== undefined ? { onto } : {}) }] } + // A deploy repair, built on the default branch, ships ahead of them all. + const repair = goal === r.repair + const state = !repair && (onto !== undefined || unfinished(r.ships).length > 0) ? 'stacked' : 'push' + return { ...r, onto: rest, ships: [...r.ships, { goal, headRef: branch, branch, worktree: wt, check, state, red: 0, since: await $.clock.now(), ...(onto !== undefined ? { onto } : {}), ...(repair ? { repair: true } : {}) }] } } // A shipping error that keeps coming back goes to the session model with what diff --git a/auto-claude/hooks/ship.ts b/auto-claude/hooks/ship.ts index 899f8ca..cfc9f6c 100644 --- a/auto-claude/hooks/ship.ts +++ b/auto-claude/hooks/ship.ts @@ -242,19 +242,26 @@ export function ciAction(item: Pick, now: number): 'https-push' | 'diagnose' | 'hold' | undefined { +export function shipRemedy(item: Pick, now: number): 'https-push' | 'diagnose' | 'hold' | undefined { if (item.note === undefined) return undefined const tried = item.tried ?? [] const repeats = item.repeats ?? 0 if (/Permission denied \(publickey\)|Host key verification failed|Could not read from remote repository/i.test(item.note) && !tried.includes('https-push')) return 'https-push' - if (repeats >= HOLD_AT && now - (item.noteSince ?? now) >= HOLD_MIN_MS) return 'hold' - if (repeats >= DIAGNOSE_AT && !tried.includes('diagnose')) return 'diagnose' - return undefined + if (repeats < DIAGNOSE_AT) return undefined + const since = item.noteSince ?? now + // A record from before askedAt: the diagnosis went out with its note. + const asked = item.askedAt ?? (tried.includes('diagnose') ? since : undefined) + if (asked === undefined) return now - since >= UNASKED_HOLD_MS ? 'hold' : 'diagnose' + return repeats >= HOLD_AT && now - since >= HOLD_MIN_MS && now - asked >= ASKED_WAIT_MS ? 'hold' : undefined } // The class of a hold's cause: two milestones in a row held for the same class @@ -305,11 +312,11 @@ export function unfinished(ships: readonly ShipItem[]): ShipItem[] { // base (a fix goal): there. Else it is stacked on the branch of the last // milestone still shipping, at most MAX_AHEAD deep, and never while the live // deploy is broken (its repair goes first). undefined: not yet. -export function startBase(r: Pick, next: { base?: string }, defaultRef: string): { base: string; stacked: boolean } | undefined { +export function startBase(r: Pick, next: { base?: string }, defaultRef: string): { base: string; stacked: boolean } | undefined { const open = unfinished(r.ships) if (next.base !== undefined) return { base: next.base, stacked: false } if (open.length === 0) return { base: defaultRef, stacked: false } - if ((r.deployFails ?? 0) > 0 || open.length > MAX_AHEAD) return undefined + if (r.repair !== undefined || open.length > MAX_AHEAD) return undefined return { base: open[open.length - 1]!.branch, stacked: true } } diff --git a/auto-claude/types/index.d.ts b/auto-claude/types/index.d.ts index 5f39b48..4023c76 100644 --- a/auto-claude/types/index.d.ts +++ b/auto-claude/types/index.d.ts @@ -28,6 +28,13 @@ export type ShipItem = { // A stacked milestone: the commit it was started from (its parent's tip // then), so `git rebase --onto ` drops the parent's commits. onto?: string + // Rebased by the driver after it was built: pushed with --force-with-lease + // (the session may have pushed the branch before the rebase). + rebased?: boolean + // When the session model was given this item's repeating error (shipRemedy). + askedAt?: number + // The repair of a broken deploy: it ships ahead of every stacked milestone. + repair?: boolean } export type AutoPhase = 'plan' | 'run' | 'final' | 'done' | 'stopped' @@ -82,6 +89,10 @@ export type AutoRecord = { // A merge happened whose deploy has not passed yet: skipped (the command is // not in the default branch yet) or failed; the run is not done until it has. deployDue?: boolean + // The queue condition of the deploy repair under way: until it lands (or is + // blocked), stacked milestones are neither started nor shipped, so no merge + // deploys on top of what is broken. + repair?: string // Why milestones were last held, as a cause class (ship.ts causeOf), and how // many in a row share it: two in a row stop the run (a cause the whole run shares). holdCause?: string