Split agent instructions into a primer and topic guides

AGENTS.md

@@ -157,6 +157,16 @@ every open source contributor run or host a server for a reviewer.
 - `db/seeds.rb` must be **idempotent** — use `find_or_create_by!` or guard with count checks
 - Seeds should provide enough data to demo features from a fresh checkout
 
+## Agent Instructions Stay Current
+
+Agents learn CoPlan only from the served instructions, not from this repo. A change that agents can observe is not done until those instructions match it. That includes API endpoints, fields, status codes and error codes, as well as comment, editing, versioning and attribution rules, Markdown features, the live-session protocol, and `engine/agent_tools/`.
+
+- The primer is `engine/app/views/coplan/agent_instructions/show.text.erb` (served at `/agent-instructions`). Keep it to what every agent needs.
+- Topic guides are in `engine/app/views/coplan/agent_instructions/guides/` (served at `/agent-instructions/<guide>`). Put task-specific detail there. A new guide needs an entry in `AgentInstructionsController::GUIDES` and a link from the primer; `spec/requests/agent_instructions_spec.rb` checks both.
+- Update `guides/api.text.erb` when you add, remove, or change an endpoint.
+- Remove anything that stops being true. Wrong instructions are worse than missing ones.
+- Write in plain, short sentences, like the existing guides. Agents follow the commands literally, so run any new command against a local server before you document it.
+
 ## Code Review
 
 - Run the `code-review` skill before considering a session complete

docs/AGENT_COLLABORATION.md

@@ -52,7 +52,7 @@ The honest ceiling: this is thread-per-agent. It's fine for a team, not
 for hundreds of concurrent attached agents — that would want a real
 pub/sub transport rather than held Rack threads.
 
-Full API reference: `GET /agent-instructions` → "Live Collaboration".
+Protocol reference: `GET /agent-instructions/live`, with per-harness guides under `/agent-instructions/live/`.
 
 ## Waiting while an agent turn is active
 
@@ -77,7 +77,7 @@ The unbounded streaming form belongs under an external supervisor that owns
 its lifecycle and has a real way to re-enter the model.
 
 The served tool is a thin convenience over the API. An agent that can make
-HTTP requests can follow `/agent-instructions` directly and does not need it.
+HTTP requests can follow `/agent-instructions/live` directly and does not need it.
 
 ## What the harness must provide
 
@@ -212,7 +212,7 @@ posture deliberately — the bridge never escalates beyond what the
 config says.
 
 An agent doesn't need the bridge to drain events during its current turn: it
-can follow the "Live Collaboration" protocol in `/agent-instructions`
+can follow the protocol in `/agent-instructions/live`
 directly. That becomes a durable attachment only when the harness can turn
 the wait's completion into another model turn. A foreground loop that merely
 blocks, or a background loop that only writes a notification, is not a wake

engine/agent_tools/coplan-bridge

@@ -46,6 +46,7 @@ require "uri"
 require "optparse"
 require "open3"
 require "shellwords"
+require "set"
 
 $stdout.sync = true
 
@@ -145,6 +146,12 @@ def request(method, path, body: nil, params: nil, read_timeout: 90)
   req.body = body.to_json if body
   Net::HTTP.start(uri.hostname, uri.port, use_ssl: uri.scheme == "https", read_timeout: read_timeout) do |http|
     res = http.request(req)
+    # A rejected token never recovers on retry, and every call after it
+    # would fail the same way. Stop so a supervisor (or a person) notices,
+    # instead of polling the server in a tight loop.
+    if res.code == "401"
+      abort "coplan-bridge: #{method.upcase} #{path} -> 401. The token is invalid, expired, or revoked. Mint a new token and restart the bridge."
+    end
     warn "coplan-bridge: #{method.upcase} #{path} -> #{res.code}" unless res.code.start_with?("2")
     res.body && !res.body.empty? ? JSON.parse(res.body) : {}
   end
@@ -168,7 +175,8 @@ def wake_prompt(event)
     #{payload["comment_author"] ? "#{payload["comment_author"]} wrote: #{payload["comment_body"].to_s.strip}" : ""}
     #{payload["anchor_text"] ? "Anchored to: \"#{payload["anchor_text"]}\"" : ""}
 
-    Respond as the plan's agent, over the CoPlan API at #{BASE} (full instructions: #{BASE}/agent-instructions):
+    Respond as the plan's agent, over the CoPlan API at #{BASE} (full instructions: #{BASE}/agent-instructions).
+    Authenticate every call with `Authorization: Bearer $TOKEN`: $TOKEN in your environment is this bridge's session token.
     1. PATCH /api/v1/plans/#{event["plan_id"]}/agent_session {"state":"active","detail":"<what you're doing>"} immediately.
     2. Reply on the thread first (narrate what you'll change), then apply any edit with PUT .../content.
     3. Finish with {"state":"complete"} (or "awaiting_input" if you asked a question).
@@ -237,7 +245,9 @@ class AcpAgent
 
   def boot
     # stderr passes through so the agent's own logging stays visible.
-    @stdin, @stdout, @wait_thread = Open3.popen2(*@command)
+    # Same token handoff as the exec adapters: the prompt tells the agent
+    # to authenticate with $TOKEN, so it has to be this bridge's token.
+    @stdin, @stdout, @wait_thread = Open3.popen2({ "TOKEN" => TOKEN, "COPLAN_TOKEN" => TOKEN }, *@command)
     rpc("initialize", {
       protocolVersion: PROTOCOL_VERSION,
       # No fs or terminal capabilities: the agent works through its own
@@ -397,32 +407,75 @@ def dispatch(event)
   # Honest pill before the harness even boots.
   Bridge.patch_state(event["plan_id"], "active", "reading your comment")
 
+  # A failed turn may still have replied or edited before it exited, and
+  # redelivery hands the same event to the agent again. Say so, the way
+  # the ACP restart path does, so the retry checks before acting.
+  prompt = wake_prompt(event)
+  if ATTEMPTED.include?(event["id"])
+    prompt = "NOTE: an earlier attempt at this event failed and may have partially completed. " \
+             "Check the plan's thread and content first, and do not duplicate replies or edits already made.\n\n#{prompt}"
+  end
+  ATTEMPTED << event["id"]
+
   cmd = template.map do |part|
-    part.gsub("{session}", CONFIG.fetch("harness_session", "coplan")).gsub("{prompt}", wake_prompt(event))
+    part.gsub("{session}", CONFIG.fetch("harness_session", "coplan")).gsub("{prompt}", prompt)
   end
   puts "coplan-bridge: dispatching #{event["type"]} -> #{adapter}"
-  system(*cmd)
+  # system returns false on a nonzero exit and nil when the command can't
+  # start; neither raises. Raise here so the loop leaves the event unacked
+  # for redelivery instead of acking a turn that never happened.
+  # The agent acts with the bridge's session token, under the name the
+  # instructions use for it. Its replies and edits are then attributed to
+  # that token, and never echoed back to this bridge's inbox.
+  return if system({ "TOKEN" => TOKEN, "COPLAN_TOKEN" => TOKEN }, *cmd)
+
+  raise "#{adapter} adapter #{$?&.exitstatus ? "exited #{$?.exitstatus}" : "could not start"}"
 end
 
+# Event ids handed to an exec adapter but not yet acked, so a redelivered
+# event can be marked as a retry. An id leaves the set when its ack
+# succeeds, so the set holds only events still in flight.
+ATTEMPTED = Set.new
+
 claim_sessions
 cursor = nil
+failures = 0
 puts "coplan-bridge: draining events from #{BASE}"
 loop do
   resp = request(:get, "/api/v1/agent/events", params: { wait: 25, cursor: cursor }.compact)
-  events = resp["events"] || []
+  # A good poll always carries `events`. Anything else (a 5xx, an HTML
+  # error page) returned immediately, so wait before asking again. So did
+  # a throttled long-poll: the server had no thread to hold for us.
+  unless resp.key?("events")
+    sleep 5
+    next
+  end
+  sleep 3 if resp["throttled"]
+  events = resp["events"]
   events.each do |event|
     dispatch(event)
     # Ack each event only once its dispatch came back — at-least-once
     # means a harness failure leaves the event unacked for redelivery,
     # never swallowed by a batch-level ack.
     cursor = event["id"]
     request(:post, "/api/v1/agent/events/ack", body: { cursor: cursor })
+    ATTEMPTED.delete(event["id"])
+    failures = 0
   rescue => e
     # Stop the batch here: acking later events would also ack this one
     # (the cursor is a high-water mark). The next poll redelivers
-    # everything unacked; the sleep keeps a wedged adapter from spinning.
-    warn "coplan-bridge: dispatch failed: #{e.class}: #{e.message} — leaving event unacked"
-    sleep 3
+    # everything unacked. Dispatch already said `active`; take that back,
+    # or every retry would refresh a pill promising a turn that isn't
+    # running. Back off so a wedged adapter doesn't spin.
+    failures += 1
+    delay = [ 3 * 2**(failures - 1), 60 ].min
+    warn "coplan-bridge: dispatch failed: #{e.class}: #{e.message} — leaving event unacked, retrying in #{delay}s"
+    begin
+      Bridge.patch_state(event["plan_id"], "complete", nil)
+    rescue StandardError
+      nil # the server may be the thing that failed; the retry will tell
+    end
+    sleep delay
     break
   end
 rescue Interrupt

engine/app/controllers/coplan/agent_instructions_controller.rb

@@ -1,49 +1,80 @@
 module CoPlan
-  # Serves the agent API instructions at /agent-instructions.
+  # Serves the agent instructions: a short primer at /agent-instructions,
+  # plus topic guides at /agent-instructions/<guide> that agents fetch only
+  # when a task needs them. The split keeps every agent's first read small —
+  # most never need the live-session protocol or the slide-deck rules.
   #
-  # This endpoint has two audiences:
+  # Two audiences:
   #
-  # * **Agents and CLIs** (curl, HTTP libraries, coding agents) fetch it as raw
-  #   Markdown. Every API response points here via the `X-Agent-Instructions`
-  #   header, so the raw behavior is load-bearing: any client that does not
-  #   explicitly ask for HTML continues to get `text/markdown`.
-  # * **Humans in a browser** go to the short setup page. They should not have
-  #   to read or copy the full API reference themselves.
+  # * **Agents and CLIs** (curl, HTTP libraries, coding agents) fetch raw
+  #   Markdown. Every API response points at the primer via the
+  #   `X-Agent-Instructions` header, so the raw behavior is load-bearing: any
+  #   client that does not explicitly ask for HTML gets `text/markdown`.
+  # * **Humans in a browser** who land on the primer go to the short setup
+  #   page; they should not have to read the API reference themselves. The
+  #   rendered primer lives at /_/agent/instructions, and guides render as
+  #   HTML in place so links from it stay browsable.
   #
   # Negotiation is deliberately conservative: we only serve HTML when the
   # Accept header *leads* with `text/html`, which is exactly what every
   # browser sends (`Accept: text/html,application/xhtml+xml,…`) and what
   # Turbo Drive sends on navigation (`Accept: text/html, application/xhtml+xml`).
   # curl's default `Accept: */*`, an absent Accept header, or
   # `Accept: text/markdown` all fall through to raw Markdown. An explicit
-  # format always wins over the Accept header: `/agent-instructions.md`
-  # forces raw Markdown, while `.html` goes to setup even from curl.
+  # format always wins over the Accept header: `.md` forces raw Markdown,
+  # and `.html` forces HTML (the setup page, for the primer).
   class AgentInstructionsController < ApplicationController
     skip_before_action :authenticate_coplan_user!
+    before_action :prepare_instructions
+
+    # URL segment => template under agent_instructions/guides/. The primer
+    # links every entry; a spec holds it to that.
+    GUIDES = {
+      "markdown" => "markdown",
+      "presentations" => "presentations",
+      "creating" => "creating",
+      "editing" => "editing",
+      "comments" => "comments",
+      "live" => "live",
+      "live/claude-code" => "live_claude_code",
+      "live/amp" => "live_amp",
+      "live/codex" => "live_codex",
+      "live/hosted" => "live_hosted",
+      "live/bridge" => "live_bridge",
+      "organizing" => "organizing",
+      "api" => "api"
+    }.freeze
 
     def show
       return redirect_to(agent_setup_path) if prefers_html?
 
-      prepare_instructions
       render layout: false, content_type: "text/markdown", formats: [ :text ]
     end
 
+    # The primer, rendered for a person who wants to see what their agent reads.
     def reference
-      prepare_instructions
-      @current_coplan_user = CoPlan::Authentication.user_from_request(request)
-      CoPlan::Current.user = current_user
+      resolve_viewer
       @instructions_markdown = render_to_string(:show, formats: [ :text ], layout: false)
       render :reference
     end
 
-    # Sub-instructions: the library-organizing guide, linked from the main
-    # doc and from library API responses. Markdown-only — agents fetch it
-    # when an organizing task needs it.
-    def organizing
-      @auth_instructions = CoPlan.configuration.agent_auth_instructions
-      @curl = CoPlan.configuration.agent_curl_prefix
-      @base = "#{request.base_url}#{root_path.chomp("/")}"
-      render layout: false, content_type: "text/markdown", formats: [ :text ]
+    def guide
+      template = GUIDES[params[:guide].to_s]
+      unless template
+        return render plain: "# Unknown guide\n\nThe guide list is at #{@base}/agent-instructions\n",
+          status: :not_found, content_type: "text/markdown"
+      end
+
+      template = "coplan/agent_instructions/guides/#{template}"
+      if prefers_html?
+        resolve_viewer
+        @instructions_url = coplan.agent_instructions_guide_url(guide: params[:guide])
+        @raw_path = coplan.agent_instructions_guide_path(guide: params[:guide], format: :md)
+        @instructions_markdown = render_to_string(template, formats: [ :text ], layout: false)
+        render :guide, formats: [ :html ]
+      else
+        render template, layout: false, content_type: "text/markdown", formats: [ :text ]
+      end
     end
 
     private
@@ -63,6 +94,14 @@ def prepare_instructions
       @create_example_json = create_example_json
     end
 
+    # The pages are public, but signed-in visitors should still see their
+    # normal nav chrome (search, inbox, sign-out) in the shared layout —
+    # same optional-resolve approach as WelcomeController.
+    def resolve_viewer
+      @current_coplan_user = CoPlan::Authentication.user_from_request(request)
+      CoPlan::Current.user = current_user
+    end
+
     # The Create Plan curl example, with a real configured plan type so
     # agents copy an instance-accurate command. Names are admin-controlled
     # free text, so the payload is JSON-serialized (never hand-interpolated)

engine/app/views/coplan/agent_instructions/guide.html.erb

@@ -0,0 +1,17 @@
+<% title = @instructions_markdown[/\A# (.+)$/, 1] || "Agent guide" %>
+<% content_for :title, "#{title} — CoPlan" %>
+
+<div class="agent-reference">
+  <%= link_to "← Agent instructions", agent_instructions_reference_path, class: "agent-reference__back" %>
+  <header class="agent-reference__header">
+    <p>
+      A topic guide that your agent fetches when a task needs it. Agents get this URL as raw Markdown;
+      you can also <%= link_to "view the raw markdown", @raw_path %>.
+    </p>
+    <%= render "coplan/shared/copy_url", url: @instructions_url, button_class: "btn btn--secondary" %>
+  </header>
+
+  <article class="agent-reference__doc">
+    <%= render_markdown(@instructions_markdown, interactive: false) %>
+  </article>
+</div>