feat: measure provider startup before adding prepared sessions

CHANGELOG.md

@@ -24,6 +24,13 @@ Versioning and Keep a Changelog conventions.
 
 ### Added
 
+- Contain observer panic-payload cleanup failures and disable failed observers
+  across runtime clones. Report event-delivery wait separately from observed
+  first-text latency, preserving bounded backpressure.
+- Opt-in payload-free startup timing observers for validation, concurrency admission,
+  sandbox preparation, process spawn, first output/text and terminal outcomes.
+  Existing process lifetime and wire events are unchanged.
+
 - Bidirectional OpenCode support through `opencode serve`. `OpenCodeTurnMode`
   selects the transport; `OpenCode::serve()` starts the server on a reserved
   loopback port and drives it over HTTP and Server-Sent Events, adding live

Cargo.toml

@@ -127,3 +127,7 @@ needless_pass_by_value = "allow"
 return_self_not_must_use = "allow"
 struct_excessive_bools = "allow"
 too_many_lines = "allow"
+
+[[example]]
+name = "startup_timings"
+required-features = ["claude", "codex"]

README.md

@@ -255,3 +255,5 @@ to separate local verification from publishing and hosting decisions.
 
 Licensed under either the Apache License, Version 2.0 or the MIT License, at
 your option.
+
+- [Measure session startup](docs/how-to/measure-session-startup.md)

docs/adr/0004-prepared-provider-processes.md

@@ -0,0 +1,110 @@
+# ADR 0004: Prepared provider processes
+
+- Status: Proposed
+- Date: 2026-09-23
+
+## Problem
+
+`InProcessRuntimeClient` retains logical session identity, not the provider process.
+The built-in executor reports `retained_process: false` and calls `AgentRuntime::run`
+for each invocation. Both Claude and Codex therefore pay process startup again on
+subsequent turns. Calling `acquire` earlier cannot remove this cost.
+
+The first implementation step is stage timing so an embedder can distinguish
+validation, concurrency wait, sandbox setup, process spawn, protocol output and
+first assistant text. A first output frame is not evidence of provider readiness
+or model request submission. Provider-specific readiness requires an explicit
+handshake acknowledgement, not a sleep or an empty model turn.
+
+## Proposed ownership
+
+The SDK owns a bounded process supervisor and provider protocol state. Fleet owns
+when a user has selected enough configuration to prepare, durable conversation
+records, authorization, feature rollout, and presentation. Listing projects or
+conversations must never spawn provider processes.
+
+Existing `AgentRuntime::run` and retained-client constructors preserve lazy,
+one-process-per-turn behavior. A separate opt-in client configuration enables
+retained processes. Unsupported adapters and remote protocol versions report a
+typed capability error rather than silently claiming preparation succeeded.
+
+## Proposed lifecycle
+
+1. Acquire a logical runtime with project, provider, sandbox and launch settings.
+2. Explicitly prepare it, without a prompt or invocation identifier. This starts
+   the process and completes the provider handshake; it does not call a model,
+   create a synthetic transcript message, or grant tool execution.
+3. Send a turn. Lazy sending performs preparation automatically. Sending during
+   preparation joins the same bounded operation and submits exactly once.
+4. On a successful turn, keep the provider connection and continue draining its
+   bounded event stream. Idle tool/background events belong to the runtime and
+   must not be attached to the next invocation.
+5. Dispose or expire the idle process, confirming process-tree teardown. Preserve
+   session identity so later work can explicitly resume from provider persistence.
+
+The process lifecycle distinguishes unprepared, queued, preparing, ready, busy,
+failed, stopping and stopped. A logical runtime being acquired is not provider
+readiness. Preparation errors expose delivery=not_sent. Failure after submission
+preserves the existing accepted/possibly_sent semantics and must not replay a
+prompt automatically.
+
+Preparation has a configurable deadline, bounded concurrency, global process
+capacity and idle expiration. Active turns cannot be evicted to admit speculative
+preparation. Abandoned preparations release their capacity. Disposing while
+preparing prevents late successful readiness from resurrecting the runtime.
+
+## Configuration and credentials
+
+Use one launch-configuration representation for preparation and sending. Fleet
+currently provides MCP definitions, capability credentials, system instructions,
+allowed tools and tailnet environment at turn time. Preparing without these and
+then spawning again on send would provide no benefit and could misconfigure the
+sandbox. Fleet must resolve that context before requesting preparation.
+
+Never share prepared processes across users, projects, sandboxes or credential
+contexts. Working-directory, environment, MCP, sandbox revision and credential
+changes require disposal/repreparation unless the driver has a tested live-update
+operation. Compare full launch configuration without logging secret values.
+Runtime and per-turn settings must have explicit precedence.
+
+Fleet's MCP capability lifetime must be reconciled with idle process retention:
+retaining a process must neither keep a revoked turn token valid nor silently
+broaden it to an unbounded credential. Use scoped runtime credentials with explicit
+revocation, or safely replace the child when credentials rotate. This contract is
+a prerequisite to Fleet enabling retention.
+
+## Provider implementation
+
+Claude keeps a streaming input channel open and separates initialization from user
+messages. Codex keeps one app-server connection and thread alive, issues one
+initialize handshake per process, and starts later turns on that connection. Both
+need bounded background draining, crash detection, cancellation, late-event
+isolation and cleanup. One-shot Codex exec and unsupported providers remain lazy.
+
+## Fleet adoption
+
+1. Upgrade to the verified Windows baseline.
+2. Collect startup stage timings without changing process behavior.
+3. Add and verify retained-process drivers and preparation in the SDK.
+4. Add a feature-gated Fleet preparation endpoint using the generated API client.
+5. Trigger preparation only after explicit project/provider selection; display
+   queued/running/ready/failed feedback while the composer stays usable.
+6. Measure cold/prepared first-turn and subsequent-turn latency, process count,
+   idle memory, abandoned preparation cost and failure rate before wider rollout.
+
+The dependency upgrade alone must not start extra processes. Existing sessions,
+legacy adapters, remote execution and unattended work retain their behavior until
+that path explicitly opts in and has equivalent lifecycle coverage.
+
+## Acceptance evidence
+
+- No prompt or model request during preparation; one process for prepare+send.
+- Concurrent prepare/send is deduplicated; exactly one submitted invocation.
+- Two turns reuse the same native PID and preserve provider session identity.
+- Sandbox/MCP/environment/credential changes never reuse a stale process.
+- Deadline, cancellation, disposal, crash and failed initialization release slots.
+- Capacity remains bounded; idle expiration does not interrupt active work.
+- Background events are drained and never misattributed to a later turn.
+- Existing non-opt-in callers and all provider permission tests continue to pass.
+- Native Windows/macOS/Linux lifecycle tests, plus authenticated Claude/Codex
+  smoke measurements with an isolated workspace before enabling Fleet by default.

docs/how-to/measure-session-startup.md

@@ -0,0 +1,132 @@
+# Measure session startup
+
+Enable an optional observer when constructing the runtime to find where a turn
+spends time. This does not change process lifetime, add model requests, or change
+provider events or the remote protocol.
+
+```rust
+use std::sync::Arc;
+use temps_agent_runtime::{AgentRuntime, StartupObserver, StartupTiming};
+
+struct Timings;
+impl StartupObserver for Timings {
+    fn observe(&self, timing: StartupTiming) {
+        // For production exporters, use a bounded channel's try_send here.
+        // Keep this callback quick: it runs on the execution task.
+        eprintln!("{} {:?} {:?} elapsed={:?} event_delivery={:?}",
+                  timing.observation_id, timing.provider, timing.stage,
+                  timing.elapsed, timing.event_delivery_elapsed);
+    }
+}
+
+let runtime = AgentRuntime::builder()
+    .startup_observer(Arc::new(Timings))
+    .build()?;
+# Ok::<(), temps_agent_runtime::RuntimeError>(())
+```
+
+Samples contain only a process-local observation ID, provider, stage and monotonic
+elapsed duration plus cumulative event-delivery wait. No prompt, path, provider session ID, account, environment value
+or error diagnostic is included. IDs distinguish concurrent runs, but do not
+survive a host restart and are not durable invocation identifiers. Treat observers
+as trusted: timing metadata can still reveal provider choice and activity. Do not
+expose a shared observer stream across tenants.
+
+For `run`, elapsed time begins before validation. Subtract consecutive timestamps
+to locate waits:
+
+- `WaitingForPermit` to `PermitAcquired`: process concurrency queue.
+- `CommandPrepared` to `SandboxPrepared`: sandbox preparation, when configured.
+- The last preparation boundary to `ProcessSpawned`: transport process launch.
+- `ProcessSpawned` to `InitialInputWritten`: initial stdin flush, when present.
+- `StreamsAttached` to `FirstOutput`: waiting for the first provider output line.
+- `FirstOutput` to `FirstText`: time until the runtime parses its first nonempty
+  assistant text. This includes waits while delivering earlier events to the
+  application's consumer, not just protocol initialization and provider/model work.
+
+Each sample includes `event_delivery_elapsed`: cumulative time spent awaiting
+`EventSink::emit` before that boundary, including sink persistence, queue waits
+and scheduler delay during delivery. Subtract the change in this field from the
+change in `elapsed` between two samples to separate direct delivery waits. A
+consumer that spends five seconds persisting a session event can otherwise make
+already-buffered text appear five seconds late. Interrupted delivery is counted
+when its future is dropped, including on timeout.
+
+The remainder is still not pure model latency: parsing, provider protocol,
+observer and scheduler overhead remain. Backpressure may also indirectly delay
+the provider itself. The SDK deliberately keeps its existing bounded read and
+event-delivery loop; it does not add an unbounded background reader to manufacture
+an arrival timestamp.
+
+`FirstOutput` may be a handshake, warning or error. `InitialInputWritten` means
+bytes were flushed, not that a prompt was accepted. `StreamsAttached` is not
+universal provider readiness. Claude and Codex may perform initialization after
+these boundaries. Only the first nonempty normalized text delta triggers
+`FirstText`, and a tool-only or failed turn may never reach it. These observations
+do not claim to separate MCP readiness from model latency yet.
+
+A healthy observer receives one terminal observation for each polled run: `Succeeded`, `Failed`,
+`Cancelled`, `TimedOut`, or `Abandoned` when its future is dropped. Observation
+callbacks are synchronous and must not block. Use bounded export with dropped
+samples under overload; observer panics are contained under unwinding builds
+(the standard panic hook can still report them). A panicking observer is disabled
+for that runtime and its clones, including later turns; callbacks already in
+flight may finish. A callback is never invoked during stack unwinding, so an
+abandoned run may have no terminal observation when its caller panics.
+
+Panic-payload cleanup runs inside a second containment boundary. If cleanup also
+panics, its secondary payload is intentionally retained rather than dropped again:
+this avoids recursively unwinding the host. Further calls are disabled, limiting
+this exceptional leak to callbacks already in flight at failure. With a disabled
+observer, telemetry can be incomplete while provider execution continues. As with
+other Rust panics, `panic=abort` terminates the process.
+
+For `run_with_sandbox_recovery`, each provider attempt receives a separate
+observation ID after sandbox policy resolution and concurrency admission. Its
+elapsed time excludes those outer steps. Attempts can therefore be compared for
+process startup, but should not be used as whole-operation latency.
+
+No observer is installed by default. Normalized provider events and invocation
+journals stay unchanged. Built-in retained clients still report
+`retained_process: false`; keeping a logical session does not avoid process
+startup. See [the preparation proposal](../adr/0004-prepared-provider-processes.md)
+for the opt-in process-retention work needed before early preparation can help.
+
+## Optional live baseline
+
+With an authenticated CLI installed, run:
+
+```sh
+cargo run --example startup_timings -- claude
+cargo run --example startup_timings -- codex
+```
+
+These commands send two small real prompts and can incur model usage. They use a
+temporary working directory, deny interactive approvals through the default
+handler, and print timing metadata only. Provider-native session persistence can
+still write to the CLI's normal home directory. The second turn resumes the first
+session; the current driver still spawns a second process. Neither command
+measures a prepared process, and the result is specific to the configured model,
+provider account, network and host load.
+
+## Initial live sample (2026-09-23)
+
+One pair per provider on macOS, with normal installed CLI configuration, an
+isolated temporary workspace and the prompt from the example. Provider pairs ran
+concurrently on the same host, so these are diagnostic samples, not percentiles or
+a controlled performance comparison. Compilation time is outside these samples.
+
+| Provider / turn | Process spawned | First output | First assistant text |
+| --- | ---: | ---: | ---: |
+| Claude, new session | 4 ms | 666 ms | 5,274 ms |
+| Claude, resumed session | 3 ms | 689 ms | 3,548 ms |
+| Codex, new session | 6 ms | 122 ms | 5,236 ms |
+| Codex, resumed session | 1 ms | 67 ms | 4,719 ms |
+
+Both resumed turns launched a fresh process. Most observed first-text latency was
+after process creation, but these samples cannot separate CLI initialization, MCP
+setup, authentication, provider network latency and model computation. They do not
+include Fleet's own context resolution, sandbox preparation or UI rendering. A
+prepared process is not a promise of an immediate model response. Measure explicit
+provider-ready and turn-accepted acknowledgements before assigning the remaining
+latency to the model or predicting a speedup.

examples/startup_timings.rs

@@ -0,0 +1,54 @@
+//! Opt-in live baseline: two turns with provider-native session continuity.
+//! Uses the selected CLI's normal authentication and can incur model usage.
+
+use std::sync::Arc;
+
+use temps_agent_runtime::{
+    AgentRuntime, NoopEventSink, Provider, StartupObserver, StartupTiming, TurnRequest,
+};
+
+struct PrintTimings;
+
+impl StartupObserver for PrintTimings {
+    fn observe(&self, sample: StartupTiming) {
+        println!(
+            "run={} provider={:?} stage={:?} elapsed_ms={} event_delivery_ms={}",
+            sample.observation_id,
+            sample.provider,
+            sample.stage,
+            sample.elapsed.as_millis(),
+            sample.event_delivery_elapsed.as_millis(),
+        );
+    }
+}
+
+#[tokio::main]
+async fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let provider = match std::env::args().nth(1).as_deref() {
+        Some("claude") => Provider::Claude,
+        Some("codex") => Provider::Codex,
+        _ => return Err("usage: cargo run --example startup_timings -- <claude|codex>".into()),
+    };
+    let workspace = tempfile::tempdir()?;
+    let mut builder = AgentRuntime::builder();
+    builder.register(temps_agent_runtime::providers::Codex::app_server());
+    let runtime = builder.startup_observer(Arc::new(PrintTimings)).build()?;
+    let mut session = None;
+    for _ in 0..2 {
+        let mut request = TurnRequest::new(
+            provider,
+            workspace.path(),
+            "Reply with exactly READY. Do not use tools or modify files.",
+        );
+        request.session_id = session;
+        request.timeout = std::time::Duration::from_secs(60);
+        let result = runtime.run(request, &NoopEventSink, None).await?;
+        session = result.session_id;
+        if session.is_none() {
+            return Err(
+                "provider did not report a resumable session; cannot measure continuation".into(),
+            );
+        }
+    }
+    Ok(())
+}

src/lib.rs

@@ -30,6 +30,7 @@ pub mod retained;
 mod runtime;
 mod sandbox;
 pub mod services;
+pub mod startup;
 pub mod transport;
 mod types;
 mod url_security;
@@ -104,3 +105,5 @@ pub use types::{
     QuestionPrompt, QuestionRequest, RunStatus, SecretString, ToolCallStatus, ToolProcessPolicy,
     TurnCapabilities, TurnEvent, TurnProvenance, TurnRequest, TurnResult, Usage,
 };
+
+pub use startup::{StartupObserver, StartupStage, StartupTiming};