Skip to content

Change: skein-observability #122

Open

Description

@androidand
Proposal

Why

Charter §17 requires structured telemetry, correlation across operations, metrics, and
diagnostic CLI commands, and the system must be diagnosable without the dashboard.

Research (R10) found the Go skein repo already has internal/metrics (per-provider/
model/stage efficiency accounting, D1 fleet-efficiency-metrics; throughput) and
internal/traces, but no OpenTelemetry in go.mod. This change adds OpenTelemetry
where suitable, an explicit correlation-ID convention across the planes, the charter §17
metrics, and the diagnostic CLI — building on the existing metrics/traces rather than
replacing them.

What Changes

  • Adopt OpenTelemetry where suitable (traces + metrics), building on the existing
    internal/metrics and internal/traces.
  • Correlate every operation through objective_id, work_item_id, execution_id,
    agent_id, session_id, turn_id, model_request_id, resource_lease_id,
    trace_id.
  • Expose the charter §17 metrics (throughput, efficiency, claim latency, lease usage,
    error rates) and the diagnostic commands skein status|projects|agents|work|fleet| trace|doctor, reconciled with the existing Go skein CLI.
  • The AHP ahp-otlp: telemetry channel is a natural fit for host telemetry.

Impact

  • Affected specs: skein-observability (new, cross-cutting), skein-control-plane
    (correlation IDs), ahp-host (OTel telemetry channel), mission-control (metrics
    views).
  • Additive telemetry; disabling the exporter restores prior behavior.

Tasks

Phase 1: Correlation IDs

  • 1.1 Define the correlation-ID set (objective/work_item/execution/agent/session/turn/
    model_request/lease/trace) and propagate it across the control plane, project
    servers, ACP runtime, and AHP host.
  • 1.2 A work item's full history is reconstructable from its correlation IDs alone.

Phase 2: OpenTelemetry

  • 2.1 Add OpenTelemetry (traces + metrics) where suitable, building on the existing
    internal/metrics and internal/traces. needs-research: exact OTel SDK and
    exporter for the Go runtime and the TS project server.
  • 2.2 Export the charter §17 metrics: throughput, efficiency, claim latency, lease
    usage, error rates.
  • 2.3 Use the AHP ahp-otlp: telemetry channel for host telemetry.

Phase 3: Diagnostic CLI

  • 3.1 skein status|projects|agents|work|fleet|trace|doctor (charter §17),
    reconciled with the existing Go skein CLI conventions.
  • 3.2 skein trace <work_item_id> reconstructs the full operation history from
    correlation IDs.
  • 3.3 The system is diagnosable headless (CLI), without the dashboard.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions