feat(activity): scale large reports with streaming aggregation

[wesm]

Large Activity reports now finish without sending raw message-pair intervals
to the browser or keeping one interval object per pair during aggregation. The
version 6 report endpoint streams honest phase and row counters when requested,
preserves plain JSON on the same URL, and returns a bounded summary plus the
first session page. The CLI uses the same paging contract in daemon and direct
SQLite modes.

Session sorting, chart-bucket filtering, and subsequent pages now run on the
server. The Activity page keeps the current report visible while requests run,
cancels obsolete requests, clears stale bucket selections after every full
report refresh, and replaces the summary and table together if the archive
generation changed.

SQLite reuses its existing session-ordered message index for candidate scans,
avoiding a global timestamp index and its append-time write amplification.
PostgreSQL and DuckDB retain range-oriented candidate indexes. All three
backends feed shared Go semantics for clipping, gap handling, model attribution,
bucket membership, and aggregation.

The size-bounded cache only accelerates requests; signed, self-describing report
IDs let cache misses and daemon restarts recompute safely. PostgreSQL now applies
the persisted signing secret, empty keys are rejected, and every decoded report
query is checked against the public range, bucket, filter, and gap limits before
a rebuild starts. Cache-miss session pages retain the long-running request
contract, while streaming progress callbacks stay local to their owning client.

The remaining deliberate limit is usage deduplication: exact, mostly unique
usage data still needs memory proportional to matching report-range usage rows.
A strict bound would require an external spill-and-reduce design and is outside
this change.

[roborev-ci]

roborev: Combined Review (1859a6a)

High-severity issues remain in the PostgreSQL Activity report path, including a guaranteed probe failure and forgeable report tokens that enable resource exhaustion.

High

• internal/postgres/activityreport_probe.go:16 — The probe queries nonexistent PostgreSQL columns (sessions.local_modified_at and messages.id), so every PostgreSQL-backed Activity report fails during its initial source probe. Use actual schema columns such as sessions.updated_at and a valid message-generation indicator, then add an integration test.

• internal/postgres/activityreport_token.go:7 — PostgreSQL report IDs are signed with a publicly computable empty HMAC key because postgres.NewStore leaves cursorSecret empty and pg serve does not apply appCfg.CursorSecret. Attackers can forge query fields, bypass range and bucket limits, and trigger potentially unbounded allocation. Initialize the store with the configured secret, reject empty keys, and revalidate decoded query bounds before rebuilding reports.

Medium

• internal/server/huma_routes_activity.go:28 — The session-page endpoint uses the standard 30-second timeout, but a cache miss rebuilds the complete report. Large reports can time out instead of completing the documented recomputation. Use the long-running route helper or exempt cache-miss rebuilds from the normal timeout.

• internal/server/huma_routes_activity.go:156 — Singleflight builds retain only the initiating request’s progress callback. If its SSE client disconnects, progress may be written through a stale ResponseWriter, while other waiters receive no progress. Give each flight a progress broadcaster with per-waiter subscriptions and cancellation, or do not share builds tied to an HTTP response callback.

• frontend/src/lib/components/activity/ActivityPage.svelte:67 — Bucket selection resets only when report_id changes, but IDs are deterministic. Refreshing an unchanged report resets the table while leaving the bucket badge selected. Clear selection on each successful full-report load generation rather than on report-ID changes.

---

Reviewers: 2 done | Synthesis: codex, 19s | Total: 15m54s

[roborev-ci]

roborev: Combined Review (a9fc43d)

Medium-severity issues remain in sorting, UI state consistency, response size, and CLI continuation efficiency.

Medium

• internal/activity/reportartifacts.go:141 — Ascending agent_minutes and first_active sorts place untimed (nil) sessions first, contrary to the documented behavior that untimed sessions remain at the bottom in both directions. Partition null timing values after timed values independently of sort direction, and test ascending and descending timing sorts.

• frontend/src/lib/components/activity/ActivityPage.svelte:76 — Bucket selection state changes before the page request succeeds. If the request fails, the badge/chart can describe a filter that was not applied; clearing a filter can similarly leave filtered rows displayed without a badge. Commit selection only after success, or roll it back with a request-generation guard to prevent superseded requests from restoring stale state.

• internal/server/huma_routes_activity_sessions.go:157 — Ordinary session-page responses include the complete report even though browser paging ignores it. Because breakdowns are not page-bounded, every sort, bucket, or next-page request can return a multi-megabyte payload. Include report only for refresh_required responses or behind an explicit opt-in for stateless clients.

• cmd/agentsview/activity.go:86 — Continuations using --sessions-report-id still fetch and fully aggregate a new report before requesting the saved generation’s page, causing unnecessary aggregation and potentially using an unrelated default range. When SessionsReportID is supplied, request the session page directly and obtain report metadata through an explicit response option if needed.

---

Reviewers: 2 done | Synthesis: codex, 15s | Total: 13m50s

[roborev-ci]

roborev: Combined Review (c05c398)

The new activity paging and streaming paths are generally sound, but three medium-severity reliability issues should be addressed.

Medium

• internal/server/activity_report_singleflight.go:34 — After the last waiter cancels, the canceled flight remains registered until its build exits. An immediate request for the same key may join that canceled flight and receive a spurious cancellation/500 instead of starting a new build. Remove or mark the flight unusable when its waiter count reaches zero, and have the build goroutine delete the map entry only if it still references that flight.

• internal/server/huma_routes_activity.go:261 — Filter limits count raw bytes without accounting for JSON/base64 expansion. Accepted values containing quotes, backslashes, or control characters can push the signed report token beyond its 8 KiB limit, causing a 500 after aggregation. Validate the fully encoded token before aggregation, or tighten the filter limits so every accepted value is guaranteed to fit.

• cmd/agentsview/activity.go:138 — Activity reports now always use SSE, with the terminal report parsed under a 10 MiB scanner-token limit. Unbounded project/model/agent breakdowns and project metadata can make that line exceed the limit, causing bufio.Scanner: token too long for reports that previously decoded successfully as JSON. Parse SSE without a fixed token ceiling or use framing that safely chunks large terminal reports.

---

Reviewers: 2 done | Synthesis: codex, 13s | Total: 17m2s

[roborev-ci]

roborev: Combined Review (653b4ae)

One medium-severity issue should be addressed before merging.

Medium

• internal/activity/streaming.go:462 — Bucket membership truncates interval timestamps to whole seconds but compares them with untruncated bucket boundaries. Custom ranges with fractional-second bounds can therefore omit short intervals or assign them to the preceding bucket. Normalize bucket boundaries to the same second-resolution wire format before comparison, and add tests covering fractional custom-range boundaries.

---

Reviewers: 2 done | Synthesis: codex, 10s | Total: 11m6s

[roborev-ci]

roborev: Combined Review (4052684)

Changes need revision: two medium-severity issues affect cursor continuity and error disclosure.

Medium

• cmd/agentsview/activity.go:329-370 — Offline cursors fail for current partial ranges. Each invocation derives EffectiveEnd from the new current time and includes it in the artifact digest, causing an unchanged archive’s prior cursor to be rejected. Include the resolved query/filter—especially the original effective end—in the signed cursor and reconstruct that report generation when continuing.

• internal/server/huma_routes_activity.go:80-82, 106-112 — Internal errors are exposed to HTTP and SSE clients. Probe, token, and report-build errors are returned verbatim, potentially leaking database and filesystem implementation details. Log internal failures server-side and return a generic error, while preserving validated 4xx responses and cancellation handling.

---

Reviewers: 2 done | Synthesis: codex, 10s | Total: 12m36s

[roborev-ci]

roborev: Combined Review (8ccac21)

Overall verdict: Two medium-severity cache correctness issues should be addressed before merging.

Medium

• internal/server/activity_report_cache.go:43 — Idle expiry runs only during cache operations, so after the final request, up to 256 MiB may remain retained indefinitely. Add a server-owned timer or periodic sweep to remove entries after 15 minutes of inactivity.

• internal/activity/reportartifacts.go:40 — The source probe omits project-identity state even though identity observations affect project keys and labels. Identity-only updates can leave cached metadata stale without setting refresh_required. Include an identity-generation value in each backend probe and test identity-only cache refreshes.

---

Reviewers: 2 done | Synthesis: codex, 10s | Total: 11m54s

[roborev-ci]

roborev: Combined Review (345134a)

Review identified one medium-severity paging bug; no high or critical issues were found.

Medium

• internal/server/huma_routes_activity_sessions.go:121, cmd/agentsview/activity.go:446 — Continuation cursors using non-default sort or bucket settings are rejected unless every paging flag is repeated, even though the signed cursor contains that state and documentation says the cursor can be passed alone. Decode the cursor first and inherit sort, direction, and bucket for options not explicitly supplied; reject only explicit mismatches.

---

Reviewers: 2 done | Synthesis: codex, 8s | Total: 12m15s

[roborev-ci]

roborev: Combined Review (d7edc7e)

Changes need a fix for cursor-based pagination.

Medium

• cmd/agentsview/activity.go:328 — Cobra’s default sort/direction values are mistaken for explicitly supplied options, so continuing a non-default page without repeating those flags returns invalid sessions cursor. Track whether each flag was changed, or apply defaults during normalization, and add a command-level continuation test.

---

Reviewers: 2 done | Synthesis: codex, 6s | Total: 10m32s

[roborev-ci]

roborev: Combined Review (ba8fcb9)

Medium-severity correctness issue found in activity report totals; no security regressions identified.

Medium

• internal/db/activityreport.go:59 — Schema-v6 reports include all session rows but leave the required sessions_total at zero across SQLite, PostgreSQL, and DuckDB implementations. Set SessionsTotal = len(artifacts.Sessions) in all three stores.

---

Reviewers: 2 done | Synthesis: codex, 17s | Total: 14m12s

[roborev-ci]

roborev: Combined Review (1e6587c)

Code review passed: no Medium, High, or Critical findings were identified.

---

Reviewers: 2 done | Synthesis: codex, 9s | Total: 12m0s