Consolidate archive sync writers on Bun transactions - #1361
Consolidate archive sync writers on Bun transactions#1361mariusvniekerk wants to merge 43 commits into
Conversation
roborev: Combined Review (
|
005ad77 to
a37d0df
Compare
roborev: Combined Review (
|
roborev: Combined Review (
|
935b4e0 to
178a8ff
Compare
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
e02f880 to
0ee0579
Compare
roborev: Combined Review (
|
A multi-session source can commit early results before a later member write or final scoped link fails. Queue each committed session itself so deferred repair can restore authoritative incoming-edge parentage instead of only repairing its outgoing children.
Append-only message paths could introduce duplicate evidence endpoints while leaving trusted recall provenance unchanged. Reconcile each affected session inside the message transaction and emit revocation diagnostics only after the write commits.
Bun bulk inserts render an entire model slice as one dialect-aware VALUES statement, so the manual 100-row loops added statement overhead without protecting a driver bind limit. Use Bun's documented slice API throughout canonical writes and keep malformed SQLite timestamps in a one-field archive model extension while strict adapters retain the shared timestamp type.
Secret findings must use the same result-event coordinate that persistence stores or reveal cannot relocate their source. Share the canonical distinct-index and collision-fallback rule between Bun writes and scanning, and cover a non-positional write/scan/reveal round trip.
Late tool results and subagent links change fields included in trusted recall digests even when no message row is appended. Reconcile whenever an incremental write changes transcript content so stale evidence cannot remain trusted.
Bun formats slice inserts into SQL, so unbounded dynamic payloads can create avoidable query and hook working sets even without a driver bind limit. Bound canonical writes by the archive's intentional maximum payload while keeping oversized logical rows intact. SQLite's permissive archive timestamp seam also needs the same canonical availability policy at every shared read boundary. Normalize timing, recent-edit, and content-search hydration, invalidate secret coordinates written before canonical result indexes, and pin PostgreSQL-compatible pricing rounding edges.
The payload estimator runs once per canonical row. Go's reflect.Value.Fields iterator allocates on each struct traversal, doubling bulk-insert allocation counts and pushing archive sync beyond the benchmark gate even though the byte-budget behavior itself was sound. Keep indexed reflection on this hot path and document the targeted modernize suppression so lint cannot rewrite the regression back in.
PostgreSQL rounds TIMESTAMPTZ fractions to microseconds while DuckDB truncates them, so preserving nanoseconds until adapter persistence can move pricing metadata by one microsecond across otherwise identical reports. Apply the existing PostgreSQL half-even rule to canonical pricing rows and revision timestamps before any Bun adapter writes them. Pin the three-backend parity fixture to a fractional timestamp that deterministically exercises the boundary.
SQLite accepts date-only, time-only, and numeric values in julianday even though the canonical Bun timestamp scanner rejects them. Ranking those legacy values before hydration can let an unavailable edit displace a valid file from a bounded page. Use the canonical parser through the SQLite adapter availability seam before recent-edit ranking, while native typed backends keep their existing expression. The regression combines an unsupported future value with a canonical edit so pagination order is observable.
Schema-v11 mirrors can retain timestamps truncated before canonical pricing normalization because timestamp-only differences are intentionally excluded from incremental pricing updates. Advance the disposable mirror boundary so the next push rebuilds and atomically swaps affected files. The upgrade regression simulates the old persisted value and verifies the rebuilt mirror carries the shared half-even microsecond timestamp.
The batch estimator measures static Go row size plus dynamic values before Bun formats SQL. Calling 16 MiB a formatted-query working-set bound overstated what the calculation guarantees because syntax, escaping, and Bun copies add overhead. Describe it consistently as a pre-format payload target while retaining the same byte budget, oversized-row behavior, and transaction boundaries.
Daily usage filtering and Claude snapshot deduplication can discard most raw candidates, so counting every message and usage event first adds two full scans and can reserve memory that the result never uses. Let the accepted result slice grow from the rows that survive streaming. Keep the existing candidate order, filters, and deduplication unchanged.
Removing separate candidate-count scans exposed the normalized usage path to repeated growth and backing-array copies on large archives. Preserve Claude cross-session deduplication with its widened stream, while filtering ordinary messages and usage events in SQL and deriving their allocation capacity from the streaming queries themselves. This keeps the query path unbounded by arbitrary row or bind limits and avoids reintroducing standalone full-table count queries.
Bun formats tuple lists into the SQL statement, so removing the old fixed-row batching left large worktree-mapping publication windows able to build one unbounded query. Apply the canonical pre-format payload budget to tombstone keys so statement growth follows actual string size without reintroducing a guessed row or bind-variable ceiling.
SQLite julianday accepts date-only, time-only, numeric, and some invalid calendar text that the canonical Bun timestamp scanner rejects. Usage projections and bounded queries must classify those message timestamps the same way so they fall back to the session timestamp instead of disappearing or failing during scan. Keep canonical UTC rows on a validated SQLite-native fast path and reserve the exact Go parser UDF for legacy or suspicious text, preserving the usage benchmark allocation profile.
The Windows DuckDB driver converts a scanned time.Time one microsecond below the persisted value, which made the mirror-rebuild contract depend on driver decoding. Assert DuckDB's stored epoch microseconds directly so the test protects canonical pricing precision on every platform.
Timing responses must use the same persisted message IDs as message reads so frontend turn and tool-call joins remain valid. Malformed provider timestamps are repaired at ingestion and during the v84 archive rebuild instead of being tolerated through backend-specific read or fingerprint fallbacks. This keeps SQLite, PostgreSQL, and DuckDB on one strict canonical timestamp contract.
The Bun writer cutover made ID-less parser messages escape and allocated fresh sanitation and canonical-row backing for every session. That pushed cold sync bytes and contributor rebuild allocations over the benchmark gate despite unchanged behavior. Keep per-database schema creation and validation intact. Format canonical timestamps directly into Bun queries, separate target-assigned message conversion from source-ID preservation, and recycle only cleared fixed-row backing bounded by the existing payload budget.
PostgreSQL shortened UTC offsets when shared queries cast canonical timestamps to text, which made strict scans reject valid rows. Keep real timestamp columns typed and type only PostgreSQL's null-only union branches. The allocation-free timestamp appender also cannot receive a typed nil pointer through a direct Bun query argument. Convert that nullable policy value to SQL NULL at the call boundary while preserving value-based timestamp formatting.
DuckDB on Windows can round an RFC3339 string cast one microsecond below the canonical value. That made rebuilt mirrors lose the PostgreSQL-compatible pricing revision even though the shared model had already normalized it. Let DuckDB format canonical Bun timestamps with its typed microsecond literal. Keep SQLite and PostgreSQL on the existing RFC3339 path so archive text remains unchanged.
The data-mode workflow can finish its mapping mutation before the follow-up project inventory reaches the page under CI load. Await both successful responses so the DOM assertion observes the completed user workflow instead of racing the refresh.
Analytics and search planners need one shared relational shape while SQLite, PostgreSQL, and DuckDB retain only the SQL expressions and native sources that differ. Define compositional fragments so Bun keeps query construction and execution ownership, including paired model turns, direct tool facts, timezone projection, lexical ranking, and vector distance/setup seams.
Shared analytics filters must retain the indexed UTC prefilter before exact local-date matching, and usage SQL must use the same activity cutoff as its final reducer. Make the analytics date grain explicit, pass one captured reference time through usage queries, and keep FTS-dependent contract assertions conditional when the optional SQLite module is not compiled.
The unified storage stack must keep archive-scale joins, grouping, windowing, ranking, and limiting inside the database instead of rebuilding backend-specific stores or hydrating full fact sets into Go. Move the remaining analytics panels and activity-report candidate selection onto shared Bun planners while retaining only compact response shaping and content classification in Go.\n\nPreserve paired model ownership, direct tool ownership, exact local date filters, active-duration ranking, and cross-session usage snapshot selection across SQLite, PostgreSQL, and DuckDB.
PostgreSQL must keep native timestamps typed until the shared Bun boundary formats them. Backend-rendered timestamp text violates the strict canonical timestamp contract. Model-filtered output-token aggregates expose their reporting flag as an integer. Match that projection type when ranking sessions, and ignore unavailable timestamps before computing active-duration gaps so database ranking preserves the previous reducer semantics. Generated with Codex (gpt-5.6-sol) Co-authored-by: Codex <codex@openai.com>
fae4c68 to
90bd14e
Compare
roborev: Combined Review (
|
Bun deprecated its legacy flat-slice IN helper, which caused the stack tip to fail staticcheck. Use the documented List formatter so the pin-remap query keeps the same SQL semantics on every dialect while remaining compatible with the current Bun API.
roborev: Combined Review (
|
Ordinary sync, full-session replacement, incremental repair, dependent accounting writes, and orphan recovery now share the canonical Bun transaction core. This removes staged commits that could expose mixed session state after a late failure and eliminates hand-maintained common child projections during archive recovery.
SQLite still owns the operations that genuinely depend on its adapter: FTS5 maintenance, pins and physical-ID remapping, ATTACH and temporary-table lifecycle, provenance repair, sanitization, and legacy capability probes. Transcript comparison remains proportional to session history, while changed-row persistence and FTS mutation stay scoped to the affected diff.
Review the atomic session-batch routing, canonical finding helper, and registry-derived orphan-copy boundary.
make test-shortremains affected only by the existing macOS FSEvents delivery timeouts; focused sync, full database, DuckDB, and PostgreSQL/activity suites pass.generated by a clanker