statev2.sqlite grows without bound (33 GB in 7 weeks): retention/compaction for orchestration events and turn items

Environment: t3 v0.1.7-nightly.20261101.4035. Linux host, one heavy user with many agent threads.

Observed: ~/.t3/userdata/statev2.sqlite is 33 GB (8.05M pages × 4 KiB). It's now the largest file on the host's root disk, and it keeps growing. These are read-only aggregates from dbstat and row counts:

object	size
`orchestration_events`	~15.6 GB, 3.46M rows, oldest 2026-09-13
`orchestration_v2_projection_turn_items`	~6.8 GB
`orchestration_v2_projection_nodes`	~2.2 GB
event indexes (autoindex + 3 stream indexes)	~3 GB

freelist_count is 0, so a plain VACUUM would reclaim nothing. Only pruning can.
About 1.15M of the events are more than 14 days old.

There's no supported way to bound this: no retention setting, no prune command, and no documented safe manual procedure.

Ask:

Server-side retention. On startup or daily, prune orchestration events (and receipts) for archived or deleted threads, and for events older than a configurable window, once their projections are current, so replay never needs pruned history.
Cap per-thread turn-item and activity history, or make it configurable.
Reclaim the space after pruning, for example auto_vacuum=incremental with periodic incremental_vacuum, so the file actually shrinks.
A supported t3 db prune / t3 db compact command, usable with the server stopped, plus docs on what's safe to delete.

I'm not posting row contents or private data; I can share more aggregate numbers on request.

Second data point, macOS, build of main ac8dce0cc3 (v0.1.6), one heavy user, ~11 weeks of history.

statev2.sqlite was 15.06 GB; VACUUM INTO brought it to 9.74 GB. The 4.6 GB freelist came
from the 6-hourly event compaction (orchestration_events grows without bound: streaming updates are stored as permanent events and never compacted #21154): it deletes superseded events, but on a database
created before auto_vacuum=INCREMENTAL became the default, nothing hands those pages back.
After the vacuum (with auto_vacuum=INCREMENTAL), DatabaseStorage.reclaim keeps the freelist at 0.
Live data still grows about 1.5 GB a week. Over the week before the vacuum, page_count stayed
at 3,676,931 while 357k free pages were reused.
dbstat on the 15 GB file: orchestration_events 4.7 GB (1.07M rows) plus 1.1 GB of event
indexes; orchestration_v2_projection_turn_items 2.1 GB; orchestration_v2_projection_nodes
0.9 GB.
Event payload bytes: turn-item.updated 1.86 GB, node.updated 0.69 GB,
provider-turn.updated 0.15 GB. Compaction keeps the first and the latest turn-item.updated
for each item, so a finished tool call's output is stored twice: in the projection row and in
the latest event.
The largest single category is Pi read tool output: 660 MB in turn items. Of that, 325 MB is
961 image reads (base64 PNG screenshots, up to 3.8 MB each), and the events hold the same
~331 MB again.
By thread origin: delegated-task child threads hold 55% of event bytes, MCP-launched threads
26% and user threads 19%.

Retention that would bound this for us while keeping settled and archived history readable:
purge deleted threads (#17845); once a thread is settled, drop the item events whose final
state the projection rows already hold; and store large tool-output and image
blobs once, by reference, instead of inline in both the event and the projection.

Note

Written by Claude Opus 5.5 (Claude Code) on behalf of the reporter. All numbers are read-only aggregates from the live database.

Data point on 0.1.7-nightly.20261102.4093 (Linux x86_64, headless, systemd user service), measured 2026-11-03. statev2.sqlite is 12.83 GB, the oldest deleted-thread event I saw is from about three weeks ago, and it's already been through t3 doctor (auto_vacuum 2, freelist_count 0).

Object	Size	Share
`orchestration_events` and its indexes (1.56M rows)	7.33 GB (table 6.22, indexes 1.11)	57%
Turn items, their indexes and positions (465k rows)	3.37 GB (table 2.55)	26%
Nodes and indexes (460k rows)	1.93 GB (table 1.61)	15%
Everything else	about 0.2 GB	2%

Supersession compaction is already near its floor here:

There are 611,805 turn-item.updated events for 465k turn items, and 759,045 node.updated events for 460k nodes.
Those two event types hold 1.86 GB and 1.08 GB of payload_json characters.
The last six completed compaction runs removed 674–7,674 events each, under 0.5% of the log, so more supersession alone won't bound this.

Turn item payload by type (characters):

Type	Rows	Size
`command_execution`	98,700	0.92 GB (largest single row 1.35 MB)
`reasoning`	325,983	0.45 GB
`dynamic_tool`	8,436	0.19 GB

Most of the data isn't recent: 69% of turn items belong to threads with no turn-item update in the past 7 days (by row count, not bytes).

Space inside allocated pages: orchestration_events had about 1.8 GB (29%) unused about an hour after the doctor run. With rows this size on 4 KiB pages, I suspect little of it can be recovered, which fits the 3.42% reduction in the rehearsal on #17845.

For anyone else measuring this: on SQLite 3.37.2, dbstat with aggregate=1 wrapped its size columns at 32 bits, so tables over 4 GiB came back too small or negative. Per-table totals only added up to the file size once I added 2^32 back for the two wrapped tables.

The time t3 doctor took on this database is in #29941.

Another data point: macOS, t3 v0.1.7-nightly.20261103.4119, one user. I measured it on 2026-11-03 with read-only queries.

statev2.sqlite is 18 GB: 4,705,290 pages of 4 KiB. freelist_count is 0 and auto_vacuum is 0 (NONE), so VACUUM gets back no space.
orchestration_events has 1.58M rows.
I took a sample of the last 100k events (sequence > 1480000):

Event type	Events	`payload_json`	Largest row
`turn-item.updated`	52,343	1.56 GB	681 KB
`subagent.updated`	2,777	238 MB	267 KB
`node.updated`	39,337	49 MB	1 KB

At this rate the event log grows by about 1.8 GB for each 100k events.
A full dbstat scan did not finish in 5 minutes. The T3 server restarted 2 times while the scan ran. I did not confirm the cause, but Any read transaction left open on statev2.sqlite freezes every server write once the WAL passes 32 MB #23482 describes the same lock pattern.

On this machine the database fills the disk faster than any other app data. I support retention for settled threads and storing large tool output once.

Another data point: Linux x86_64, 0.1.7-nightly.20261103.4106, one user running ~5–8 agent threads in parallel most of the day. Read-only aggregates only.

statev2.sqlite: 39.7 GB (9,689,297 pages of 4 KiB), freelist_count 0, so VACUUM won't shrink it.
Growth: 18.5 GB on 10-20 → 39.7 GB on 11-03, about 1.5 GB/day.
By table (dbstat):

object	MB
orchestration_events	20,201
orchestration_v2_projection_turn_items	7,171
orchestration_v2_projection_nodes	2,622
sqlite_autoindex_orchestration_events_1	2,152
idx_orch_events_stream_sequence	672
idx_orch_events_stream_version	664
idx_orchestration_events_agent_stream_sequence	629

Rows: 5,237,656 events, 1,221,385 turn items.

Events are ~51 % of the file and the four event indexes another ~10 %, so retention or compaction of settled threads' events would cover most of it.

Note

🤖 Claude Opus 5.5 (Claude Code, via t3 triage) posting on behalf of the reporter

Another data point: Linux x86_64, server 0.1.4-nightly.20261027.3796 (systemd user service, since updated to 0.1.7-nightly.20261104.4151), one heavy user with about 30 concurrent Claude sessions. All numbers come from read-only queries on 2026-11-04.

statev2.sqlite is 40.7 GB: 9,944,311 pages of 4 KiB, freelist_count 0, auto_vacuum 0. VACUUM would reclaim nothing.
orchestration_events: the highest rowid is 6.84M. Using rowids as a proxy, about 1M events were written on 10-08, 3M by 10-17, 5M by 10-22 and 6M by 10-27. That is roughly 100k events a day lately.
Event types among the 40k most recent events: turn-item.updated 19,010, node.updated 13,782, provider-turn.updated 2,938, subagent.updated 1,183, message.updated 1,128.
Average payload_json size is 3.3 KB; the largest is 236 KB.
Other tables, by rowid: orchestration_v2_projection_turn_items about 1.26M, orchestration_v2_projection_nodes about 892k, orchestration_command_receipts about 1.0M.
There are 10,985 v2 threads, of which 10,970 are not archived.
The effect on 0.1.4: primary-key reads reached a p99 of 15 s and a maximum of 57 s, and the server froze for 20–40 s (see Server event loop freezes 6–28 s on a normal-sized database (5.9 GB), so remote devices lose Send #26224). On 0.1.7 the p99 dropped to under 80 ms, but the database is the same size and still growing.

Another data point, plus a question about what's left after #29806.

Setup. macOS (Mac Studio, 32 GB RAM), 0.1.7-nightly.20261103.4138, one heavy user running many agent threads in parallel. All numbers are read-only, aggregate-only sqlite3 -readonly scans taken while T3 was running.

Size and growth

statev2.sqlite: 33.4 GB (8,148,267 pages × 4 KiB), auto_vacuum = 0, freelist ~0.12 GB. It grew 2.2 GB in the last ~18 h. The disk has 12 GiB free (98% used).
Retained event JSON per week: 0.48 → 1.87 → 2.79 → 3.76 GB, with ~1,900 new threads a week. 99.4% of the 6,117 threads were active in the last 30 days.
Breakdown from a dbstat pass at 29.3 GB:

Part	Size
`orchestration_events` + 8 indexes	18.8 GB (64%)
turn-item projection + indexes	6.7 GB
node projection + indexes	2.4 GB
other projections	~1.2 GB
receipts + outbox + message FTS	< 0.3 GB

#29806 should take care of most of the first row. It wasn't in my build yet, so I'll post before/after numbers once it has run. Here's what I think it leaves, measured on this DB:

Deleted threads keep their rows. Two threads deleted here still held 542,376 events (1.02 GB of event JSON) and ~0.42 GB of projection rows, with identical counts before and after a completed compaction pass. perf(server): prune events the projections already cover #29806 will prune their old events like any others. As far as I can tell from main, though, nothing deletes their projection rows ([Bug]: Deleting a thread says it permanently clears its history, but V2 keeps every event and projection row #17845).
Large tool output dominates the projections. Turn-item projection rows ≥ 4 KB hold ~2.8 GB, about 71% of that table's payload. In the event log, rows ≥ 4 KB are 7% of rows but 38% of the bytes. fix(server): keep large tool output out of SQLite #28167 (keep large tool output out of SQLite) targets exactly this.
Freed space can't be reclaimed on a full disk. Pruning puts pages on the freelist. On an auto_vacuum = NONE database, getting them back needs t3 doctor, which runs a full VACUUM with T3 stopped and needs roughly the live data size in free space. On a nearly full disk, that's the one moment it can't run.
No way to act now. The prune only deletes behind a mark that is at least a day old. doctor runs the same pruneEventStore(), so on a database without a mark it prunes nothing and then VACUUMs all 33 GB.

Small things that would help, smallest first

An on-demand prune, e.g. by event age, so a user with a full disk can start it right away. I believe the snapshot fallback for resumes already covers the client side, but you know that path better.
Letting doctor write the compacted copy to another volume (VACUUM INTO <path> plus a swap), so reclaiming doesn't need headroom on the same disk.
A further step could reuse T3's remote layer: the compacted copy, or a vault file, could go to a volume on another of the user's environments (LAN, Tailscale or T3 Connect). As far as I can tell, nothing moves database files between environments today, so this would be new. It would fit as a later step on top of the local-path option, in line with the "remote ready" model.
Deleting a thread removes its projection rows, or at least the deletion dialog doesn't imply it frees space.

A design question for later: moving inactive history out of the live DB. For example, into a separate file that stays searchable and restorable, so the live DB holds only active work. On this DB it would free about 1.1 GB today: settled threads idle for 7+ days hold ~20% of the projection payload, and threads idle for 30+ days hold almost none. Here the growth comes from volume. On installs that run for many months, where history accumulates, it would matter more. Is that a direction you'd want, or is prune + external tool output + deletion purge the intended model?

Adding a data point: our state database is now 24.4 GB (decimal), 8.6 days after a fresh start, averaging roughly 2.8 GB/day.

An earlier read-only breakdown at 23.0 GB showed:

60.6% streamed-event history, 37.8% saved thread data and supporting tables, and 1.6% other storage (shares include indexes).
Of the payload attributable to threads, 54.7% belonged to child/subagent threads and 45.3% to top-level threads.
Long tool output and screenshots were stored twice: in events and saved thread content. Tool output accounted for about half of stored payload.

Deleting or archiving threads does not free database space on our build.

We were on 0.1.4-nightly.20261027 and moved to v0.1.7 today for event pruning.

Is thread retention (forgetting old threads) planned, and is there a recommended interim practice?

Posted on behalf of Lars by Claude Opus 5.5 on T3 Code.

The one-page reclaim loop reads and writes far more than it frees

Separate from the retention gap above: on a database that already has auto_vacuum=INCREMENTAL and a large freelist, DatabaseStorage.reclaim (apps/server/src/persistence/DatabaseStorage.ts, still the same on main at d5fc29fe5cb6 and in 0.1.8-nightly.20261106.4288) is itself the steady disk load.

The comment says the driver steps once per statement, so the loop calls PRAGMA incremental_vacuum(1) up to 1024 times a minute, with a PRAGMA freelist_count before each one. That premise is wrong for node:sqlite's DatabaseSync, which is what the server uses. One exec("PRAGMA incremental_vacuum(N)") frees N pages in a single statement.

Measured on Node's own DatabaseSync (Linux, Node with --experimental-sqlite), on a throwaway 8,030-page database with 4,000 free pages, nothing else touching it:

call	pages freed	time
one `exec("PRAGMA incremental_vacuum(200)")`	200	11 ms
200 separate `exec("PRAGMA incremental_vacuum(1)")`	200	962 ms

So the per-page loop is about 90× the work of one statement for the same pages, and it never catches a freelist of hundreds of thousands of pages: 1,024 pages a minute is roughly 4 MB a minute, about 6 GB a day.

On a live 5.7 GB database (1,486,689 pages, 801,554 of them free, auto_vacuum=2, server 0.1.8-nightly.20261105.4234), the server's own trace for a quiet stretch was 16,200 sqlite spans, of which 8,064 were incremental_vacuum(1) and 8,071 were freelist_count. The other 65 spans were ordinary lookups. The same shape holds across the other rotated trace files: several thousand vacuum statements per file, each 1–9 ms.

incremental_vacuum relocates a live page from the end of the file into the free page and truncates. With 800k free pages scattered through a 5.7 GB file, each of those statements is a random read plus a write, and the process's own /proc/<pid>/io confirms the volume. The t3 serve process (up since 5 Nov, never restarted for this) read and wrote:

day	read	written
4 Nov	132 GB	234 GB
5 Nov	268 GB	202 GB
6 Nov	82 GB	42 GB
7 Nov, first 47 minutes	2.1 GB	5.0 GB

The cgroup's own io.stat is lower (6.7, 15.2 and 6.6 GB read on 4, 5 and 6 Nov), so some of the process figure is re-read of pages the kernel counted once. Either way it is orders of magnitude above the 4 MB a minute the loop actually frees.

Two one-off minutes are much larger than the loop explains and look like a scan: 62 GB read in the 04:14 minute on 6 Nov, and 2.7 GB in the 02:21 minute. The six-hourly compaction (pruneEventStore) walks orchestration_events even when it deletes nothing, and no trace file survives from those minutes, so I can't attribute them more tightly.

Suggested change: one PRAGMA incremental_vacuum(1024) per pass instead of 1,024 statements of one page. That keeps the one-minute bound and the yield between passes, and frees the same pages in about a ninetieth of the work. Capping the freelist the loop will chase (or only reclaiming right after compaction frees pages) would stop it paging a multi-GB file all day for 4 MB of progress.

Happy to retest a build. I have not restarted the server or vacuumed the live database.

Another data point, with one requirement that hasn't been stated yet.

Host: Linux team host, t3 0.1.8-nightly.20261106.4288 (auto-updated today), one heavy owner running many concurrent agent threads, plus remote environments.
Size: statev2.sqlite is 40.8 GB (40,791,117,824 bytes; file size only, no database reads). It is the largest single item on a 344 GB root filesystem that is 83% full. T3 worktrees under ~/.t3/worktrees add another 41 GiB on the same disk.

Requirement: retention and space reclaim have to work while the server is busy. On this host the server is never idle. There's always an in-flight turn, a delegated child, or a scheduled task. So any fix that needs "stop the server, then prune/compact" (a stopped-server t3 db compact, or t3 doctor, which pauses and restarts the service) can't be scheduled here. What would work:

Online retention that prunes events for archived/settled/deleted threads, and events older than a window, in small bounded transactions that never hold the write lock long enough to stall the event loop (SQLite writes run on the server's event loop, so one slow transaction disconnects every client #30065, Any read transaction left open on statev2.sqlite freezes every server write once the WAL passes 32 MB #23482).
Incremental reclaim that is rate-limited and bounded per step, instead of the per-page loop described in the latest comment.
A prune and reclaim pass tied to the restarts T3 already does for its own updates. Those restarts happen anyway, so a bounded pass there costs no extra downtime. It would also help Service updates keep the server offline while copying a large database on filesystems without reflink #21715, since update copies would be smaller.
A read-only t3 db stats that reports size, prunable bytes and the freelist without users running ad hoc sqlite3 -readonly scans.

Happy to share more aggregate numbers on request. No row contents or private data were read.

Correction and growth rate for the data point above: the 40.8 GB figure was measured earlier today. It is now 41.5 GB (41,546,035,200 bytes) about five hours later, so roughly +0.75 GB in 5 h (~3.6 GB/day) under normal concurrent use. File size only.

@ViperJuice

@ViperJuice You have great suggestions!

Mine grew 15GB more in just a few days. I can't keep this up. Looks like it's pushing the entire chat history for every update or something because it's way too big.

I'm strained on DB size atm because I can't buy more SSDs. On top of that, my worktrees are already causing storage constraints. I wish T3 Code worktrees were hardlinks instead of copies of files :/.

Navigation Menu

Repository navigation

statev2.sqlite grows without bound (33 GB in 7 weeks): retention/compaction for orchestration events and turn items #28903

The one-page reclaim loop reads and writes far more than it frees

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

statev2.sqlite grows without bound (33 GB in 7 weeks): retention/compaction for orchestration events and turn items #28903

Description

Activity

astarktc commented on Nov 2, 2026

chris-sanders commented on Nov 3, 2026

poltorrr commented on Nov 3, 2026

simasrazinskas commented on Nov 3, 2026

nick-potts commented on Nov 4, 2026

ranvier2d2 commented on Nov 4, 2026

larzp commented on Nov 5, 2026

Waseemilyas commented on Nov 7, 2026

The one-page reclaim loop reads and writes far more than it frees

ViperJuice commented on Nov 7, 2026

ViperJuice commented on Nov 7, 2026

Sawtaytoes commented on Nov 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions