perf(datastore): replace single-column event indexes with composite (bucket_id, timestamp) index

[ErikBjare]

Problem

aw-server (python) was found pinned at ~60% CPU with terabytes of disk reads per day (38 TB over ~30 days on the machine where this was diagnosed), and the log full of Heartbeat lock could not be acquired within timeout 503s, causing dropped heartbeats and fragmented events.

Root cause: the peewee storage only has single-column indexes on eventmodel (bucket_id, timestamp). Every event query filters on bucket_id and orders by timestamp, for which SQLite picks:

SEARCH eventmodel USING INDEX eventmodel_bucket_id (bucket_id=?)
USE TEMP B-TREE FOR ORDER BY

i.e. it reads all of a bucket's rows and sorts them in a temp B-tree — even for the LIMIT 1 last-event query that replace_last() issues on every merged heartbeat. On a 1.65M-row bucket that's a multi-second full-bucket scan per heartbeat, back to back, forever. (The WAL change in v0.14.0b1 fixed write amplification; this read-side pathology is independent of it, and grows linearly with bucket size.)

Fix

Replace the single-column indexes with a composite (bucket_id, timestamp) index, which serves the seek, the range and the ORDER BY in one pass. This mirrors aw-server-rust's v5 migration (events_bucketrow_starttime_endtime_index), including dropping the old indexes before creating the new one in the same transaction so freed pages are reused (no file growth) and inserts get cheaper (2 indexes → 1).

• New databases: model no longer declares the single-column indexes; auto_migrate() creates the composite.
• Existing databases: auto_migrate() detects the missing composite index, drops the old ones, and builds it (logged, since it takes a while on large DBs).

Measured (real 1.45 GB DB, 6.7M events, largest bucket 1.65M rows)

query	before	after
last-event (replace_last, per merged heartbeat)	5.06 s	0.028 s
1-day range (get_events)	4.16 s	0.037 s
one-time migration	—	~54 s, no file growth

This also eliminates the heartbeat-lock 503s, since the lock was being held across those multi-second scans.

Tests

• test_index_migration: fresh DBs get the composite index; simulated pre-migration DBs (old single-column indexes) are migrated.
• test_query_plan_uses_composite_index: query-plan regression guard — asserts the hot queries are served by the composite index and never fall back to USE TEMP B-TREE FOR ORDER BY. Deterministic (no timing flakiness), catches this class of regression in CI.

Note: cross-bucket queries filtering on timestamp alone would no longer have an index, but nothing in aw-core issues such queries (all event queries are bucket-scoped) — same trade-off aw-server-rust made.

Intended for inclusion in v0.14.0b3 (needs an aw-core release + bump in aw-server).

[greptile-apps]

Greptile Summary

This PR replaces Peewee’s single-column event indexes with a composite bucket-and-timestamp index.

• Migrates existing databases transactionally by dropping the old indexes and creating the composite index.
• Updates new-database model metadata to avoid recreating the obsolete indexes.
• Adds migration and SQLite query-plan regression tests.

Confidence Score: 5/5

The PR appears safe to merge, with the migration matching the repository’s bucket-scoped event-query patterns.

The initialization sequence creates the event table before migration, the index replacement is atomic and idempotent for supported schema states, and all repository event queries retain bucket_id as the composite index’s leading predicate.

Important Files Changed

Filename	Overview
aw_datastore/storages/peewee.py	Adds an idempotent composite-index migration and aligns EventModel metadata with the migrated schema.
tests/test_peewee.py	Verifies legacy-index replacement and guards the bucket-scoped hot queries against temporary sort plans.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[PeeweeStorage initialization] --> B[Create tables if absent]
    B --> C[Inspect eventmodel indexes]
    C --> D{Composite index exists?}
    D -->|Yes| E[Continue startup]
    D -->|No| F[Drop legacy bucket and timestamp indexes]
    F --> G[Create bucket_id, timestamp composite index]
    G --> E

Loading

_{Reviews (1): Last reviewed commit: "perf(datastore): replace single-column e..." | Re-trigger Greptile}