feat(change-stream): SQLite-backed garbage collector by aldy505 · Pull Request #653 · getsentry/objectstore

aldy505 · 2026-09-26T04:54:54Z

Introduces SqliteGarbageCollector and SqliteGarbageCollectorStream which implements ChangeStream.
This is required for filesystem & S3-compatible API backends.
Note that it's not wired to the backends yet.

Supersedes #582
Refs FS-482

Supersedes #582

cursor

Cursor Bugbot has reviewed your changes and found 5 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7dbb16c. Configure here.}

cursor · 2026-09-26T04:58:19Z

+
+            // Decrement counter when done
+            active_tasks.fetch_sub(1, Ordering::SeqCst);
+        });


Unordered tasks lose object updates

High Severity

write, update, and delete each tokio::spawn an independent task with no per-object ordering. A later update or delete can commit before the matching write, so expirations are skipped and deleted objects can be reinserted.

Additional Locations (2)

objectstore-service/src/change_stream/garbage_collector_sqlite.rs#L89-L108

objectstore-service/src/change_stream/garbage_collector_sqlite.rs#L118-L136

^{Reviewed by Cursor Bugbot for commit 7dbb16c. Configure here.}

we were actually talking about something related to this today. i don't think we can guarantee today that, if two requests are racing, the storage backend and the change stream will resolve the race in the same way. i signed off on that for COGS because it's not really a big issue there, but it is more meaningful for this GC use case

i don't have a great answer for what to do about that right now. just surfacing it here, i guess

I remember I talked about this with Jan and Lorenzo, AFAIK we didn't came up to a conclusion yet.

cursor · 2026-09-26T04:58:19Z

 #[cfg(feature = "storage-cogs")]
 mod cost_tracker;
 mod factory;
+mod garbage_collector_sqlite;


Collector is never constructed

Medium Severity

SqliteGarbageCollectorStream lives in a private module and is never re-exported or built by ChangeStreamFactory. No backend can select this collector, so the new feature is unreachable.

Additional Locations (1)

objectstore-service/src/change_stream/garbage_collector_sqlite.rs#L17-L33

^{Reviewed by Cursor Bugbot for commit 7dbb16c. Configure here.}

It's so hard to develop this on Windows. I need to grab my Linux laptop.

codecov · 2026-09-27T12:41:18Z

Codecov Report

❌ Patch coverage is 98.44720% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.36%. Comparing base (bf02738) to head (282c346).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
...vice/src/change_stream/garbage_collector_sqlite.rs	98.44%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #653      +/-   ##
==========================================
+ Coverage   91.27%   91.36%   +0.08%     
==========================================
  Files         116      117       +1     
  Lines       23496    24028     +532     
==========================================
+ Hits        21445    21952     +507     
- Misses       2051     2076      +25

Components	Coverage Δ
Rust Backend	`94.87% <98.44%> (+0.05%)`	⬆️
Rust Client	`81.70% <ø> (-0.26%)`	⬇️
Python Client	`93.75% <ø> (ø)`

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

linear-code · 2026-09-29T09:07:48Z

FS-482

matt-codecov

a couple questions:

do we have performance goals for self-hosted deployments? i have been burned by poor single-insert sqlite performance before. if we need high throughput, we may want to introduce batched queries, but it introduces some risk of inventory corruption if we crash without flushing batches
do we have scaling requirements for self-hosted deployments? i've seen a few different "distributed sqlite" projects (https://rqlite.io/ / https://docs.fly.io/litefs/how-it-works/) but generally consider sqlite to be node-local

matt-codecov · 2026-09-30T03:48:18Z

+
+            // Decrement counter when done
+            active_tasks.fetch_sub(1, Ordering::SeqCst);
+        });


we were actually talking about something related to this today. i don't think we can guarantee today that, if two requests are racing, the storage backend and the change stream will resolve the race in the same way. i signed off on that for COGS because it's not really a big issue there, but it is more meaningful for this GC use case

i don't have a great answer for what to do about that right now. just surfacing it here, i guess

aldy505 · 2026-09-30T04:20:49Z

do we have performance goals for self-hosted deployments? i have been burned by poor single-insert sqlite performance before. if we need high throughput, we may want to introduce batched queries, but it introduces some risk of inventory corruption if we crash without flushing batches

If they're using k8s (the unofficial helm chart), they want to use the postgres implementation of this garbage collector. In which I will implement after SQLite is correctly implemented. This is how Taskbroker is doing as well, they provide both options, with SQLite as default.

do we have scaling requirements for self-hosted deployments? i've seen a few different "distributed sqlite" projects (https://rqlite.io/ / https://docs.fly.io/litefs/how-it-works/) but generally consider sqlite to be node-local

I've contributed to rqlite before, but no thanks, let's not consider that approach.

lcian

The biggest point to resolve will be #653 (comment) but in the meantime we can get the rest of the things in order before we find a solution for that.

matt-codecov

#677 changed the ChangeStream trait to help write GC correctly. unfortunately you will need to rebase and tweak your PR to implement:

begin_write(): create the GC record before committing in the storage backend
begin_update(): lengthen the expiration in GC before committing in the storage backend
commit_delete(): delete the GC record after deleting the stored object

if begin_write() or begin_update() fail, it's not safe to allow the backend commit to happen. otherwise we can permanently leak objects or cause GC to delete them prematurely. so those functions have to .await rather than spawn and they have to surface failure to the caller

if commit_delete() fails it's kind of harmless. GC will just have a ghost record floating around until its expiry. if the ghost record has manual expiration then it'll float around forever, but that's... probably a small edge case.

feat(change-stream): SQLite-backed garbage collector

7dbb16c

Supersedes #582

aldy505 requested a review from a team as a code owner September 26, 2026 04:54

This was referenced Sep 26, 2026

feat(keeper): integrate filesystem backend with keeper trait #583

Closed

feat: keeper trait & sqlite-backed implementation #582

Closed

cursor Bot reviewed Sep 26, 2026

View reviewed changes

sentry Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread objectstore-service/src/change_stream/garbage_collector_sqlite.rs Outdated

Comment thread objectstore-service/src/change_stream/garbage_collector_sqlite.rs Outdated

Comment thread objectstore-service/src/change_stream/garbage_collector_sqlite.rs Outdated

aldy505 added 3 commits September 26, 2026 12:03

fix: build issues

296786a

It's so hard to develop this on Windows. I need to grab my Linux laptop.

fix(change-stream): handle null expires_at

e9f46ff

fix: more missing imports

d8207ac

sentry Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread objectstore-service/src/change_stream/garbage_collector_sqlite.rs Outdated

feat(change-stream): working implementation for sqlite garbage collector

20935a3

sentry Bot reviewed Sep 27, 2026

View reviewed changes

Comment thread objectstore-service/src/change_stream/garbage_collector_sqlite.rs Outdated

chore: make clippy happy

b2e54d4

matt-codecov reviewed Sep 30, 2026

View reviewed changes

lcian reviewed Sep 30, 2026

View reviewed changes

fix: code review comments

b1b0db6

sentry Bot reviewed Oct 1, 2026

View reviewed changes

Comment thread objectstore-service/src/change_stream/garbage_collector_sqlite.rs Outdated

aldy505 added 2 commits October 1, 2026 08:44

test(change-stream): updated tests for sqlite garbage collector

3a21138

feat(change-stream): take timeout into account for join operation

282c346

aldy505 requested review from lcian and matt-codecov October 1, 2026 01:46

matt-codecov reviewed Oct 9, 2026

View reviewed changes

Conversation

aldy505 commented Sep 26, 2026 • edited by lcian Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot Sep 26, 2026

Choose a reason for hiding this comment

Unordered tasks lose object updates

Uh oh!

matt-codecov Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

aldy505 Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot Sep 26, 2026

Choose a reason for hiding this comment

Collector is never constructed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

linear-code Bot commented Sep 29, 2026

Uh oh!

matt-codecov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

matt-codecov Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aldy505 commented Sep 30, 2026

Uh oh!

lcian left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

matt-codecov left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aldy505 commented Sep 26, 2026 •

edited by lcian

Loading

codecov Bot commented Sep 27, 2026 •

edited

Loading

lcian left a comment •

edited

Loading