Skip to content

feat(change-stream): Account for upload sessions and markers - #673

Draft
lcian wants to merge 11 commits into
mainfrom
lcian/feat/resumable-upload-accounting
Draft

lcian wants to merge 11 commits into
mainfrom
lcian/feat/resumable-upload-accounting

Conversation

@lcian

@lcian lcian commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Reports resumable upload-related changes to the ChangeStream.

  • Markers are reported by the HighVolumeBackend with standard WRITE/DELETE changes and their hardcoded 5 day TTLs.
  • In-progress LT upload sessions are reported with new WRITE_SESSION and DELETE_SESSION op_types, with the length requested by the user plus backend metadata bytes at upload creation and hardcoded 7 days TTL.

We cannot reuse WRITE/DELETE for in-progress LT uploads, as GCS actually doesn't bill storage costs for them, so the COGS consumers would incorrectly report extra costs.
We therefore need to update the Kafka schemas to receive this new op_type (getsentry/sentry-kafka-schemas#504) as well as the COGS query (https://github.com/getsentry/etl/pull/2936).
The above needs to be done before this change is deployed.

@matt-codecov please let me know what you think about this approach, alternatively we can go with one of these alternatives:

  • Reuse WRITE/DELETE for in-progress sessions, and avoid sending them through the ChangeStream only in GCS. We would only do that because we know this is being used for COGS, even though ChangeStream is a general interface, so that doesn't sound right.
  • In the ChangeStream, if the change is coming from GCS and it's a session, ignore it. This also doesn't sound right, because the COGS ChangeStream by design doesn't know or care which backend the change came from.

Closes FS-563

Track upload sessions and HighVolume markers separately from published objects.
Report advertised upload sizes until completion, cancellation, or accounting
expiration, preserving existing object identities and sampling.

Refs FS-563
@linear-code

linear-code Bot commented Oct 7, 2026

Copy link
Copy Markdown

FS-563

@lcian

This comment has been minimized.

@codecov

This comment has been minimized.

@lcian lcian changed the title feat(cogs): Account for resumable upload sessions feat(change-stream): Account for resumable upload sessions Oct 7, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

lcian added 2 commits October 8, 2026 15:31
Default upload-session accounting to enabled in CostTrackerStream and disable
it for GCS at construction. Preserve generic session lifecycle events and
existing inventory identities.
Leave storage-cost filtering to the downstream COGS consumer.

This reverts commit 0294b0d.
@lcian lcian changed the title feat(change-stream): Account for resumable upload sessions feat(change-stream): Account for resumable upload sessions and markers Oct 8, 2026
@lcian lcian changed the title feat(change-stream): Account for resumable upload sessions and markers feat(change-stream): Account for upload sessions and markers Oct 8, 2026
Emit WRITE_SESSION and DELETE_SESSION for upload estimates while keeping
stored upload markers on WRITE and DELETE. Preserve existing record identities
and add session-specific inventory tracker methods without changing callers.
@lcian

This comment has been minimized.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 250bed0. Configure here.

Comment thread objectstore-service/src/backend/gcs.rs
@lcian
lcian marked this pull request as ready for review October 8, 2026 18:02
@lcian
lcian requested a review from a team as a code owner October 8, 2026 18:02
@matt-codecov

Copy link
Copy Markdown
Contributor

We cannot reuse WRITE/DELETE for in-progress LT uploads, as GCS actually doesn't bill storage costs for them, so the COGS consumers would incorrectly report extra costs.

actually, isn't this solved by emitting a write on completion rather than session creation in the LT backend? and if there is a backend that does bill for pre-finalized objects, then they could emit writes earlier?

there is still the problem of enabling GC to clean up abandoned sessions. but i think that problem might go away with changes we're considering:

with those changes, backends don't have to keep pre-finalized objects in separate namespaces anymore because there won't be a tombstone pointing at the LT revision until it's finalized. so when a session is opened, begin_write() will create a sizeless record of it in the GC inventory with the short expiry, and when it's finalized commit_write() will update with the size and the finalized expiry

@lcian
lcian marked this pull request as draft October 9, 2026 11:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants