Repository navigation
Conversation
Track upload sessions and HighVolume markers separately from published objects. Report advertised upload sizes until completion, cancellation, or accounting expiration, preserving existing object identities and sampling. Refs FS-563
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Default upload-session accounting to enabled in CostTrackerStream and disable it for GCS at construction. Preserve generic session lifecycle events and existing inventory identities.
Leave storage-cost filtering to the downstream COGS consumer. This reverts commit 0294b0d.
Emit WRITE_SESSION and DELETE_SESSION for upload estimates while keeping stored upload markers on WRITE and DELETE. Preserve existing record identities and add session-specific inventory tracker methods without changing callers.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 250bed0. Configure here.
actually, isn't this solved by emitting a there is still the problem of enabling GC to clean up abandoned sessions. but i think that problem might go away with changes we're considering:
with those changes, backends don't have to keep pre-finalized objects in separate namespaces anymore because there won't be a tombstone pointing at the LT revision until it's finalized. so when a session is opened, |
Reports resumable upload-related changes to the
ChangeStream.HighVolumeBackendwith standardWRITE/DELETEchanges and their hardcoded 5 day TTLs.WRITE_SESSIONandDELETE_SESSIONop_types, with the length requested by the user plus backend metadata bytes at upload creation and hardcoded 7 days TTL.We cannot reuse
WRITE/DELETEfor in-progress LT uploads, as GCS actually doesn't bill storage costs for them, so the COGS consumers would incorrectly report extra costs.We therefore need to update the Kafka schemas to receive this new
op_type(getsentry/sentry-kafka-schemas#504) as well as the COGS query (https://github.com/getsentry/etl/pull/2936).The above needs to be done before this change is deployed.
@matt-codecov please let me know what you think about this approach, alternatively we can go with one of these alternatives:
WRITE/DELETEfor in-progress sessions, and avoid sending them through theChangeStreamonly in GCS. We would only do that because we know this is being used for COGS, even thoughChangeStreamis a general interface, so that doesn't sound right.ChangeStream, if the change is coming from GCS and it's a session, ignore it. This also doesn't sound right, because the COGS ChangeStream by design doesn't know or care which backend the change came from.Closes FS-563