Release 0.77.8 - #3790
Merged
Merged
Release 0.77.8#3790
Conversation
* allow null in webhook serializer * serializer and spec update * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* add reciept page inital implementation * remove hallucinated "reciept not found" page in favor of traditional 404, more concise and accurate comments * design updates * fix row flattening issue * copilot feedback * add auth gate * handle additional cases where order can be null in receiptMenuItem. * show refunds * accessibility changes * remove accidentally committed memory limit config change * fix back button * update mitxonline api package * fix typecheck
) * Batch the PostHog view-event loader instead of querying per event Measured against the existing per-event path: 100 new events cost 700 queries before and 7 after, and the batched count is constant rather than proportional to the batch. That ratio is the point -- on 2026-08-18 a full backlog pass outran the Redis visibility_timeout, and because the task is acks_late the broker handed the still-running message to another worker, repeatedly, until every slot in the default celery pool held a duplicate and fleet throughput fell from ~17,000 completions/hr to 4-7. Per batch the loader now issues one resource lookup, one existence check, one legacy-candidate fetch, at most one bulk_update, and one bulk_create. bulk_create(ignore_conflicts=True) emits ON CONFLICT DO NOTHING against the partial unique index on event_uuid. That is strictly better than the get_or_create it replaces: atomic, where get_or_create is a read-then-write race that #3714 had to wrap in a savepoint. Repeat runs are the norm here, not the exception -- the extract re-reads the newest S3 object every time, because its last_modified always postdates the events it holds. Four things this had to get right, none of them obvious from the diff: Resource ids are validated before the pk__in query, not after. The per-event path caught ValueError and skipped one event; a bad id inside pk__in raises for the whole batch. Uuids are normalised to uuid.UUID. The per-event path only ever passed them to the ORM, which coerced them; the batch path compares them in Python, where a str never equals the UUID the ORM returns. The legacy-candidate query is a cross product of ids and dates, not a set of exact pairs, so it returns rows no event in the batch asked for. Matching is done in Python, and each candidate row is claimed at most once -- the legacy tail can hold several rows for one (resource, event_date). bulk_update fails the whole batch on IntegrityError where the per-event savepoint failed a single event, which would make overlapping runs worse rather than better. On conflict it falls back to load_posthog_lrd_view_event, so the blast radius is no wider than today's. Verified safe: nothing listens for post_save on LearningResourceViewEvent (the only non-ETL references are admin.py, a management command and api.py), the partial unique index works with Django's untargeted ON CONFLICT DO NOTHING, and bulk_create runs field pre_save so TimestampedModel's auto_now_add populates. This fixes runtime, not the work set. last_event_time is still read once at task start and the newest S3 object is re-read regardless, so a run still covers everything since the watermark. Nor does it make overlapping runs safe -- a dedicated queue for this ETL, or a lock, is still worth doing. Refs mitodl/hq#12917 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011yGLhunNHrh4DLyujeH7QR * Normalise resource ids, and cover the batch adoption fallback Two findings from review, both real. A resource id that arrives as a string was silently dropped. PostHog properties are JSON, so learning_resource_id can be an int or a string; the per-event path handed it to filter(pk=...), which coerced it, but the batch path compared it against ids read back from the database, where "3235" != 3235. The event fell out of the batch with no warning and no row. Confirmed by removing the coercion and watching the new test report `assert set() == {111}`. Both identifiers are now coerced once, up front, into a _NormalizedEvent record carrying resource_id as int and event_uuid as uuid.UUID alongside the original event for the fallback path. That replaces the scattered _event_uuid() calls and keeps membership tests, legacy adoption and the returned id set in agreement on types. The batch-level IntegrityError fallback had no test. The existing concurrency test calls load_posthog_lrd_view_event directly, so it covers the per-event savepoint and not the branch that stops one adoption race losing a whole batch. Added a test that forces bulk_update to raise and asserts all three events survive via the per-event path. Both tests were mutation-checked. Note the first attempt at mutating the fallback hit the identically-indented `except IntegrityError` in load_posthog_lrd_view_event, which sits earlier in the file, and the test passed against it -- the branch has to be mutated by position, not by text, or the check is vacuous. Refs mitodl/hq#12917 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011yGLhunNHrh4DLyujeH7QR --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
OpenAPI Changes16 changes: 2 error, 0 warning, 14 info Unexpected changes? Ensure your branch is up-to-date with |
odlbot
added
waiting for checkboxes
all checkboxes checked
and removed
deploying to rc
waiting for checkboxes
all checkboxes checked
labels
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tobias Macey
Danielle Frappier
Carey P Gumaer
Shankar Ambady
Anastasia Beglova