Repository navigation
fix: atomic webhook/delivery repository writes, batched and Redis-side listing (#356–#359) - #441
Merged
ritaifeoluwa merged 1 commit intoOct 6, 2026
Conversation
- SmartDropLabs#356: webhookRepository.remove() commits the record DEL, both ZREMs (global + per-owner) and the per-event index SREMs in a single MULTI/EXEC instead of a three-step delete, so a crash can no longer leave an index pointing at a webhook findById() 404s on. - SmartDropLabs#357: webhookRepository.listAll() hydrates every record with one MGET (new cache.mget(), a batched sibling of cache.get() that keeps the same null-on-miss/parse-error contract) instead of one GET round trip per webhook; an empty index short-circuits without touching record keys. - SmartDropLabs#358: deliveryRepository.create() commits record write, index insert, index trim and index TTL as one MULTI/EXEC rather than four awaited round trips. - SmartDropLabs#359: deliveryRepository.listByWebhook() filters in Redis. A per-status index (webhook:<id>:deliveries:<status>) is maintained on create and on every status transition — inside the same transaction as the record write, scored by creation time so ordering matches the recency index — and the listing reads at most `limit` ids from the right index, then hydrates them with a single MGET. The unfiltered path also stops reading the whole 100-entry window to return a page. Also repairs four tests in test/deliveryRepository.test.js that were already failing on main: they predate the SmartDropLabs#411 webhook-existence check and the SmartDropLabs#371 cancelRetry existence guard, so they now seed their fixture webhook / delivery record first. Closes SmartDropLabs#356 Closes SmartDropLabs#357 Closes SmartDropLabs#358 Closes SmartDropLabs#359
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four repository-level fixes in
webhookRepositoryanddeliveryRepository:two crash-consistency bugs from multi-step Redis writes, and two listings
that scaled with the whole table instead of with the page being returned.
#356 —
webhookRepository.remove()is atomicremove()rancache.del+ZREM webhooks:ids+ZREM webhooks:owner:<ip>(and the
#369event-indexSREMs) as separate awaited round trips. Dyingbetween them left a phantom id: an index still listed a webhook that
findById()could no longer return, solist()/countByOwner()reported itwhile every read 404'd. All of it now goes out as one
MULTI/EXEC.#357 —
listAll()stops loading every webhook one round trip at a timeZREVRANGE 0 -1followed byPromise.all(ids.map(cache.get))cost a fullRedis round trip per webhook, so the internal fan-out read
(
routes/metrics.js) degraded linearly with the number of registeredwebhooks. Added
cache.mget(keys)— a batched sibling ofcache.get()thatissues a single
MGETand keeps the identical contract (values in keyorder,
nullfor misses and unparsable entries, one warning per corruptvalue) — and
listAll()now uses it, returning early when the index isempty.
#358 —
deliveryRepository.create()is atomiccache.set+ZADD+ZREMRANGEBYRANK+EXPIREwere four awaited roundtrips. A crash in between produced an index id pointing at no record (a
phantom
listByWebhook()silently dropped) or a record no listing couldever find. All four now commit in one
MULTI/EXEC; no prior state is readto produce any of the writes, so there is no read-modify-write to preserve.
#359 —
listByWebhook()filters in RedisPreviously it read up to
RECENT_DELIVERIES_LIMIT(100) ids, hydrated everyrecord and then discarded most of them with a JS
.filter()onstatus.Now:
webhook:<id>:deliveries:<status>, mirrors each id —created alongside the recency index on
create()(statuspending) andmoved on every status transition. This is the same pattern as the
per-event index introduced for
#369.MULTI/EXECthat writes the newrecord, so a status-filtered listing can never disagree with the stored
record. Scores stay the creation timestamp, so ordering within a status
index matches the unfiltered index.
and re-arms the 30-day TTL, so neither can grow unbounded or outlive its
records.
listByWebhook()reads at mostlimitids from whichever index the queryneeds, then hydrates that page with a single
MGET. Alimitof0reads nothing at all.
Behavioural notes
?status=listings only cover deliveries written from this changeforward (ids written earlier self-heal on their next status transition);
deliveries that were already terminal at deploy time are absent from a
status-filtered listing until their 30-day TTL lapses. The unfiltered
listing is unaffected.
limitindex entries rather thanfrom whichever of the newest 100 records happened to still be alive.
Testing
test/webhookRepository.test.js—remove()commits onetransaction and clears every index while leaving siblings intact;
listAll()hydrates N webhooks with exactly oneMGETand zeroGETs,skips expired index entries, still decrypts secrets.
test/deliveryRepository.test.js—create()commits record, bothindexes, trim and TTL in one transaction with no writes outside it;
status filter reads only the status index and hydrates only matching
records; status transitions move the id between indexes in the same
transaction and keep the creation-time score; unfiltered listing reads
only
limitids.test/cache.test.js—mget()ordering/miss/corruption/errorbehaviour.
test/helpers/cacheMock.jsgained anmgetat both the raw and thecache-wrapper layer.
test/deliveryRepository.test.jsthat were alreadyfailing on
main(they predate the#411webhook-existence check andthe
#371cancelRetryexistence guard) by seeding their fixturewebhook / delivery record.
npx eslint src test→ 0 errors. Full suite before vs. after: 108 → 104failures — the difference is exactly those 4 repaired tests, with no new
failures and 18 new tests (744 → 762).
Closes #356
Closes #357
Closes #358
Closes #359