Skip to content

fix: atomic webhook/delivery repository writes, batched and Redis-side listing (#356–#359) - #441

Merged
ritaifeoluwa merged 1 commit into
SmartDropLabs:mainfrom
markdavid000:fix/issues-356-357-358-359
Oct 6, 2026
Merged

ritaifeoluwa merged 1 commit into
SmartDropLabs:mainfrom
markdavid000:fix/issues-356-357-358-359

Conversation

@markdavid000

Copy link
Copy Markdown

Summary

Four repository-level fixes in webhookRepository and deliveryRepository:
two crash-consistency bugs from multi-step Redis writes, and two listings
that scaled with the whole table instead of with the page being returned.

#356 — webhookRepository.remove() is atomic

remove() ran cache.del + ZREM webhooks:ids + ZREM webhooks:owner:<ip>
(and the #369 event-index SREMs) as separate awaited round trips. Dying
between them left a phantom id: an index still listed a webhook that
findById() could no longer return, so list()/countByOwner() reported it
while every read 404'd. All of it now goes out as one MULTI/EXEC.

#357 — listAll() stops loading every webhook one round trip at a time

ZREVRANGE 0 -1 followed by Promise.all(ids.map(cache.get)) cost a full
Redis round trip per webhook, so the internal fan-out read
(routes/metrics.js) degraded linearly with the number of registered
webhooks. Added cache.mget(keys) — a batched sibling of cache.get() that
issues a single MGET and keeps the identical contract (values in key
order, null for misses and unparsable entries, one warning per corrupt
value) — and listAll() now uses it, returning early when the index is
empty.

#358 — deliveryRepository.create() is atomic

cache.set + ZADD + ZREMRANGEBYRANK + EXPIRE were four awaited round
trips. A crash in between produced an index id pointing at no record (a
phantom listByWebhook() silently dropped) or a record no listing could
ever find. All four now commit in one MULTI/EXEC; no prior state is read
to produce any of the writes, so there is no read-modify-write to preserve.

#359 — listByWebhook() filters in Redis

Previously it read up to RECENT_DELIVERIES_LIMIT (100) ids, hydrated every
record and then discarded most of them with a JS .filter() on status.
Now:

  • A per-status index, webhook:<id>:deliveries:<status>, mirrors each id —
    created alongside the recency index on create() (status pending) and
    moved on every status transition. This is the same pattern as the
    per-event index introduced for #369.
  • The move happens inside the same MULTI/EXEC that writes the new
    record
    , so a status-filtered listing can never disagree with the stored
    record. Scores stay the creation timestamp, so ordering within a status
    index matches the unfiltered index.
  • Both indexes go through one helper that trims to the 100 newest members
    and re-arms the 30-day TTL, so neither can grow unbounded or outlive its
    records.
  • listByWebhook() reads at most limit ids from whichever index the query
    needs, then hydrates that page with a single MGET. A limit of 0
    reads nothing at all.

Behavioural notes

  • ?status= listings only cover deliveries written from this change
    forward (ids written earlier self-heal on their next status transition);
    deliveries that were already terminal at deploy time are absent from a
    status-filtered listing until their 30-day TTL lapses. The unfiltered
    listing is unaffected.
  • A page is now filled from the newest limit index entries rather than
    from whichever of the newest 100 records happened to still be alive.

Testing

  • New: test/webhookRepository.test.js — remove() commits one
    transaction and clears every index while leaving siblings intact;
    listAll() hydrates N webhooks with exactly one MGET and zero GETs,
    skips expired index entries, still decrypts secrets.
  • New: test/deliveryRepository.test.js — create() commits record, both
    indexes, trim and TTL in one transaction with no writes outside it;
    status filter reads only the status index and hydrates only matching
    records; status transitions move the id between indexes in the same
    transaction and keep the creation-time score; unfiltered listing reads
    only limit ids.
  • New: test/cache.test.js — mget() ordering/miss/corruption/error
    behaviour.
  • test/helpers/cacheMock.js gained an mget at both the raw and the
    cache-wrapper layer.
  • Repaired 4 tests in test/deliveryRepository.test.js that were already
    failing on main (they predate the #411 webhook-existence check and
    the #371 cancelRetry existence guard) by seeding their fixture
    webhook / delivery record.

npx eslint src test → 0 errors. Full suite before vs. after: 108 → 104
failures — the difference is exactly those 4 repaired tests, with no new
failures and 18 new tests (744 → 762).

Closes #356
Closes #357
Closes #358
Closes #359

- SmartDropLabs#356: webhookRepository.remove() commits the record DEL, both ZREMs
  (global + per-owner) and the per-event index SREMs in a single
  MULTI/EXEC instead of a three-step delete, so a crash can no longer
  leave an index pointing at a webhook findById() 404s on.
- SmartDropLabs#357: webhookRepository.listAll() hydrates every record with one MGET
  (new cache.mget(), a batched sibling of cache.get() that keeps the same
  null-on-miss/parse-error contract) instead of one GET round trip per
  webhook; an empty index short-circuits without touching record keys.
- SmartDropLabs#358: deliveryRepository.create() commits record write, index insert,
  index trim and index TTL as one MULTI/EXEC rather than four awaited
  round trips.
- SmartDropLabs#359: deliveryRepository.listByWebhook() filters in Redis. A per-status
  index (webhook:<id>:deliveries:<status>) is maintained on create and on
  every status transition — inside the same transaction as the record
  write, scored by creation time so ordering matches the recency index —
  and the listing reads at most `limit` ids from the right index, then
  hydrates them with a single MGET. The unfiltered path also stops reading
  the whole 100-entry window to return a page.

Also repairs four tests in test/deliveryRepository.test.js that were
already failing on main: they predate the SmartDropLabs#411 webhook-existence check
and the SmartDropLabs#371 cancelRetry existence guard, so they now seed their fixture
webhook / delivery record first.

Closes SmartDropLabs#356
Closes SmartDropLabs#357
Closes SmartDropLabs#358
Closes SmartDropLabs#359
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants