Repository navigation
Commit 3fbef41
fix(plugin-webhooks): report a webhook delivery the outbox could not record at error, once per webhook (#24862)
Part of #24835
Clause-②: no
#24835 remains open after this PR: it carries the log-level half only.
The delivery loss itself has a measured root cause in
`packages/drivers/driver-sql` (with the event built in
`packages/objectql`), which is `domain:engine`'s code, so it is reported
on the card as a fork and not patched here.
## What this PR changes
When the `sys_http_delivery` write for a webhook event failed,
`AutoEnqueuer.reportWriteFailure` logged one bare `WARN
[webhook-auto-enqueuer] enqueue failed` per event. Under AGENTS.md
"Degradation log levels" that is a durability loss, not a functional
one. The record write that produced the event already answered 200, the
webhook still reads active, and no delivery row exists, so the event is
never sent, retried or redeliverable, and nothing else looks wrong.
- The first lost delivery for a webhook is now logged at `error`. The
line names the webhook, says the event is lost and why nothing else
shows it, and says where the cause is (the record's meta) and what to
restore.
- It is said once per webhook (`writeFailing`, the enqueue twin of
`droppedForSecret`). Each further lost event for that webhook is kept at
`debug` with its `eventId`.
- The next delivery recorded for that webhook clears the mark, so a
later outage is loud again. Both enqueue call sites now route the
success outcome too (`.then(recordedWrite, reportWriteFailure)`).
- A logger without `error` still gets the line at `warn` (the file's
existing fallback). The parked-subscription branch is unchanged.
- No edit falls inside the mirror and declared-name region (the region
the merged step-7c change rewrote). All hunks start at line 680 or
later.
## Measured root cause of the PostgreSQL loss (not changed here)
I reproduced it with the real `ObjectQL` engine and the real `SqlDriver`
on a throwaway local PostgreSQL 16.15 cluster, and in-memory SQLite. The
real `AutoEnqueuer` was wired the production way to the real
`MessagingService` and `SqlHttpOutbox`. I ran one insert and one by-id
update, then read `sys_http_delivery`:
| run | rows | non-plain value in the `data.record.updated` payload |
log |
|---|---|---|---|
| PG, base `41646a5788` | 1 (created only) |
`payload.changes.updated_at` is a knex `Raw` | `WARN … enqueue failed …
Converting circular structure to JSON … 'Timeout'` |
| SQLite, base | 2 | none | none |
| PG, this branch at `9a8a7f5103`, 2 updates | 1 | same `Raw` | one
`ERROR` naming the webhook, then one `DEBUG` for the second loss |
| PG at `9a8a7f5103`, driver handed a copy of the patch, 2 updates | 3 |
none | none |
The chain, at `41646a5788`:
1. `ObjectQL.update` passes `hookContext.input.data` to `driver.update`
(`packages/objectql/src/engine.ts:18094`). It then publishes the same
object as `DataEvent.changes` (`:18530`).
2. `SqlDriver.update` builds `formatted` from `formatInput`
(`packages/drivers/driver-sql/src/sql-driver.ts:8767`). For a patch that
needs no coercion, `formatInput` returns the caller's object itself
(`:20336`, `:20526`), and so does `applyWriteColumnMap` for a managed
object. Then `formatted.updated_at = this.updatedAtStamp()` (`:8778`)
writes into the caller's patch.
3. On PostgreSQL and MySQL, `updatedAtStamp()` is `knex.fn.now()`
(`:8579`), a `Raw` that holds the knex client, its pool and a Node
`Timeout`. On SQLite it is an ISO string (`:8578`). That string is just
as foreign to the caller's patch, but it serialises.
4. `projectRecordBodies` copies `changes` shallowly, and
`sql-http-outbox.ts:192` `JSON.stringify` throws on it.
`rotatedUpdateById` has the same in-place stamp (`:10207`). `updateMany`
already builds a fresh object instead (`:10152`). The fork, the
candidate fix sites and the evidence are on the card.
## Not added: a serialisation guard in `sql-http-outbox.ts`
Reading of the dispatch's decision frame: it does not belong beside the
root-cause fix.
- **Real business need.** The only measured producer of a non-plain
value is the driver stamp. Once that is fixed upstream, a guard has no
measured pull.
- **Long-term soundness.** Fixing the producer is the contract-first
route. A guard is a second place that knows about payload shape.
- **Preventing AI mistakes.** The outbox already refuses: it throws, and
no row is written. The `error` line in this PR makes that refusal loud,
with its cause in the meta.
- **Startup focus.** A new guard is a new surface with no pull.
## Tests
Run on head `377a12915e` (after merging `origin/main`, which brought the
step-7c rewrite of the same file):
- `pnpm --filter @objectstack/plugin-webhooks exec vitest run
--maxWorkers=2`: `Test Files 21 passed (21)`, `Tests 226 passed (226)`.
- `pnpm --filter @objectstack/plugin-webhooks typecheck`: exit 0 (`tsc
--noEmit`, `tsconfig.scripts.json`, `check:test-typecheck: OK`).
- `pnpm exec turbo run build --filter=@objectstack/plugin-webhooks...`:
`17 successful, 17 total`.
- New pins, in `src/auto-enqueuer.test.ts` under "a delivery that could
not be recorded is reported at error":
- the first loss is one `error` naming the webhook and
`sys_http_delivery`, with the `eventId` and the cause in meta, and later
losses go to `debug`, never `warn`;
- two failing webhooks each get their own `error`;
- a recorded delivery in between makes the next loss loud again;
- the bulk path reports through the same register.
They assert level, count and the named subjects, not prose.
## Ablation (one leg, the changed behaviour)
`node scripts/ablation-replace.mjs` replaced
`this.writeFailing.add(sub.id);` with the old bare `warn` and `return`.
The anchor went from 1 hit to 0, and the blob from `81ae0a56ccfa` to
`17c85086162a`. Then `vitest run src/auto-enqueuer.test.ts` gave `Tests
4 failed | 40 passed (44)`: exactly the four new pins. The restore was
`git checkout HEAD`. The restored blob is `81ae0a56ccfa`, equal to the
HEAD blob, and `git diff HEAD` is empty. This ran on head `377a12915e`;
the same leg on the pre-merge head `9a8a7f5103` read the same 4 failed
and 40 passed. The subject is imported from source
(`./auto-enqueuer.js`), so no `dist/` was involved.
## Gates
The local subset derived by `node scripts/pm/dispatch-gates.mjs
--commands --repo objectstack-ai/objectstack` at `377a12915e` has 70
families. Each exit was captured before any pipe.
- **67 families exited 0.** Among them: `check:engine-double-contract`,
`check:nul-bytes`, `check:doc-authoring`, `check:issue-citations`,
`check:cross-package-test-inputs`, `check:test-source-alias`,
`check:type-check-coverage`, `check:published-files`,
`check:dts-closure`, `check:lean-entry-closure`,
`check:adr-0087-registration --base origin/main`,
`check:changeset-no-major` and `check:empty-changeset`.
- **2 families are NOT MEASURED: exit 3, PREREQUISITE NOT MET.**
`check:dual-build-cjs-loads` needs every package's `dist/`. `check:i18n`
needs the CLI build closure. Neither reads anything this diff changes.
CI's `Lint & Repo Gates` runs both.
- **1 family is a declared narrowing, not run locally.**
`check:type-check-debt` (`--re-measure`) rebuilds the whole monorepo
under the shared lock. CI's `Lint & Repo Gates` runs it.
`--ran` reconciliation: `70 derived, 67 run, 2 NOT-MEASURED, 1 UNRUN`
(the UNRUN is the declared `check:type-check-debt`). The full-battery
`Temporal Conformance (live PG + MySQL)` job does not run on a PR. CI
check-runs were not concluded when this body was written; the report on
#24835 cites them.
## Acceptance notes
- The card's other half, the lost PostgreSQL deliveries, is unchanged by
this PR and is routed as a fork on #24835.
- The same in-place stamp reaches every other reader of the update
event's `changes` and of `hookContext.input.data` after the driver call.
On PostgreSQL and MySQL that is a knex `Raw`; on SQLite it is a
driver-stamped `updated_at` the caller never wrote. I measured only the
webhook reader here.
- The parked-subscription branch still logs every lost record at
`error`, with no say-once. I left it as it was. It is outside this
card's defect.
- `pnpm check:durability-log-level` cannot see this seam, because it
judges `try`/`catch` blocks and this is a promise rejection handler. I
did not extend it (new gates default to no).
---
_Generated by [Claude
Code](https://claude.ai/code/session_011TdZ8GjixNZFLz1xJD9zzK)_
Co-authored-by: Claude <noreply@anthropic.com>1 parent 4cef6e2 commit 3fbef41
3 files changed
Lines changed: 187 additions & 8 deletions
File tree
- .changeset
- packages/plugins/plugin-webhooks/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
592 | 592 | | |
593 | 593 | | |
594 | 594 | | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
595 | 709 | | |
596 | 710 | | |
597 | 711 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
677 | 677 | | |
678 | 678 | | |
679 | 679 | | |
680 | | - | |
681 | | - | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
682 | 695 | | |
683 | | - | |
684 | | - | |
685 | | - | |
| 696 | + | |
686 | 697 | | |
687 | 698 | | |
688 | 699 | | |
| |||
699 | 710 | | |
700 | 711 | | |
701 | 712 | | |
702 | | - | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
703 | 737 | | |
704 | 738 | | |
705 | 739 | | |
| |||
718 | 752 | | |
719 | 753 | | |
720 | 754 | | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
721 | 769 | | |
722 | 770 | | |
723 | 771 | | |
| |||
1195 | 1243 | | |
1196 | 1244 | | |
1197 | 1245 | | |
1198 | | - | |
| 1246 | + | |
| 1247 | + | |
| 1248 | + | |
| 1249 | + | |
1199 | 1250 | | |
1200 | 1251 | | |
1201 | 1252 | | |
| |||
1333 | 1384 | | |
1334 | 1385 | | |
1335 | 1386 | | |
1336 | | - | |
| 1387 | + | |
| 1388 | + | |
| 1389 | + | |
| 1390 | + | |
1337 | 1391 | | |
1338 | 1392 | | |
1339 | 1393 | | |
| |||
0 commit comments