diff --git a/.changeset/24910-declared-webhook-undeliverable.md b/.changeset/24910-declared-webhook-undeliverable.md new file mode 100644 index 00000000000..8061a18c6e1 --- /dev/null +++ b/.changeset/24910-declared-webhook-undeliverable.md @@ -0,0 +1,20 @@ +--- +'@objectstack/plugin-webhooks': patch +--- + +fix(plugin-webhooks): a declared webhook that the deployment can never deliver is reported once at boot, at `error`, with the consequence and the fix + +Clause-②: no + +The webhook auto-enqueuer reads record events only from the realtime service. A stack with `requires: ['webhooks']` and no `'realtime'` therefore never delivers a declared webhook. + +- **Before.** + - The boot logged one `WARN [webhook-auto-enqueuer] disabled — ObjectQL, Realtime, or Messaging service not available`. The line named no webhook, gave no consequence and gave no fix. + - Every record write still answered 200, the declaration still read active, and no delivery row was ever written. +- **After.** When the stack declares at least one webhook, the boot logs one `error` line instead of the warning. The line names each declared webhook and says none of them will be delivered while record writes keep succeeding. It names the missing service and the fix, for example "add 'realtime' to `requires`". A stack that declares no webhook gets the same `warn` as before. +- **Unchanged.** + - Every deployment whose auto-enqueuer runs. + - A host that constructs `WebhookOutboxPlugin` with `autoEnqueue: false` gets no new log line. It turned the enqueuer off itself. + - `os validate` and `os build`: they still accept a stack that declares webhooks without these tokens. + - The activation door (`POST /api/v1/security/_activation/webhook/NAME`): it answers exactly as before. +- **What to do.** A stack that declares webhooks needs `requires` to include both `'webhooks'` and `'realtime'`. diff --git a/content/docs/automation/webhooks.mdx b/content/docs/automation/webhooks.mdx index 5b642798588..095ef028770 100644 --- a/content/docs/automation/webhooks.mdx +++ b/content/docs/automation/webhooks.mdx @@ -100,6 +100,28 @@ A webhook reaches the dispatcher one of two ways: - **Authored by an organization** — a `sys_webhook` row (§3.1), switched by its own `active` field. +**The capabilities a declared webhook needs.** A stack that declares a webhook +must name two tokens in `requires`, beside whatever else it declares: + +```ts +requires: ['automation', 'triggers', 'webhooks', 'realtime'], +``` + +- `'webhooks'` mounts `plugin-webhooks`: the declared-webhook reader, the + auto-enqueuer and the redeliver door. Without it nothing reads the + declaration at all. +- `'realtime'` mounts the realtime service, which is the auto-enqueuer's only + source of record events: the engine publishes `data.record.*` through it, and + nothing else feeds the enqueuer (§4.1). With `'webhooks'` and without + `'realtime'` the declaration is read but never dispatched. Record writes still + succeed and no delivery is ever enqueued. The boot logs one `error` that names + each declared webhook and the missing token. Switching a declared webhook on + through the activation door delivers nothing on such a deployment either. + +The outbox itself comes from `messaging`, which every boot mounts unless it runs +with `--preset minimal`. A scaffolded app's `requires` is +`['automation', 'triggers']`, so it carries neither of the two tokens above. + On a database an 18.0 pre-release (`18.0.0-next.*`) created, the rows that release's boot seeder wrote for declared webhooks (`managed_by: 'package'`) are still there. Until the ownership migration removes them, a pristine one is ignored in favour of its diff --git a/packages/plugins/plugin-webhooks/src/auto-enqueuer-disabled-report.test.ts b/packages/plugins/plugin-webhooks/src/auto-enqueuer-disabled-report.test.ts new file mode 100644 index 00000000000..1b87f57ac62 --- /dev/null +++ b/packages/plugins/plugin-webhooks/src/auto-enqueuer-disabled-report.test.ts @@ -0,0 +1,242 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#24910] A kernel with `webhooks` and without `realtime` delivers no declared + * webhook — and now says so ONCE, at `error`, with the consequence and the fix. + * + * ## The defect + * + * The auto-enqueuer's only source of record events is the realtime service: the + * engine publishes `data.record.*` through it and nothing else feeds the + * enqueuer. Without it the plugin disabled the enqueuer with one boot `warn`, + * while every record write still answered 200, the declaration still said the + * webhook was active, and the activation door answered `enabled: true` for a + * switch that switched nothing. By AGENTS.md "Degradation log levels" that is + * the durability class: the system looks normal and a delivery it claims never + * lands. + * + * ## What is pinned, on a real kernel boot + * + * A real `LiteKernel` runs the plugin's real `kernel:ready`; the engine is a + * real `ObjectQL` over a real in-memory `SqlDriver`, the stack's webhook comes + * from `defineWebhook` through the engine's real manifest decomposition + * (`registerApp`), and the outbox is messaging's real `MessagingService`. Only + * the realtime service is a stand-in, and only on the legs that mount one. + * + * 1. Declared webhook, no realtime: one `error` naming the webhook, `realtime` + * and the fix; no `disabled` warn; and a record write enqueues nothing (the + * loss itself). + * 2. CONTROL, the same stack with realtime: no error, and the same write + * enqueues its deliveries — so leg 1's zero is the missing service and not a + * fixture that cannot deliver. + * 3. No declared webhook, no realtime: the `warn` it always was, no `error`. + * 4. `autoEnqueue: false`: the host's own choice — nothing reported. + */ + +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { LiteKernel } from '@objectstack/core'; +import type { Plugin, PluginContext } from '@objectstack/core'; +import { ObjectQL } from '@objectstack/objectql'; +import { SqlDriver } from '@objectstack/driver-sql'; +import { defineWebhook } from '@objectstack/spec/automation'; +import type { IRealtimeService, RealtimeEventHandler, RealtimeEventPayload } from '@objectstack/spec/contracts'; +import { MemoryHttpOutbox, MessagingService, type EnqueueHttpInput } from '@objectstack/service-messaging'; +import { DeclaredWebhookSource, declaredWebhookRefId } from './declared-webhooks.js'; +import { SysWebhook } from './sys-webhook.object.js'; +import { WebhookOutboxPlugin, type WebhookOutboxPluginOptions } from './webhook-outbox-plugin.js'; + +const OBJECT = 'my_app_note'; +const HOOK = 'my_app_note_changed'; +const silentLogger = { info() {}, warn() {}, error() {}, debug() {} } as any; + +/** The webhook the card's reproduction declares, as `defineWebhook` authors it. */ +const DECLARED = defineWebhook({ + name: HOOK, + object: OBJECT, + triggers: ['create', 'update', 'delete'], + url: 'http://127.0.0.1:3999/hook', + method: 'POST', + isActive: true, +}); + +class FakeRealtime implements IRealtimeService { + private subs = new Map(); + private n = 0; + async publish(event: RealtimeEventPayload): Promise { + for (const sub of this.subs.values()) { + if (sub.opts?.object && event.object !== sub.opts.object) continue; + await sub.handler(event); + } + } + async subscribe(_channel: string, handler: any, opts?: any): Promise { + const id = `s-${++this.n}`; + this.subs.set(id, { handler, opts }); + return id; + } + async unsubscribe(id: string): Promise { this.subs.delete(id); } +} + +/** messaging's real outbox, with every enqueue it receives recorded. */ +class RecordingOutbox extends MemoryHttpOutbox { + readonly enqueued: EnqueueHttpInput[] = []; + override async enqueue(input: EnqueueHttpInput): Promise { + this.enqueued.push(input); + return super.enqueue(input); + } +} + +interface BootOptions { + realtime?: boolean; + declare?: boolean; + plugin?: WebhookOutboxPluginOptions; +} + +/** + * Stands in for the plugin's declared dependency + * (`dependencies = ['com.objectstack.service.messaging']`) and registers what a + * served boot holds for `requires: ['webhooks']`: the engine, messaging, and + * realtime only where the case asks. + */ +class FixturePlugin implements Plugin { + name = 'com.objectstack.service.messaging'; + type = 'standard' as const; + version = '1.0.0'; + readonly outbox = new RecordingOutbox(); + readonly messaging = new MessagingService({ logger: silentLogger }); + + constructor( + private readonly engine: ObjectQL, + private readonly realtime: FakeRealtime | undefined, + ) { + this.messaging.setHttpOutbox(this.outbox); + } + + init(ctx: PluginContext): void { + ctx.registerService('manifest', { register: () => {}, list: () => [] }); + ctx.registerService('objectql', this.engine); + ctx.registerService('messaging', this.messaging); + if (this.realtime) ctx.registerService('realtime', this.realtime); + } +} + +/** The real plugin, with what its boot logs observed. */ +class ObservedWebhookPlugin extends WebhookOutboxPlugin { + readonly logs: Array<{ level: string; message: string; meta: unknown }> = []; + + override async init(ctx: PluginContext): Promise { + for (const level of ['debug', 'info', 'warn', 'error'] as const) { + vi.spyOn(ctx.logger, level).mockImplementation((message: unknown, ...rest: unknown[]) => { + this.logs.push({ level, message: String(message), meta: level === 'error' ? rest[1] : rest[0] }); + }); + } + await super.init(ctx); + } + + lines(level: string, fragment: string) { + return this.logs.filter((l) => l.level === level && l.message.includes(fragment)); + } +} + +const open: Array<() => Promise> = []; + +afterEach(async () => { + while (open.length) { + try { await open.pop()?.(); } catch { /* already stopped */ } + } + vi.restoreAllMocks(); +}); + +async function boot(options: BootOptions) { + const driver = new SqlDriver({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true }); + const engine = new ObjectQL(); + engine.registerDriver(driver, true); + await engine.init(); + engine.registerApp({ + id: 'my_app', + namespace: 'my_app', + version: '0.0.0', + objects: [{ + name: OBJECT, + label: 'Note', + // Owner sharing is not under test here — keep RLS out of the way. + sharingModel: 'public_read_write', + fields: { title: { type: 'text', label: 'Title' } }, + }], + ...(options.declare === false ? {} : { webhooks: [DECLARED] }), + }); + engine.registry.registerObject(SysWebhook as any, 'test'); + await engine.syncSchemas(); + const realtime = options.realtime ? new FakeRealtime() : undefined; + if (realtime) engine.setRealtimeService(realtime); + + const fixture = new FixturePlugin(engine, realtime); + const plugin = new ObservedWebhookPlugin(options.plugin); + const kernel = new LiteKernel({ logger: { level: 'silent' } as any }); + kernel.use(fixture); + kernel.use(plugin); + await kernel.bootstrap(); + open.push(async () => { await kernel.shutdown(); await driver.disconnect(); }); + + const reader = kernel.getService('declaredWebhooks'); + const enqueuerStarted = (() => { + try { return Boolean(kernel.getService('webhook.autoEnqueuer')); } catch { return false; } + })(); + /** Insert then update one record, and give fire-and-forget enqueues a moment to land. */ + const writeRecord = async () => { + const created: any = await engine.insert(OBJECT, { title: 'seed' }); + await engine.update(OBJECT, { title: 'edited' }, { where: { id: created.id } }); + for (let i = 0; i < 20 && fixture.outbox.enqueued.length < 2; i++) await new Promise((r) => setTimeout(r, 25)); + }; + return { plugin, reader, enqueuerStarted, writeRecord, outbox: fixture.outbox }; +} + +describe('[#24910] the auto-enqueuer disabled on a stack that declares webhooks', () => { + it('no realtime: ONE error naming the webhook, the missing service and the fix — and a record write enqueues nothing', async () => { + const { plugin, reader, enqueuerStarted, writeRecord, outbox } = await boot({}); + + const errors = plugin.lines('error', '[webhook-auto-enqueuer]'); + expect(errors).toHaveLength(1); + expect(errors[0].message).toContain(HOOK); + expect(errors[0].message).toContain("add 'realtime' to `requires`"); + expect(errors[0].meta).toMatchObject({ hasEngine: true, hasRealtime: false, hasMessaging: true, declared: [HOOK] }); + expect(plugin.lines('warn', 'auto-enqueuer] disabled')).toEqual([]); + + expect(enqueuerStarted).toBe(false); + expect(reader).toBeInstanceOf(DeclaredWebhookSource); + // The reader still answers the declaration — it is only dispatch that is missing. + expect((await reader.declaredWebhookNamed(HOOK))?.active).toBe(true); + + await writeRecord(); + expect(outbox.enqueued).toEqual([]); + }); + + it('CONTROL — the same stack with realtime: no error, and the same write is delivered', async () => { + const { plugin, enqueuerStarted, writeRecord, outbox } = await boot({ realtime: true }); + + expect(plugin.lines('error', '[webhook-auto-enqueuer]')).toEqual([]); + expect(enqueuerStarted).toBe(true); + + await writeRecord(); + expect(outbox.enqueued.map((d) => [d.refId, d.label])).toEqual([ + [declaredWebhookRefId(HOOK), 'data.record.created'], + [declaredWebhookRefId(HOOK), 'data.record.updated'], + ]); + }); + + it('no declared webhook, no realtime: the warn it always was, never the error', async () => { + const { plugin } = await boot({ declare: false }); + + expect(plugin.lines('error', '[webhook-auto-enqueuer]')).toEqual([]); + const warns = plugin.lines('warn', 'auto-enqueuer] disabled'); + expect(warns).toHaveLength(1); + expect(warns[0].meta).toMatchObject({ hasEngine: true, hasRealtime: false, hasMessaging: true }); + }); + + it('`autoEnqueue: false` is the host\'s own composition: nothing reported', async () => { + const { plugin, enqueuerStarted } = await boot({ plugin: { autoEnqueue: false } }); + + expect(plugin.lines('error', '[webhook-auto-enqueuer]')).toEqual([]); + expect(plugin.lines('warn', '[webhook-auto-enqueuer]')).toEqual([]); + expect(enqueuerStarted).toBe(false); + }); +}); diff --git a/packages/plugins/plugin-webhooks/src/declared-webhooks.ts b/packages/plugins/plugin-webhooks/src/declared-webhooks.ts index ec1e3a3ec0a..0b503f4a85f 100644 --- a/packages/plugins/plugin-webhooks/src/declared-webhooks.ts +++ b/packages/plugins/plugin-webhooks/src/declared-webhooks.ts @@ -159,6 +159,21 @@ export function readDeclaredWebhookItems(engine: unknown, metadataService: unkno } } +/** + * [#24910] The distinct `name`s of the raw declared items — what the boot + * report names when the auto-enqueuer cannot run, so nothing declared would be + * delivered. Read the way {@link readDeclaredWebhookItems} reads them; never + * throws. + */ +export function readDeclaredWebhookNames(engine: unknown, metadataService: unknown): string[] { + const names = new Set(); + for (const raw of readDeclaredWebhookItems(engine, metadataService)) { + const name = (raw as { name?: unknown })?.name; + if (typeof name === 'string' && name) names.add(name); + } + return [...names]; +} + /** * The reader the plugin binds: the declarations from * {@link readDeclaredWebhookItems}, the switch from the activation ledger. diff --git a/packages/plugins/plugin-webhooks/src/webhook-outbox-plugin.ts b/packages/plugins/plugin-webhooks/src/webhook-outbox-plugin.ts index 737ab9aa911..1ca4e826143 100644 --- a/packages/plugins/plugin-webhooks/src/webhook-outbox-plugin.ts +++ b/packages/plugins/plugin-webhooks/src/webhook-outbox-plugin.ts @@ -23,6 +23,7 @@ import { DeclaredWebhookSource, DECLARED_WEBHOOK_NAME_GUARD_PACKAGE, bindDeclaredWebhookNameGuard, + readDeclaredWebhookNames, } from './declared-webhooks.js'; import { migrateLegacyWebhookSecrets } from './migrate-webhook-secrets.js'; import { createWebhookRedeliverGuard } from './redeliver-guard.js'; @@ -121,6 +122,55 @@ function requestDoorContext(request: Request): RedeliverDoorContext { }; } +/** + * [#24910] Which of the auto-enqueuer's three prerequisites this kernel holds: + * the data engine (it reads the subscriptions), the realtime service (its ONLY + * source of record events — nothing else feeds it) and messaging (the outbox + * it enqueues onto). Read once, at `kernel:ready`, when every plugin's + * `init()` has registered its services, so an absence is final. + */ +interface AutoEnqueueGap { + readonly hasEngine: boolean; + readonly hasRealtime: boolean; + readonly hasMessaging: boolean; +} + +/** What the auto-enqueuer runs with on this kernel, decided once at `kernel:ready`. */ +type AutoEnqueueWiring = + /** The host turned it off itself (`autoEnqueue: false`) — deliberate, nothing to report. */ + | { readonly kind: 'off' } + | { + readonly kind: 'ready'; + readonly engine: IDataEngine; + readonly realtime: IRealtimeService; + readonly messaging: MessagingHttpSurface; + } + | { readonly kind: 'gap'; readonly gap: AutoEnqueueGap }; + +/** + * [#24910] Why the auto-enqueuer is not running, and the composition change + * that starts it, as one sentence: the reason and the fix the boot report owes + * (AGENTS.md "Degradation log levels"). + */ +function autoEnqueueGapReason(gap: AutoEnqueueGap): string { + const missing: string[] = []; + const fixes: string[] = []; + if (!gap.hasRealtime) { + missing.push('the realtime service (the only source of the record events it dispatches)'); + fixes.push("add 'realtime' to `requires`"); + } + if (!gap.hasMessaging) { + missing.push('the messaging service (the outbox it enqueues onto)'); + fixes.push("add 'messaging' to `requires` (a `--preset minimal` boot mounts none of the always-on capabilities)"); + } + if (!gap.hasEngine) { + missing.push('the ObjectQL data engine (it reads the subscriptions)'); + fixes.push('mount the ObjectQL data engine'); + } + return `the webhook auto-enqueuer is disabled because ${missing.join(' and ')} ` + + `${missing.length === 1 ? 'is' : 'are'} not available. Fix: ${fixes.join('; ')}, then restart.`; +} + export interface WebhookOutboxPluginOptions { /** * Auto-enqueue config. When enabled (default `true` if the realtime + data @@ -360,21 +410,70 @@ export class WebhookOutboxPlugin implements Plugin { } } - private async bootAutoEnqueue( + /** + * [#24910] What the auto-enqueuer runs with on this kernel: off by the + * host's own choice, ready with its three prerequisites, or missing some of + * them. Read at `kernel:ready` — every `init()` has registered its services + * by then, so a missing one is a fact of this composition, not a boot that + * has not finished. + */ + private resolveAutoEnqueueWiring( ctx: PluginContext, opt: boolean | AutoEnqueuerOptions, - ): Promise { - if (opt === false) return; + ): AutoEnqueueWiring { + if (opt === false) return { kind: 'off' }; const engine = this.tryGetService(ctx, ['objectql', 'data']); const realtime = this.tryGetService(ctx, ['realtime']); const messaging = this.getMessaging(ctx); - if (!engine || !realtime || !messaging) { - ctx.logger.warn?.( - '[webhook-auto-enqueuer] disabled — ObjectQL, Realtime, or Messaging service not available', - { hasEngine: !!engine, hasRealtime: !!realtime, hasMessaging: !!messaging }, + if (engine && realtime && messaging) return { kind: 'ready', engine, realtime, messaging }; + return { kind: 'gap', gap: { hasEngine: !!engine, hasRealtime: !!realtime, hasMessaging: !!messaging } }; + } + + /** + * [#24910] Say once that the auto-enqueuer is not running — at `error` when + * the stack DECLARES webhooks, `warn` when it declares none. + * + * By AGENTS.md's one question (Degradation log levels): with declarations + * present the system looks normal from outside — every record write answers + * 200, the declaration says the webhook is active — while not one delivery + * it claims is ever enqueued. That is the durability class, so the line + * owes the consequence and the fix, and says it here, once, before any + * write rather than once per lost event. With nothing declared the kernel + * is visibly smaller than it could be (no webhook to lose yet), which is the + * functional class: `warn`, as before. + */ + private reportAutoEnqueueGap(ctx: PluginContext, gap: AutoEnqueueGap): void { + const declared = readDeclaredWebhookNames( + this.tryGetService(ctx, ['objectql', 'data']), + this.tryGetService(ctx, ['metadata']), + ); + if (declared.length > 0) { + ctx.logger.error?.( + `[webhook-auto-enqueuer] the ${declared.length} webhook${declared.length === 1 ? '' : 's'} this stack ` + + `declares (${declared.join(', ')}) will NEVER be delivered: record writes still succeed and nothing ` + + `else reports the loss, but no delivery is ever enqueued — ${autoEnqueueGapReason(gap)}`, + undefined, + { ...gap, declared }, ); return; } + ctx.logger.warn?.( + '[webhook-auto-enqueuer] disabled — ObjectQL, Realtime, or Messaging service not available', + { ...gap }, + ); + } + + private async bootAutoEnqueue( + ctx: PluginContext, + opt: boolean | AutoEnqueuerOptions, + ): Promise { + const wiring = this.resolveAutoEnqueueWiring(ctx, opt); + if (wiring.kind === 'off') return; + if (wiring.kind === 'gap') { + this.reportAutoEnqueueGap(ctx, wiring.gap); + return; + } + const { engine, realtime, messaging } = wiring; if (!messaging.isHttpDeliveryReady()) { ctx.logger.warn?.( '[webhook-auto-enqueuer] messaging HTTP outbox not ready (no data engine / reliableDelivery off) — webhook deliveries will not be durable',