Skip to content

fix(schedule): register ScheduleModule.forRoot() exactly once - #605

Merged
tompscanlan merged 1 commit into
mainfrom
fix/single-schedule-module-only
Sep 5, 2026
Merged

tompscanlan merged 1 commit into
mainfrom
fix/single-schedule-module-only

Conversation

@tompscanlan

@tompscanlan tompscanlan commented Sep 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem

ScheduleModule.forRoot() was called in four separate modules (matrix, metrics, external-calendar, atproto-publisher). @nestjs/core does not dedupe dynamic modules by shape:

  • NestContainer builds a ByReferenceModuleOpaqueKeyFactory with keyGenerationStrategy: 'random' (injector/container.js:26-32)
  • that keys a dynamic module on the identity of the returned object literal (by-reference-module-opaque-key-factory.js:17-33, via compiler.js:14-16)
  • forRoot() returns a fresh literal on every call (@nestjs/schedule/schedule.module.js:26-38)

So four call sites produced four ScheduleModule instances, each with its own ScheduleExplorer, SchedulerOrchestrator and cronJobs map. Each explorer scans every provider in the app, so every @Cron method was mounted as four independent CronJobs and fired four times per schedule.

This affects all four crons, not just the metrics rollup: the atproto sync retry, the calendar periodic sync and the database pool collector were each running 4x too.

Measured impact in production

The per-tenant metrics rollup was the #1 execution-time consumer on the entire shared Postgres instance (all 14 databases), measured as a pg_stat_statements delta over one 5-minute tick:

calls/tick exec time/tick
metrics rollup (this bug) 16 3655 ms
pg_database_size (exporter) 952 2717 ms

The same rollup query runs in 17 ms standalone (EXPLAIN ANALYZE, 100% shared-buffer hits, proper index scans) against small tables. The 3655 ms is self-inflicted: it arrives as one burst of 16 concurrent queries at each 5-minute boundary, and against the instance's 300m CPU limit that burst needs ~15s of wall clock to drain, so everything queues behind it.

Attribution of the burst, by sampling pg_stat_activity at 0.05s across a boundary (calling pg_stat_clear_snapshot() each iteration, since the view is otherwise transaction-cached):

boundary 13:40:00 tenant_lsdfaopkljdfs tenant_oiupsdknasfdf
pod 10.109.0.91 4 4
pod 10.109.0.222 4 4

4 firings per pod = exactly the 4 forRoot() call sites, x 2 tenants x 2 replicas = 16.

Change

ScheduleModule.forRoot() is now called exactly once, in AppModule, and no other module imports ScheduleModule at all. forRoot() sets global: true and its ScheduleExplorer scans every provider in the container, so feature modules need no import for their @Cron methods to be discovered.

Note the second half of that, because the first attempt at this PR got it wrong and failed to boot: importing the bare ScheduleModule is not a harmless no-op. The class-level @Module metadata provides SchedulerMetadataAccessor and SchedulerOrchestrator, but SchedulerRegistry and ScheduleExplorer exist only in the forRoot() dynamic providers (schedule.module.js:26-38 vs :43-46). A bare import therefore creates a second ScheduleModule whose SchedulerOrchestrator cannot resolve its SchedulerRegistry dependency, and bootstrap dies with Nest can't resolve dependencies of the SchedulerOrchestrator (?). This also removes the pre-existing bare import in DatabaseMetricsModule, which was previously masked by the four forRoot() calls.

Expected result

16 -> 4 calls per tick (1 firing x 2 tenants x 2 replicas), i.e. the 4x duplication removed. Not 16 -> 1: the two tenants are legitimate work, and both replicas legitimately run the rollup because each exports its own process-local Prometheus gauges.

Verification

  • npx tsc --noEmit is byte-identical to the pre-change baseline (308 pre-existing errors, repo is already red on main for unrelated reasons; zero in any touched file).
  • src/metrics/metrics.service.spec.ts: 8/8 pass.
  • Framework claim checked independently against the @nestjs/core and @nestjs/schedule sources, and confirmed empirically with throwaway Nest integration tests: four sibling modules each calling forRoot() fired a @Cron 4x per tick; one forRoot() fired it 1x. A second throwaway test pinned the three registration shapes — forRoot + bare import boots, forRoot + no import boots and fires exactly once, and bare import with no forRoot anywhere reproduces the bootstrap failure above.
  • CI e2e-test boots the real app under docker compose, which is what caught the bare-import mistake (it passed on PR Remove a stray verification script from the repository root #604 at this same base commit and failed here at 3m30s during provider instantiation).
  • Reviewed in a second, clean-context pass; no blocking findings against this commit.

Post-deploy check

Snapshot pg_stat_statements, wait one boundary, diff. Expect the rollup to drop to 4 calls/tick and the 5-minute throttle spike (cgroup cpu.stat, baseline nr_throttled 22683 / nr_periods 276077 = 8.2%) to flatten.

Caution when verifying: pg_stat_statements is cluster-wide (join to pg_database on dbid), queryid is not unique, and on PostgreSQL 18 both tenants' rollups collapse into a single entry whose stored query text names only one tenant — so one entry is not one tenant.

Not included

Gating the rollup to a single replica was implemented and then held back on fix-hold/metrics-rollup-gate. Review found a blocking defect: Prometheus gauges are process-local and never expire, so electing one writer leaves the losing replica exporting an absent or frozen snapshot, and the dashboards reduce across pods with max by (tenant) — which would turn active_users_30d (a 30-day rolling window that must be able to fall) into a permanent high-water mark. That needs a gauge-reset on the losing path plus a dashboard audit before it can ship.

ScheduleModule.forRoot() was called in four separate modules (matrix,
metrics, external-calendar, atproto-publisher). @nestjs/core does not
dedupe dynamic modules by shape: NestContainer builds a
ByReferenceModuleOpaqueKeyFactory with keyGenerationStrategy 'random'
(injector/container.js:26-32), which keys a dynamic module on the
identity of the DynamicModule object literal
(by-reference-module-opaque-key-factory.js:17-33, via
compiler.js:14-16). Every forRoot() call returns a fresh object literal
(@nestjs/schedule schedule.module.js:26-38), so four call sites produced
four distinct ScheduleModule instances, each with its own
ScheduleExplorer, SchedulerOrchestrator and cronJobs map. Each explorer
scans every provider in the application, so each @Cron method was
mounted as four independent CronJob instances and fired four times per
schedule.

forRoot() sets global: true, so a single registration in AppModule
covers the whole app; feature modules now import the bare ScheduleModule
(matching the existing DatabaseMetricsModule convention) and MatrixModule
drops the import entirely because it declares no scheduled methods.

Affects all four crons, not just the metrics rollup: the atproto sync
retry, the calendar periodic sync and the database pool collector were
each running 4x as well.
@tompscanlan
tompscanlan force-pushed the fix/single-schedule-module-only branch from 5636eab to 0d0ce12 Compare September 5, 2026 13:53
@tompscanlan
tompscanlan merged commit b4212d2 into main Sep 5, 2026
4 checks passed
@tompscanlan
tompscanlan deleted the fix/single-schedule-module-only branch September 5, 2026 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant