Repository navigation
fix(schedule): register ScheduleModule.forRoot() exactly once - #605
Merged
Merged
Conversation
ScheduleModule.forRoot() was called in four separate modules (matrix, metrics, external-calendar, atproto-publisher). @nestjs/core does not dedupe dynamic modules by shape: NestContainer builds a ByReferenceModuleOpaqueKeyFactory with keyGenerationStrategy 'random' (injector/container.js:26-32), which keys a dynamic module on the identity of the DynamicModule object literal (by-reference-module-opaque-key-factory.js:17-33, via compiler.js:14-16). Every forRoot() call returns a fresh object literal (@nestjs/schedule schedule.module.js:26-38), so four call sites produced four distinct ScheduleModule instances, each with its own ScheduleExplorer, SchedulerOrchestrator and cronJobs map. Each explorer scans every provider in the application, so each @Cron method was mounted as four independent CronJob instances and fired four times per schedule. forRoot() sets global: true, so a single registration in AppModule covers the whole app; feature modules now import the bare ScheduleModule (matching the existing DatabaseMetricsModule convention) and MatrixModule drops the import entirely because it declares no scheduled methods. Affects all four crons, not just the metrics rollup: the atproto sync retry, the calendar periodic sync and the database pool collector were each running 4x as well.
tompscanlan
force-pushed
the
fix/single-schedule-module-only
branch
from
September 5, 2026 13:53
5636eab to
0d0ce12
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ScheduleModule.forRoot()was called in four separate modules (matrix,metrics,external-calendar,atproto-publisher).@nestjs/coredoes not dedupe dynamic modules by shape:NestContainerbuilds aByReferenceModuleOpaqueKeyFactorywithkeyGenerationStrategy: 'random'(injector/container.js:26-32)by-reference-module-opaque-key-factory.js:17-33, viacompiler.js:14-16)forRoot()returns a fresh literal on every call (@nestjs/schedule/schedule.module.js:26-38)So four call sites produced four
ScheduleModuleinstances, each with its ownScheduleExplorer,SchedulerOrchestratorandcronJobsmap. Each explorer scans every provider in the app, so every@Cronmethod was mounted as four independentCronJobs and fired four times per schedule.This affects all four crons, not just the metrics rollup: the atproto sync retry, the calendar periodic sync and the database pool collector were each running 4x too.
Measured impact in production
The per-tenant metrics rollup was the #1 execution-time consumer on the entire shared Postgres instance (all 14 databases), measured as a
pg_stat_statementsdelta over one 5-minute tick:pg_database_size(exporter)The same rollup query runs in 17 ms standalone (
EXPLAIN ANALYZE, 100% shared-buffer hits, proper index scans) against small tables. The 3655 ms is self-inflicted: it arrives as one burst of 16 concurrent queries at each 5-minute boundary, and against the instance's 300m CPU limit that burst needs ~15s of wall clock to drain, so everything queues behind it.Attribution of the burst, by sampling
pg_stat_activityat 0.05s across a boundary (callingpg_stat_clear_snapshot()each iteration, since the view is otherwise transaction-cached):tenant_lsdfaopkljdfstenant_oiupsdknasfdf10.109.0.9110.109.0.2224 firings per pod = exactly the 4
forRoot()call sites, x 2 tenants x 2 replicas = 16.Change
ScheduleModule.forRoot()is now called exactly once, inAppModule, and no other module importsScheduleModuleat all.forRoot()setsglobal: trueand itsScheduleExplorerscans every provider in the container, so feature modules need no import for their@Cronmethods to be discovered.Note the second half of that, because the first attempt at this PR got it wrong and failed to boot: importing the bare
ScheduleModuleis not a harmless no-op. The class-level@Modulemetadata providesSchedulerMetadataAccessorandSchedulerOrchestrator, butSchedulerRegistryandScheduleExplorerexist only in theforRoot()dynamic providers (schedule.module.js:26-38vs:43-46). A bare import therefore creates a secondScheduleModulewhoseSchedulerOrchestratorcannot resolve itsSchedulerRegistrydependency, and bootstrap dies withNest can't resolve dependencies of the SchedulerOrchestrator (?). This also removes the pre-existing bare import inDatabaseMetricsModule, which was previously masked by the fourforRoot()calls.Expected result
16 -> 4 calls per tick (1 firing x 2 tenants x 2 replicas), i.e. the 4x duplication removed. Not 16 -> 1: the two tenants are legitimate work, and both replicas legitimately run the rollup because each exports its own process-local Prometheus gauges.
Verification
npx tsc --noEmitis byte-identical to the pre-change baseline (308 pre-existing errors, repo is already red onmainfor unrelated reasons; zero in any touched file).src/metrics/metrics.service.spec.ts: 8/8 pass.@nestjs/coreand@nestjs/schedulesources, and confirmed empirically with throwaway Nest integration tests: four sibling modules each callingforRoot()fired a@Cron4x per tick; oneforRoot()fired it 1x. A second throwaway test pinned the three registration shapes —forRoot+ bare import boots,forRoot+ no import boots and fires exactly once, and bare import with noforRootanywhere reproduces the bootstrap failure above.e2e-testboots the real app under docker compose, which is what caught the bare-import mistake (it passed on PR Remove a stray verification script from the repository root #604 at this same base commit and failed here at 3m30s during provider instantiation).Post-deploy check
Snapshot
pg_stat_statements, wait one boundary, diff. Expect the rollup to drop to 4 calls/tick and the 5-minute throttle spike (cgroup cpu.stat, baselinenr_throttled22683 /nr_periods276077 = 8.2%) to flatten.Caution when verifying:
pg_stat_statementsis cluster-wide (join topg_databaseondbid),queryidis not unique, and on PostgreSQL 18 both tenants' rollups collapse into a single entry whose storedquerytext names only one tenant — so one entry is not one tenant.Not included
Gating the rollup to a single replica was implemented and then held back on
fix-hold/metrics-rollup-gate. Review found a blocking defect: Prometheus gauges are process-local and never expire, so electing one writer leaves the losing replica exporting an absent or frozen snapshot, and the dashboards reduce across pods withmax by (tenant)— which would turnactive_users_30d(a 30-day rolling window that must be able to fall) into a permanent high-water mark. That needs a gauge-reset on the losing path plus a dashboard audit before it can ship.