fix(schedulers): re-check leadership in scheduler ticks (SC-3)

[TerrifiedBug]

SC-3 — de-SPOF schedulers

Problem

Background schedulers start based on leadership at boot (src/instrumentation.node.ts → startSingletonServices) but then run forever via setInterval/node-cron and never re-check leadership. When a leader is demoted (LeaderElection flips _isLeader=false after 3 failed Redis renewals), the old instance's timers keep firing for up to one TTL (~15s) while the newly-elected leader is already running — producing duplicate alerts, backups, anomaly runs, cost-analysis passes, delivery retries and telemetry heartbeats.

Change

Add an early leadership guard at the top of each scheduler's periodic work so a demoted instance becomes a no-op:

if (!isLeader()) {            // from @/server/services/leader-election
  debugLog("<tag>", "Skipping … — instance is no longer leader");
  return;
}

Guard is guard-only — timers/cron tasks are left registered, so each scheduler resumes cleanly if the instance re-acquires leadership (no teardown/re-init churn).

Guards added to:

• fleet-alert-service.ts — top of tick()
• anomaly-detection-job.ts — top of tick()
• lake/lake-alerts.ts — top of module-level tick()
• backup-scheduler.ts — top of the per-org cron.schedule callback
• cost-optimizer-scheduler.ts — top of the shared scheduleJob callback (covers both the daily and continuous cron jobs, which share that one callback)
• telemetry-scheduler.ts — top of the daily heartbeat cron callback
• retry-service.ts — top of processRetries() (also covers processOutboundRetries(), which it calls)

cert-expiry-checker.ts intentionally gets no guard: it has no own timer/cron and its only production caller is checkCertificateExpiry() inside the now-guarded fleet-alert tick (confirmed by searching all callers).

Tests

Extended the two critical scheduler specs (fleet-alert-service.test.ts, anomaly-detection-job.test.ts) with a leadership guard block that mocks @/server/services/leader-election + Prisma and asserts:

• isLeader() === false → tick is a no-op: no org scan (organization.findMany), no rule/pipeline evaluation, no channel delivery.
• isLeader() === true → tick proceeds and evaluates per-org as before.

Both specs default the mock to leader so the existing tick-driven tests keep doing work.

Verification

• pnpm exec vitest run on fleet-alert-service.test.ts + anomaly-detection-job.test.ts → 30 passed (incl. 4 new guard tests).
• Re-ran the 5 other touched schedulers' existing specs (lake-alerts, backup-scheduler-per-org, cost-optimizer-scheduler[-per-org], telemetry-scheduler, retry-service) → 38 passed, no regressions (test env has no REDIS_URL, so the real isLeader() returns true and the guards pass through).
• Filtered tsc --noEmit: no type errors in any changed file. The only diagnostic is the known local @clickhouse/client gap in lake/clickhouse.ts (package not installed locally) — ignored per task guidance.

Notes

• No Prisma migration required.
• License boundary respected: all new imports are @/… (src-internal); no @cloud/*.
• The lake/clickhouse path can't be exercised locally (no @clickhouse/client); the guard sits before any ClickHouse access and is covered structurally by the lake-alerts spec. Failover/demotion behaviour itself isn't locally reproducible without a multi-instance Redis setup — unit tests + types are the bar here.