Migrate heartbeat storage to ClickHouse

[skyfallwastaken]

Summary of the problem

Heartbeat volume makes PostgreSQL scans increasingly expensive across dashboards, projects, leaderboards and administrative tools. Existing caches and rollups also duplicate state and make heartbeat lifecycle operations harder to recover safely.

Describe your changes

• Store canonical heartbeat payloads and identity aliases in ClickHouse 26.7.3.19, with separate user-first and time-first query layouts.
• Preserve exact deduplication, retries, individual deletion and restore, account transfer and JA4 nullification through versioned replacement state and durable PostgreSQL lifecycle controls. Whole-second and fractional Float64 values retain the historical payload hash across ClickHouse JSON reads.
• Reserve new heartbeat IDs in the alias index before canonical payload insertion. A retry after an ambiguous payload-insert timeout reconstructs the payload at that reserved ID rather than allocating a second canonical identity.
• Keep PostgreSQL as the default until an explicit fenced cutover. Store configuration is allowlisted, each direct or imported PostgreSQL insert takes a shared lock and rechecks the cutover row and a stale process cannot write after durable purge. Ingests remain concurrent while purge takes a conflicting exclusive lock.
• Bound ClickHouse work while PostgreSQL admission locks are held with per-request timeouts, a ten-second aggregate ingest deadline and 1,000-row import transactions. Historical writes remain split by destination month and repair is resumable in bounded canonical-store batches.
• Send the current PostgreSQL archived-project set to the cached homepage aggregate as ClickHouse external data, keeping SQL bounded even for large sets without creating duplicated permanent state.
• Apply all pending ClickHouse migrations before validating the final canonical schema, so upgrades from older migration histories work correctly.
• Add a dedicated GitHub ClickHouse job using an authenticated pinned server. It runs real API ingestion, repository integration, differential, concurrency and cutover/recovery task coverage. CLICKHOUSE_REQUIRED=1 makes gated suites fail rather than skip.
• Preserve API response fields and PostgreSQL query semantics. Expensive dashboard, profile, streak, Sailors Log and homepage reads retain bounded caches or deferred loading.
• Remove the production-derived benchmark and its public report.

Deployment contract

HEARTBEAT_STORE defaults to postgresql and accepts only postgresql or clickhouse. Production web and worker processes remain on PostgreSQL while ClickHouse is provisioned, migrated and backfilled. Mutations are fenced during online backfill, then all writes are fenced for the final pass and verification. Web and worker switch to clickhouse together while both fences remain closed.

Before PostgreSQL purge, rollback means switching reads back while writes remain fenced. Purge is the irreversible boundary because new ClickHouse heartbeats are not dual-written to PostgreSQL. Its final transaction locks the cutover row, rechecks the verified source boundary, records purged_at, truncates payloads and rollups and resets user rollup generations atomically. An interruption rolls every change back. The empty PostgreSQL tables remain in schema history until a later explicit Rails migration after the rollback period.

Account deletion applies ClickHouse tombstones before relational anonymisation completes. Account merge first commits a durable transfer control, then moves heartbeats asynchronously. Both source and surviving target reject writes while the transfer is pending. Source activity can remain globally visible and the surviving account can omit that activity until the retryable job completes.

heartbeat_store is the recovery source of truth. Delivery reconciliation repairs aliases and incomplete layout acknowledgements. Query-layout repair rebuilds both layouts from the canonical store but cannot recreate canonical payload absent from a backup. After a stale ClickHouse restore, recovery reseeds PostgreSQL allocators, replays retained account transfer, account deletion and JA4 controls, repairs layouts and verifies state before traffic reopens. Ordinary heartbeats plus individual heartbeat delete/restore actions newer than the restored backup remain subject to the documented RPO because those actions have no durable PostgreSQL lifecycle controls.

The supported server is exactly 26.7.3.19. Upgrades require an isolated backup restore, version-guard update, real ClickHouse coverage, production-sized load validation and a restore/rollback drill. The runbook also requires per-query and per-user limits, spill thresholds, disk and part alerts, off-node backups and recorded RPO/RTO before cutover.

The deploy job depends on test_clickhouse; repository branch protection must separately configure it as a required merge check. The retained MD5 fields hash is historical deduplication compatibility only, not cryptographic security or sensitive-data hashing.

Screenshots / Media

No public production-derived data is attached. The only visual change restores deferred dashboard loading skeletons.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[greptile-apps]

Greptile Summary

The PR migrates canonical heartbeat storage and querying from PostgreSQL to ClickHouse while retaining PostgreSQL-backed allocation, lifecycle controls, and a fenced cutover process.

• Adds canonical, alias, user-first, and time-first ClickHouse layouts with migration and repair tasks.
• Routes ingestion, statistics, lifecycle mutations, exports, and administrative queries through the heartbeat repository.
• Adds resumable backfill, verification, purge, transfer, deletion, and JA4-nullification workflows.
• Adds pinned ClickHouse CI coverage and updates dashboard/project loading behavior for the new query path.

Confidence Score: 5/5

The PR appears safe to merge because no eligible blocking failure remains in this follow-up review.

No blocking failure remains.

Important Files Changed

Filename	Overview
app/repositories/heartbeat_repository.rb	Introduces the central ClickHouse persistence, alias resolution, versioned lifecycle mutation, delivery, reconciliation, and query implementation.
app/services/heartbeat_ingest.rb	Routes direct and imported heartbeat ingestion through the selected store while enforcing PostgreSQL admission fencing and bounded ClickHouse operations.
app/models/heartbeat_cutover.rb	Defines the durable backfill, verification, locking, and irreversible PostgreSQL purge boundary.
lib/tasks/clickhouse.rake	Adds schema migration, resumable backfill, verification, repair, drain, and purge operational tasks.
app/jobs/heartbeat_transfer_job.rb	Implements retryable asynchronous transfer of canonical heartbeat ownership.
app/jobs/heartbeat_deletion_job.rb	Implements durable retry and reconciliation for account-wide heartbeat deletion.
db/clickhouse/012_create_heartbeat_store.sql	Defines the canonical versioned heartbeat store used as the recovery source of truth.
.github/workflows/ci.yml	Adds a pinned authenticated ClickHouse service and required integration, differential, concurrency, and recovery test job.

Sequence Diagram

sequenceDiagram
  participant Client
  participant Rails as Rails / HeartbeatIngest
  participant PG as PostgreSQL controls
  participant Store as ClickHouse canonical store
  participant Alias as ClickHouse aliases
  participant Layouts as Query layouts
  participant Repair as Delivery / repair jobs

  Client->>Rails: Submit heartbeat
  Rails->>PG: Acquire admission lock and allocate ID/version
  Rails->>Alias: Reserve canonical and legacy hashes
  Rails->>Store: Insert canonical payload
  Rails->>Layouts: Publish user-first and time-first rows
  Rails-->>Client: Return persisted heartbeat
  alt Partial write or ambiguous timeout
    Rails->>Repair: Enqueue reconciliation
    Repair->>Alias: Resolve reserved identity
    Repair->>Store: Restore canonical payload if missing
    Repair->>Layouts: Repair layout acknowledgements
  end

Loading

_{Reviews (17): Last reviewed commit: "Fix ClickHouse admin queue profile" | Re-trigger Greptile}