Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
193 lines (156 loc) · 9.98 KB

ARCHITECTURE.md

File metadata and controls

193 lines (156 loc) · 9.98 KB
Raw

Architecture

gdatum is a single Go binary that performs three concurrent jobs:

  1. Collect — a background worker periodically scrapes player-count data from external master-server APIs.
  2. Store — collected snapshots are inserted into ClickHouse, which handles per-table aggregation via materialized views.
  3. Serve — a public HTTP server exposes a REST API and an embedded web UI; a separate admin HTTP server exposes ops endpoints.
                ┌────────────────────────────────────┐
                │       External master servers      │
                │  (rage.mp / majestic-files.com)    │
                └────────────────┬───────────────────┘
                                 │ HTTP (hourly)
                                 ▼
   ┌────────────────────────────────────────────────────┐
   │                    gdatum                          │
   │                                                    │
   │  ┌─────────┐  ┌─────────────────────────────────┐  │
   │  │ worker  │─▶│ ClickHouse                      │  │
   │  └─────────┘  │  servers_metrics_raw (Null)     │  │
   │               │      ├─▶ servers_online_mv      │  │
   │               │      │      └─▶ servers_online  │  │
   │               │      └─▶ servers_info_mv        │  │
   │               │             └─▶ servers_info    │  │
   │               └─────────────────────────────────┘  │
   │                                                    │
   │  ┌───────────────────────────┐  ┌───────────────┐  │
   │  │ Public HTTP   (:8080)     │  │ Admin (:8081) │  │
   │  │  /        → Web UI        │  │  /health      │  │
   │  │  /api/*   → REST          │  │  /metrics     │  │
   │  │  /openapi.yaml            │  │  /debug/pprof │  │
   │  │  /docs    → OpenAPI UI    │  │               │  │
   │  └───────────────────────────┘  └───────────────┘  │
   └────────────────────────────────────────────────────┘

The three subsystems are started as goroutines inside an errgroup in cmd/app/main.go; shutting any of them down (via SIGINT or a fatal error) cancels the rest.

Collector

Defined in internal/collector. Each tick of the worker fans out to a fixed set of platform collectors, merges their results, and inserts them into the repository with exponential backoff.

Adding a new platform means: a new client, an adapter, a new domain.Multiplayer constant, and an entry in the Handler.collectors slice.

Storage model

All schema changes live in migrations/ and are managed by goose. Migrations are embedded into the binary via //go:embed migrations and applied by running gdatum --migrate.

Table Engine Purpose
servers_metrics_raw Null Ingest endpoint. Discards rows after MVs fan out.
servers_online_mv MaterializedView Projects raw rows into servers_online.
servers_online MergeTree Time-series of player counts per (host, mp).
servers_info_mv MaterializedView Projects raw rows into servers_info.
servers_info ReplacingMergeTree Current metadata per (host, mp), latest-wins.

Both MergeTree tables partition by toYYYYMM(collected_at). Online counts use ClickHouse codecs (T64+ZSTD for the integer, DoubleDelta+ZSTD for the timestamp); metadata is plain.

Reads are served by internal/infrastructure/repository/clickhouse, which is wrapped by an adapter in internal/adapters/clickhouse that satisfies the domain.Repository interface.

HTTP layer

Two independent HTTP servers run side-by-side, each on its own listen address.

Public server

Mounted in cmd/app/main.go:

Path Handler Notes
/ webui.FrontendHandler Serves the embedded Next.js export.
/api/* pkg/api (ogen) → internal/handlers/api REST API. See api/spec.yaml.
/openapi.yaml webui.OpenAPIHandler Raw OpenAPI 3.0 spec.
/docs webui.DocsHandler Rendered API reference.

The REST stub in pkg/api/ is generated by ogen — the source of truth is api/spec.yaml. The hand-written business handlers in internal/handlers/api implement the generated api.Handler interface.

Admin server

Defined in internal/handlers/admin:

Path Handler Notes
GET /health inline {"status":"ok"}
GET /metrics promhttp.Handler() Prometheus scrape.
/debug/pprof/... net/http/pprof Profiles + symbols.

The admin server is bound to 127.0.0.1 by default — it must not be exposed publicly.

Web UI

The frontend is a Next.js 16 static export living in web/. At build time, hack/build-frontend.sh produces web/out/, which the Go binary embeds via //go:embed all:web/out in embed.go.

The frontend is independent of the Go backend at build time and only depends on the contract in api/spec.yaml.

Observability

  • Logs: structured JSON via zap, set as the global logger in main.
  • Metrics: internal/metrics defines collector counters (RecordServersCollected, RecordCollectionError, RecordInsertError). The standard Prometheus Go client also exports process and Go runtime metrics.
  • Tracing: OpenTelemetry SDK is a direct dependency and ready to be wired in; no exporter is configured by default.
  • Profiles: all default net/http/pprof endpoints are exposed on the admin server.

Code layout

cmd/app/                       Application entry point
api/spec.yaml                  OpenAPI 3.0 contract (source of truth)
pkg/api/                       ogen-generated server stub (do not edit)
internal/
  collector/                   Hourly scraper orchestration
  config/                      Env-based configuration
  domain/                      Core types, repository interface, validation
  adapters/                    Platform → domain mappers
    ragemp/  magestic/         Per-platform adapters
    clickhouse/                Repository decorator over ClickHouse
  infrastructure/
    clients/{ragemp,magestic}/ HTTP clients for master servers
    repository/clickhouse/     ClickHouse SQL repository
    server/                    HTTP server with graceful shutdown
    worker/                    Periodic worker loop
  handlers/{api,admin}/        HTTP handlers (REST + ops)
  metrics/                     Prometheus metric definitions
  webui/                       Static frontend and OpenAPI handlers
  utils/                       Cross-cutting helpers (backoff, sql, migrations…)
migrations/                    Goose-managed ClickHouse migrations
hack/                          Release / build scripts
web/                           Next.js 16 frontend

The package boundary follows a light hexagonal split: domain holds the core types and the Repository interface; adapters and infrastructure provide implementations; handlers are the inbound adapters; cmd/app wires everything together.