test(longhaul): add long-haul test driver core

[WentingWu666666]

Part 2 of 5: long-haul test driver core

Part 2 of the #348 split (design doc landed in #400, shared test module in #401).

What this PR adds

A new test/longhaul Go module — a long-running canary driver that exercises a persistent DocumentDB cluster with continuous workload, scheduled disruptive operations, health monitoring, and reporting. See the design in docs/designs/long-haul-test-design.md (merged via #400) for the why.

Packages:

Path	Role
cmd/longhaul	Driver entrypoint; wires workload + ops + monitor + reporting
config	Env-driven configuration with validation + unit tests
workload	Writer / verifier / atomic metrics (acked-write durability oracle)
operations	Weighted-random scheduler + scale/upgrade ops
monitor	Cluster client (consumes test/shared), health gating, leak detector
journal	Thread-safe event log + per-operation outage-policy oracle
report	Markdown report, periodic checkpoint to ConfigMap, GitHub Actions alerts
deploy/setup.yaml	Bootstrap manifest (namespace + Secret + DocumentDB CR) for canary cluster
Dockerfile, README.md	Container image + operator-facing documentation

Code reuse with e2e

Per the design doc (and #401), the driver consumes:

• test/shared/documentdb — Get, PatchSpec, PatchInstances, WaitHealthy, Delete, ReadyStatus (no duplicated CR logic; no hardcoded readiness strings)
• test/shared/mongo — NewClient, BuildURI, Ping, Seed, Count, TLS plumbing (no raw mongo.Connect)

go.mod uses replace for both modules so changes to shared helpers flow into the driver immediately.

Excluded from this PR (follow-ups)

• PR-3: CI/CD workflows (.github/workflows/longhaul-*.yaml) + deploy/deployment.yaml + deploy/rbac.yaml
• PR-4: auto-upgrade logic for operator + DocumentDB version drift

Verification

cd test/longhaul
go build ./...   # clean
go test ./...    # 6/6 packages pass: config / journal / monitor / operations / report / workload

Reviewer notes

• The whole driver is dormant until feat: long-haul (canary) test driver for the operator (#220) #348's CI workflows land in PR-3 — this PR is "the binary exists and its unit tests pass". No CI runs the driver yet.
• monitor/k8sclient.go is the typed controller-runtime wrapper that addresses xgerman's feat: long-haul (canary) test driver for the operator (#220) #348 review feedback (dropped raw dynamic.Interface + hardcoded GVR).
• operations/upgrade.go reads the desired DocumentDB version from a longhaul-versions ConfigMap that PR-4's workflow will populate; today it no-ops if the ConfigMap is missing.

[Copilot]

Pull request overview

Adds the core of the new test/longhaul Go module: a long-running “canary” driver that applies continuous workload, schedules disruptive operations (scale/upgrade), monitors cluster health/resources, journals events/windows, and produces checkpoint + markdown reporting.

Changes:

• Introduces workload writer/verifier with atomic metrics and uniqueness/indexing for durability checks.
• Adds the disruption scheduler and operations (scale up/down, DocumentDB version upgrade) with outage-policy tracking via a journal.
• Adds Kubernetes monitoring client (typed controller-runtime + test/shared), leak detection, and reporting/checkpoint + GitHub Actions annotations.

Reviewed changes

Copilot reviewed 41 out of 42 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
test/longhaul/cmd/longhaul/main.go	Driver wiring: config, mongo, monitors, workload, scheduler, checkpoint reporting
test/longhaul/config/config.go	Env-driven configuration defaults + validation
test/longhaul/config/config_test.go	Unit tests for env parsing/validation/enabled flag
test/longhaul/config/suite_test.go	Ginkgo suite bootstrap for config package
test/longhaul/workload/metrics.go	Atomic counters + snapshot helpers for workload metrics
test/longhaul/workload/metrics_test.go	Unit tests for metrics snapshot/rates/concurrency safety
test/longhaul/workload/writer.go	Majority-write workload writer + index creation
test/longhaul/workload/writer_test.go	Unit tests for checksum and writer counter behavior
test/longhaul/workload/verifier.go	Majority-read verifier scanning per-writer sequences for gaps/checksum mismatches
test/longhaul/workload/verifier_test.go	Constructor/resume-point tests (with integration boundary noted)
test/longhaul/workload/suite_test.go	Ginkgo suite bootstrap for workload package
test/longhaul/operations/scheduler.go	Weighted operation scheduler with cooldown + steady-state gating
test/longhaul/operations/scheduler_test.go	Unit tests for selection, disruption windows, and error journaling
test/longhaul/operations/scale.go	Scale up/down operations with budgets + recovery waits
test/longhaul/operations/scale_test.go	Unit tests for scale operation bounds and policy values
test/longhaul/operations/upgrade.go	Upgrade operation reading desired version from ConfigMap + readiness waits
test/longhaul/operations/upgrade_test.go	Unit tests for desired-version reads, preconditions, and outage policy
test/longhaul/operations/suite_test.go	Ginkgo suite bootstrap for operations package
test/longhaul/monitor/k8sclient.go	Typed K8s/CR client + metrics-server integration; uses test/shared/documentdb
test/longhaul/monitor/k8sclient_test.go	Unit tests with fake clientsets/controller-runtime fake client
test/longhaul/monitor/health.go	Health monitor with steady-state tracking and wait helper
test/longhaul/monitor/health_test.go	Unit tests for steady-state logic and wait behavior
test/longhaul/monitor/leakdetect.go	Linear regression leak detector for memory/CPU trends
test/longhaul/monitor/leakdetect_test.go	Unit tests for slope/leak threshold logic
test/longhaul/monitor/suite_test.go	Ginkgo suite bootstrap for monitor package
test/longhaul/journal/journal.go	Thread-safe append-only event journal + disruption window tracking
test/longhaul/journal/journal_test.go	Unit tests for events/windows/policy violation detection
test/longhaul/journal/policy.go	Outage policy + window semantics (duration, exceeded policy)
test/longhaul/journal/policy_test.go	Unit tests for policy/window semantics
test/longhaul/journal/suite_test.go	Ginkgo suite bootstrap for journal package
test/longhaul/report/report.go	Markdown report generator (metrics, windows, leaks, recent events)
test/longhaul/report/report_test.go	Unit tests for report content/sections/truncation
test/longhaul/report/checkpoint.go	Periodic checkpoint reporter writing ConfigMap + stdout + annotations
test/longhaul/report/checkpoint_test.go	Unit tests using fake clientset for ConfigMap persistence
test/longhaul/report/alert.go	GitHub Actions workflow command annotations (error/notice/warning)
test/longhaul/report/alert_test.go	Unit tests capturing stdout for annotations
test/longhaul/report/suite_test.go	Ginkgo suite bootstrap for report package
test/longhaul/deploy/setup.yaml	Bootstrap namespace/secret/DocumentDB CR manifest for canary cluster
test/longhaul/Dockerfile	Container image build for the longhaul driver
test/longhaul/README.md	Usage/configuration documentation for running + deploying longhaul
test/longhaul/go.mod	New longhaul module definition with replace to local operator/shared modules
test/longhaul/go.sum	Dependency lockfile for the longhaul module

[documentdb-triage-tool]

🤖 Auto-triaged by documentdb-triage-tool.

Applied: test, dependencies, documentation, enhancement
Project fields suggested: Component test · Priority P2 · Effort XL · Status In Progress
Confidence: 0.88 (mixed)

Reasoning

component from path globs (test, dependencies, docs); effort from diff stats (5029+0 LOC, 42 files); LLM: Large multi-package long-haul test driver core spanning several new Go packages, a Dockerfile, manifests, and docs — cross-cutting test infrastructure addition as part of a 5-PR series.

If a label is wrong, remove it manually and ping @patty-chow so the rules can be tuned. The bot will not re-label items that already have component labels.

[xgerman]

Review: structure, duplication & terseness

Focused pass on code duplication, structure, and style (the existing Copilot threads already covered correctness/security and are addressed in 3e9ac70). The module is well-documented and go build/go vet/go test ./... all pass locally. 👍

🟠 Must fix — gofmt (CI blocker)

Four files are not gofmt-clean (extra-space alignment), so make fmt/lint will fail:
operations/scale.go, operations/scheduler.go, monitor/health.go, report/checkpoint.go.
Fix: gofmt -w operations/ monitor/ report/.

🟡 Duplication / structure (highest-value dedup)

1. ScaleUp and ScaleDown are ~95% identical — collapse into one parameterized type (inline).
2. main.buildRESTConfig re-implements monitor.buildRestConfig and builds a second clientset that K8sClusterClient already owns (inline).
3. shareddb.Get(...) repeated 4× in k8sclient.go — extract a getCR helper (inline).
4. The ticker + select{ctx.Done/ticker.C} loop recurs 5× (writer/verifier/scheduler/health/metrics) — optional shared helper (inline).

🟢 Style / terseness / dead code

• journal.DefaultOutagePolicy() is unused (inline).
• report.go: b.WriteString(fmt.Sprintf(...)) → fmt.Fprintf(&b, ...) (inline).
• checkpoint.go: map[string]interface{} → map[string]any (inline).
• metrics.go: MetricsSnapshot doc comment should start with the type name (inline).

No blocking design issues — just the gofmt fix for CI plus the two consolidations (#1, #2) which are the biggest wins. Nice work on the durability oracle and the bounded verifier scan.

[xgerman]

ScaleUp (above) and ScaleDown are ~95% identical: same fields, and Precondition/Execute differ only by +1/-1, >=/<=, the bound, and a couple of constants. Consider collapsing into one parameterized type to halve the file:

type scaleOp struct {
    client    monitor.ClusterClient
    healthMon *monitor.HealthMonitor
    name      string
    weight    int
    delta     int        // +1 up, -1 down
    limit     int        // max (up) or min (down)
    atLimit   func(cur int) bool
    recovery  time.Duration
    policy    journal.OutagePolicy
}

Thin NewScaleUp/NewScaleDown constructors keep call sites unchanged.

(Also: Name()/Weight() on lines 37-38 and 98-99 fail gofmt — extra spaces.)

[WentingWu666666]

Fixed in ed498df — collapsed into a shared scaleOp struct parameterized by delta/bound/boundKind/policy. ScaleUp and ScaleDown are now thin embedding wrappers so call sites and the existing NewScaleUp/NewScaleDown constructors are unchanged. Tests use small maxInstances()/minInstances() accessors instead of field access. Net: scale.go went from 136 -> ~130 lines and has zero duplication. Also gofmt-fixed (the extra-space alignment you flagged on lines 37-38 / 98-99 is gone).

[xgerman]

buildRESTConfig here re-implements monitor.buildRestConfig (k8sclient.go:86) almost verbatim, and initK8sClient then creates a second clientset (line 231) even though K8sClusterClient already built one (k8sclient.go:59). Expose the existing one instead:

func (k *K8sClusterClient) Clientset() kubernetes.Interface { return k.clientset }

Then delete buildRESTConfig and the redundant NewForConfig — one client, one config path.

[WentingWu666666]

Fixed in ed498df — added K8sClusterClient.Clientset() kubernetes.Interface, deleted main.buildRESTConfig and initK8sClient entirely, and dropped the unused k8s.io/client-go/{kubernetes,rest,tools/clientcmd} imports from main.go. main.go now does the one monitor.NewK8sClusterClient(...) call and reuses clusterClient.Clientset() for the reporter. main.go shrank by ~40 lines.

[xgerman]

shareddb.Get(ctx, k.crClient, types.NamespacedName{Namespace: k.namespace, Name: k.clusterName}) is repeated verbatim in 4 methods (GetClusterHealth, GetInstancesPerNode, GetCurrentDocumentDBImageTag, UpgradeDocumentDB). Extract a private helper:

func (k *K8sClusterClient) getCR(ctx context.Context) (*previewv1.DocumentDB, error) {
    return shareddb.Get(ctx, k.crClient, types.NamespacedName{Namespace: k.namespace, Name: k.clusterName})
}

[WentingWu666666]

Fixed in ed498df — added private (k *K8sClusterClient) getCR(ctx) helper. All four call sites (GetClusterHealth, GetInstancesPerNode, GetCurrentDocumentDBImageTag, UpgradeDocumentDB) now use it. types/shareddb are still imported once at the top for the helper itself.

[xgerman]

This ticker + select { <-ctx.Done() / <-ticker.C } loop is duplicated 5× across the module (here, writer.Run, verifier.Run, scheduler.Run, runMetricsSampling). Optional, since they span packages, but a tiny shared helper would DRY them:

func tick(ctx context.Context, d time.Duration, fn func(context.Context)) {
    t := time.NewTicker(d); defer t.Stop()
    for { select { case <-ctx.Done(): return; case <-t.C: fn(ctx) } }
}

[WentingWu666666]

Skipped intentionally. Agree the pattern recurs 5 times, but a cross-package utility for a 5-line idiom isn't a clear win — it would mean either a new test/longhaul/internal package (overhead) or sticking it in one of the existing packages (awkward import direction). Each call site is also a Run() method with package-specific logging, so the saved lines per site are closer to 2-3. Happy to revisit if more components get added in PR-3/PR-4.

[xgerman]

DefaultOutagePolicy() is dead code — it's never referenced anywhere (each operation builds its OutagePolicy inline). Either use it as the base for the scale/upgrade policies or remove it.

[WentingWu666666]

Keeping DefaultOutagePolicy() — it's referenced from 3 test files (journal/journal_test.go:70-71, journal/policy_test.go:80, operations/scheduler_test.go:38) where the tests need a "reasonable defaults" stand-in to construct fake operations / open disruption windows. Removing it would push three literal OutagePolicy{...} copies into the tests, which is worse. Open to renaming if you'd prefer something like BaselineOutagePolicy() to signal it's a test/baseline default rather than a production default.

[xgerman]

b.WriteString(fmt.Sprintf(...)) appears ~15× in this function. fmt.Fprintf(&b, ...) is the idiomatic, terser form (a strings.Builder implements io.Writer), e.g.:

fmt.Fprintf(&b, "**Result:** %s\n", s.Result)

[WentingWu666666]

Fixed in ed498df — all ~15 occurrences switched to fmt.Fprintf(&b, ...). Pure b.WriteString("literal") calls (no formatting needed) kept as-is since Fprintf would be unnecessary indirection there.

[xgerman]

map[string]interface{} → map[string]any (Go 1.18+, and go.mod is 1.25). Minor: fmt.Printf("\n%s\n", "=== CHECKPOINT REPORT ===") on lines 93/95 can just be fmt.Println(...). Note this file also fails gofmt (the JSON map literal alignment).

[WentingWu666666]

Fixed in ed498df — map[string]any, Printf("\n%s\n", "...") collapsed to Println("\n...") / Print("...\n\n"), and gofmt -w cleaned the JSON map alignment. go vet and gofmt -l are both clean now.

[xgerman]

This doc comment is attached to type MetricsSnapshot but reads "Snapshot returns a point-in-time copy...". Per Go convention the comment should start with the identifier it documents, e.g. // MetricsSnapshot is a point-in-time copy of all metric values.

[WentingWu666666]

Fixed in ed498df — comment now reads // MetricsSnapshot is a point-in-time copy of all metric values.