Skip to content

perf(test): cut the local test run from ~150s to ~65s (#1174) - #1186

Open
sdairs wants to merge 11 commits into
claude/audit-crypto-bigintfrom
claude/1174-test-speed
Open

sdairs wants to merge 11 commits into
claude/audit-crypto-bigintfrom
claude/1174-test-speed

Conversation

@sdairs

@sdairs sdairs commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Part of #1174.

Summary

A warm cargo test -p clickhousectl on the stack top went from 152.8s to about 61–75s. The range is run-to-run noise on a shared machine. Coverage is unchanged; two slow tests were restored in fast form after review.

  • Hidden test overrides for production waits. src/timing.rs adds one helper, duration_override(env, default). Each wait keeps its shipped constant as the default; an unset or malformed variable keeps the default. The overrides are documented only in docs/development.md, not in help or the user docs:
    • CHCTL_UDF_LOAD_TIMEOUT_MS
    • CHCTL_TERM_TIMEOUT_MS
    • CHCTL_SERVICE_STOP_POLL_INTERVAL_MS
    • CHCTL_UDF_POLL_INTERVAL_MS
    • CHCTL_QUERY_READINESS_TIMEOUT_MS
    • CHCTL_KEY_PROPAGATION_TIMEOUT_MS
    • CHCTL_UPDATE_CHECK_TIMEOUT_MS
  • Subprocess tests no longer hit the release server. Each spawned CLI with a fresh HOME made a real HTTPS update check, and main waits up to 500ms for it. Tests now set CHCTL_UPDATE_CHECK_TIMEOUT_MS=0, which skips only the background check:
    • cli_request_shape_test uses an isolated_env() helper in place of env_clear().
    • The local_* spawn helpers set the variable.
    • This took cli_request_shape_test from 47s to 14s.
  • Slow tests made fast:
    • The 30s propagation-window rollback, the forced-delete stop-poll tests and the UDF never-loads tests now use short windows.
    • The two #[ignore] 120s readiness tests now run with a 2s window.
    • The redundant human-mode UDF re-run was dropped; Mapping::parity makes its message the same as the JSON one.
    • The two overlapping Postgres 1s-timeout tests were merged into one.
  • Parallelism: local_server_state_machine is now 5 #[test] functions.
  • Telemetry: each test waits once for stray events, not once per command. One failed-exec test still checks "exactly one event", the only path where both the exec hook and main's tail run.
  • Flaky timing checks removed:
    • The elapsed() < 250ms deadline tests in version_manager run on a paused clock. They were confirmed to fail with the deadline removed.
    • A stale "Waiting for the new API key" absence check now uses KEY_PROPAGATION_NOTICE.
  • Classifier: src/timing.rs is classified as an install path, alongside stdout.rs and http.rs.
Binary Before After
unit tests 5.3s 4.4s
cli_request_shape_test 47.2s 14.0s
local_udf_reload_test 22.8s 3.7s
local_server_state_machine_test 10.3s 4.0s
local_postgres_readiness_test 7.2s 4.1s
telemetry_test 6.1s 3.3s
local_server_watchdog_pid_test 5.0s 2.5s

Not done here: cargo nextest, and the low-value test cleanup, which follows in this stack (#1175).

Test plan

  • cargo fmt --all --check
  • Both clippy configurations, and cargo check -p clickhousectl --no-default-features
  • cargo test -p clickhousectl, three full runs with no flakes
  • python3 -m unittest discover -s scripts/tests -p 'test_*.py'
  • An independent verifier broke the exactly-once guard and the deadline code and confirmed the tests fail

🤖 Generated with Claude Code


Note

Low Risk
Shipped timeouts unchanged for normal users; overrides are test-only env vars with safe fallbacks to defaults.

Overview
Speeds up the clickhousectl test suite (roughly ~150s → ~65s) without dropping coverage by making subprocess tests avoid real waits and network side effects.

Hidden millisecond overrides live in new timing.rs (duration_override). Production poll/timeout constants stay the defaults unless a CHCTL_*_MS variable is set; invalid values are ignored. Call sites updated include forced service delete stop polling, cloud UDF waits, key propagation and Query API readiness, local SIGTERM grace, UDF reload windows, and the background update check (0 skips the request only for that path).

Test harness changes: spawned CLIs set CHCTL_UPDATE_CHECK_TIMEOUT_MS=0 so tests do not hit the release server; cli_request_shape_test adds isolated_env() for that after env_clear. Previously ignored or multi‑minute tests use short override windows instead of shipping timeouts. local_server_state_machine splits into five parallel #[test] functions; telemetry tests batch “no stray event” waits; version-manager deadline tests use start_paused = true.

Docs/classifier: docs/development.md documents the override table for developers; install-path classification includes timing.rs.

Reviewed by Cursor Bugbot for commit caed885. Bugbot is set up for automated code reviews on this repo. Configure here.

@sdairs
sdairs requested a review from iskakaushik as a code owner November 2, 2026 10:49
@sdairs
sdairs changed the base branch from claude/1173-api-split-clickstack-models to claude/audit-crypto-bigint November 2, 2026 10:49
@sdairs
sdairs added this pull request to stack #1153 November 2, 2026 10:49
sdairs and others added 11 commits November 2, 2026 10:49
Subprocess tests waited out real timeouts and poll intervals. Add
`src/timing.rs` with one helper, `duration_override`, that reads a
`CHCTL_*_MS` variable and falls back to the shipped constant, and use it
for the UDF reload window, the SIGTERM grace period, the forced-delete
stop poll, the Cloud UDF service poll and the Query API readiness window.

The slow tests now set small values. The two readiness tests that were
`#[ignore]`d for waiting out 120 s run by default with a 2 s window.
The overrides are documented for developers in docs/development.md only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `a_400_that_outlives_the_propagation_window_rolls_the_repair_back`
  waited out the real 30 s key-propagation window. The deadline and the
  last-error result are unit-tested with a paused clock
  (`a_persisting_400_fails_with_the_last_error_once_the_deadline_passes`),
  and `failed_repair_restores_bindings_deletes_new_key_and_preserves_records`
  covers the rollback.
- `forced_service_delete_reports_only_poll_state_transitions_when_redirected`
  pinned progress prose; `poll_progress_*` in `cloud/shared.rs` covers the
  rendering. The closed-stderr test stays as the stop-poll test.
- The human re-run in
  `a_function_the_server_never_loads_is_an_error_that_keeps_the_files`
  repeated the JSON run: `UdfNotLoaded` maps with message parity.
- `wall_clock_timeout_fails_and_rolls_back_fresh_data` and
  `initialization_timeout_removes_partial_pgdata` both waited out the same
  1 s Postgres readiness timeout; the second keeps the first's rollback
  ordering assertions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1174)

`local_server_state_machine` ran five scenarios in sequence inside one
`#[test]`. Each now has its own test, sharing the fixture and the
post-scenario invariant checks through `run_scenario`, so they run in
parallel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two JSON telemetry tests slept 750 ms after every sandbox; they now
run every command first and wait once (`assert_no_requests_in`). Only
`successful_handoff_is_one_censored_exec_attempt` still waits for a
second exec event; the guard is shared by every handoff path, so the
other exec tests take the first event as soon as it arrives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two operation-deadline tests in `version_manager` asserted
`elapsed() < 250ms` on a real clock, which a loaded CI machine can
break. They now run on a paused clock, so the bound is exact: time moves
only when every task waits, and the scan still fails it if the shared
deadline is missing.

The commit-lock test in `local_remove_default_test` slept 1 s for a
negative assertion; 300 ms is ample for an unlocked `local use`, and a
slow machine can only make the check weaker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1174)

Every CLI run with a stale or missing update cache makes a real HTTPS
request to the release server and waits up to 400 ms for it. Subprocess
tests run in fresh HOMEs, so nearly every spawn paid that wait and
reached the network: in `cli_request_shape_test` it was most of the
binary's 40 s.

`CHCTL_UPDATE_CHECK_TIMEOUT_MS` now overrides the background timeout,
and `0` skips the request. `cli_request_shape_test` sets it through
`isolated_env()`, which replaces `env_clear()`, and on the commands that
inherit the environment; each `local_*` spawn helper sets it too. The
tests that seed the update cache already skip the check, and
`telemetry_test` keeps it so its read-only checks on the cache file stay
meaningful. The install-matrix tests are left untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The one exec test that still waits for a stray second event was the
successful handoff, where `exec()` replaces the process and `main`'s tail
can never run, so it cannot catch a broken exactly-once guard. With the
guard removed from `finalize`, every telemetry test passed. Move the wait
to `launch_failure_after_the_pre_flight_is_censored_never_successful`,
where the hook and the tail both run; it fails again without the guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…est (#1174)

`failed_repair_restores_bindings_deletes_new_key_and_preserves_records`
now stands in for the deleted 30 s propagation test's rollback checks,
but its "no propagation wait on a 5xx" assertion looked for text the CLI
never prints. Assert on `KEY_PROPAGATION_NOTICE` instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… form (#1174)

Both deleted wiremock tests covered handler behaviour no unit test does.

- `forced_service_delete_reports_only_poll_state_transitions_when_redirected`
  checks that the stop notice reaches stderr and that `--json` with a
  redirected stderr reports state transitions only. It now uses the
  100 ms stop-poll override and asserts the `state:` lines, not prose.
- `a_400_that_outlives_the_propagation_window_rolls_the_repair_back`
  covers the multi-call repair flow: 400s outlast the window, the repair
  rolls back, the notice is printed once and the 400 is reported. A new
  `CHCTL_KEY_PROPAGATION_TIMEOUT_MS` override shortens the 30 s window to
  1.5 s; the backoff is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The overrides sit on the local server stop path (`TERM_TIMEOUT`), like
`stdout.rs` and `http.rs`, so a change to them runs the live install
matrix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1174)

`CHCTL_UPDATE_CHECK_TIMEOUT_MS=0` skips the background request; the
explicit checks behind `--version` and `update` still reach the release
server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant