Skip to content

the nightly runs the big C++ tests - ctest -L big after the AOT sweep on every Release cell; the two style_lint fixture tests that were red on master lint their fixtures again and run per PR as small - #3975

Merged
borisbat merged 1 commit into
masterfrom
bbatkin/nightly-big-tests
Sep 9, 2026
Merged

Conversation

@borisbat

@borisbat borisbat commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Why. Every per-PR and nightly ctest in CI is -L small, so the big-labelled C++ tests - a generated standalone context linked and run, the nano context, concurrent init, the C API split init - are built by the default target on every lane and executed on none. v0.6.4-RC2 shipped a -ctx whose generated code did not link and, once linked, never ran its initializers (#3967); tests-cpp/big/standalone_ctx shows both, and no lane ran it.

What changes.

  • The nightly "Slow Release Tests" step runs ctest -L big -E memory_model_4gb after the AOT sweep, on every Release cell the step already covers - Linux, macOS and Windows, sanitizer cells included. No extra build: the binaries are already in the default target. memory_model_4gb allocates a real 4 GB chunk and stays local-only, as its CMakeLists says.
  • Running the set showed the two style_lint fixture tests red on master: the lint CLI's content skip for an expect file fired before the --lint-fixtures override the comment beside it promised, so both fixtures were skipped and neither STYLE014 nor STYLE015 was ever printed. lint_file_skip_reason takes an expect_skips flag (default true, so the MCP lint tool is unchanged) that the CLI clears under --lint-fixtures; a // lint-skip-file directive keeps winning whatever the caller, which tests/lint/test_stale_nolint.das pins.
  • Those two tests are relabelled small: each is one sub-second lint of one fixture, so every per-PR ctest runs them and the lint fix has a per-PR row.
  • tests-cpp/big/REVIEW.md states the contract as it now is: per-PR CI runs -L small, the nightly runs -L big after the merge.

Observable behavior.

  • A generated standalone context that does not link, or does not initialize, goes unnoticed until a release -> fails the next nightly on every platform.
  • daslang utils/lint/main.das -- --lint-fixtures true <expect fixture>: skipped -> linted, findings printed. Without the flag an expect file is still skipped.
  • Per-PR CI: two sub-second lint tests added to the small set; nothing else changes.

Where to look. .github/workflows/build.yml, the "Slow Release Tests" step - the ctest line sits after the case, so every arm, the 32-bit Windows one included, runs it; utils/lint/main.das set_skip_reason, one condition.

Validation, claims, ledger

Validation

  • The exact nightly command, locally (macOS arm64, Release): ctest --test-dir build --build-config Release -L big -E memory_model_4gb --output-on-failure - 6 of 6 pass in 21 s (standalone_ctx, nano_ctx, concurrent_init, concurrent_init_dyn, capi_split_init, cpp_module_pull). Before the lint fix and relabel the same command was 6 of 8, the two style_lint tests failing on a skipped fixture.
  • ctest -L small -R style_lint: 2 of 2 after the relabel; tests/lint/test_stale_nolint.das 7 of 7, its lint-skip-file arm being the one a first cut of this fix broke on the modules lane.
  • Negative control on the lint fix: the fixture without --lint-fixtures still reports SKIP; with it, two STYLE014 findings.
  • python3 ci/test_ci_matrix.py: 18 tests OK; the workflow parses.
  • Preflight fast tier on the tip: 10 gates pass, 0 fail, 3 skip as not reached (format, lint on both rails, hash-refs, review-md, md-ascii, ast-verify, ci-das, ci-matrix, compile-sweep over 760 roots).
  • The codex round ran over the branch at its tip, read the workflow file, and returned no findings.
  • The dragon read tests-cpp/big/REVIEW.md twice: the first round corrected the rationale (a manual full-workflow run can target a PR branch), the fresh cold read after it is all-OK, and it found the sentence in skills/internal/writing_cpp_tests.md that this change made false, fixed here.
  • The comment harvest and style-hygiene rows were skipped by standing instruction.

Lint candidate

  • A PR gate that maps each changed path to its CMakeLists.txt LABELS and fails a big hit whose PR body names no ctest -L big run; tests-cpp/REVIEW.das already parses those labels. Not written here.

Claims - stated, not tested

  • The Windows and sanitizer cells were not run here; the step's case arms and env exports are unchanged, and the added line uses the same ctest --test-dir build --build-config shape the RelWithDebInfo nightly already uses. A big test that fails under a sanitizer shows up on the first nightly, which is the point.
  • The 32-bit Windows arm skips the AOT sweep but runs the big set: its binaries come from the same Build step that already generates and compiles the watchdog's -ctx context on that lane.

Not done

  • lookibed's proposed tests-cpp/big/standalone_module_global (a required module owning initialized globals, linked and run) is a real gap beside master's emit-only test and is not added here.

🤖 Generated with Claude Code

Copilot AI lite review requested due to automatic review settings September 8, 2026 21:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The changes are cohesive and low-risk, and the updated skip/label logic matches the stated CI/test intent without introducing obvious behavioral regressions.

Pull request overview

This PR closes a CI coverage gap by actually executing the big-labelled C++ tests in nightly Release runs (after the full AOT sweep), and fixes utils/lint so the --lint-fixtures override truly forces linting of expectation/fixture files. It also reclassifies the two style_lint fixture tests as small so they run on every PR.

Changes:

  • Run ctest -L big -E memory_model_4gb in the nightly “Slow Release Tests” step for Release matrix cells.
  • Fix utils/lint skipping logic so --lint-fixtures bypasses content-based skip reasons (including expect-style files).
  • Relabel the two style_lint fixture tests from big to small, and update the related checklist/docs accordingly.
File summaries
File Description
.github/workflows/build.yml Adds nightly execution of big ctest label after the AOT sweep in Release cells.
utils/lint/main.das Makes --lint-fixtures override content-based skip reasons so fixtures are actually linted.
tests-cpp/big/style_lint/CMakeLists.txt Relabels fixture lint tests as small and wires them into the test-small aggregate target.
tests-cpp/big/REVIEW.md Updates the “big tests” checklist language to reflect nightly/manual workflow coverage.
skills/internal/writing_cpp_tests.md Updates guidance to reflect that big tests run on nightly/manual full-workflow runs, not per-PR.
Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…e AOT sweep - ctest -L big on every Release cell, memory_model_4gb excluded: the standalone context, the nano context, concurrent init and the C API split init have always been built by the default target on every lane and never executed, since every per-PR ctest is -L small, so no lane linked and ran a generated context (v0.6.4-RC2 shipped a -ctx that did not link and never ran its initializers, issue #3967); memory_model_4gb allocates a real 4 GB chunk and stays local-only. Running the set showed the two style_lint fixture tests red on master: the lint CLI's content skip for an `expect` file fired before the --lint-fixtures override the comment beside it promised, so both fixtures were skipped and neither STYLE014 nor STYLE015 was ever printed - lint_file_skip_reason takes an expect_skips flag the CLI clears under --lint-fixtures, while a lint-skip-file directive keeps winning whatever the caller (tests/lint/test_stale_nolint.das pins that), and the two tests are labelled small since each is one sub-second lint of one fixture, so every per-PR ctest runs them. tests-cpp/big/REVIEW.md and the C++ test skill state the contract as it now is: per-PR CI runs -L small, -L big runs on the nightly and on a manual full-workflow run

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat
borisbat force-pushed the bbatkin/nightly-big-tests branch from deaedc4 to 25a716f Compare September 8, 2026 22:00
Copilot AI review requested due to automatic review settings September 8, 2026 22:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The changes are coherent and consistent across workflow, lint behavior, and documentation, and I didn’t find any correctness or integration issues in the updated call sites/targets.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@borisbat
borisbat merged commit dc1e81f into master Sep 9, 2026
32 checks passed
@borisbat
borisbat deleted the bbatkin/nightly-big-tests branch September 9, 2026 00:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants