Skip to content

feat(ci): collect and publish test coverage artifacts - #24111

Open
007vasy wants to merge 16 commits into
developfrom
feat/pr-code-cov
Open

007vasy wants to merge 16 commits into
developfrom
feat/pr-code-cov

Conversation

@007vasy

@007vasy 007vasy commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds Go coverage collection to unit, integration, deployment, tools, and E2E test workflows, with a collector workflow that merges artifacts and publishes raw coverage to the PR.

  • Collects covdata from unit, integration, deployment, tools, and E2E jobs.
  • Uploads short-term GitHub artifacts with commit SHA and run ID.
  • Merges unit/integration/E2E profiles and computes global + patch coverage.
  • Publishes a report-only PR comment with threshold targets.
  • Adds a coverage build tag/flush hook so E2E containers emit counters before teardown.

Validation

Local unit/integration coverage and GHA coverage runs are being collected. Evidence for merged PR #24064 will be added as a comment.

Notes

This is a draft for CI validation.

Add covdata collection to unit, integration, deployment, tools, and E2E workflows. Add a coverage collector that merges artifacts, computes global and patch coverage, and publishes a report to the PR. Add a coverage flush hook for instrumented E2E nodes.
Send SIGUSR1 to coverage-instrumented containers while tests run so counters survive container teardown. Fix coverage CLI lint findings and sanitize the CI Tools artifact name.
Coverage-instrumented E2E nodes can be SIGKILLed before graceful shutdown. Flush counters every 10 seconds in coverage-tagged builds so counter data survives container teardown.
Add merge-profiles for fast text-profile union and let the collector compute patch coverage for a separate merged-PR range. Avoids merging hundreds of covdata directories at once.
Temporary pull_request trigger so the collector can be validated before the workflow exists on the default branch. Remove the manual dispatch trigger before the final PR.
@github-actions

Copy link
Copy Markdown
Contributor

✅ No conflicts with other open PRs targeting develop

@github-actions

Copy link
Copy Markdown
Contributor

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Coverage report for 2803024b5

Commit: 2803024b589923fc3f2d3528de0bbb5a8e0a40a8
PR: #24111
Base: f9b02920ea9f8949ba13e4689c8c698a5292a686

Targets: project coverage should not drop more than 1%; patch target is 80%. This report is informational.

Test results

{"tests": 13340, "failures": 2, "errors": 0, "skipped": 110, "files": 97}

Coverage — any

5df9ce31cd830a93ca27c2baa70c714d29aba73c..2803024b589923fc3f2d3528de0bbb5a8e0a40a8

Global

Scope Statements Covered Missed Coverage
raw 144757 78992 65765 54.57%
source-only 115422 70470 44952 61.05%

Patch

Scope Changed files Changed lines Statements Covered Missed Coverage
raw 3 760 349 153 196 43.84%
source-only 3 760 349 153 196 43.84%

Changed files

File Statements Covered Missed Coverage Missed spans
core/coverage_flush.go 18 16 2 88.89% 20-21, 32-33
tools/coverage/main.go 331 137 194 41.39% 100-102, 106-112, 115-118, 119-120, 121-122, 123-124, 125-125, 126-126, 127-128, 130-131, 132-133, 134-135, 136-139, 140-141, 142-145, 146-147, 148-148, 152-155, 156-157, 158-159, 160-161, 162-163, 164-165, 166-167, 168-169, 170-171, 172-173, 174-175, 176-177, 178-178, 179-180, 181-182, 183-183, 184-185, 186-186, 187-188, 189-189, 190-191, 192-192, 194-194, 195-196, 197-198, 200-200, 204-214, 215-216, 217-218, 219-220, 222-223, 224-225, 226-231, 232-232, 233-233, 235-235, 236-236, 237-238, 239-240, 241-242, 243-243, 244-244, 245-246, 249-250, 251-251, 252-253, 255-255, 256-257, 258-259, 260-260, 261-262, 264-264, 265-266, 268-269, 270-271, 272-272, 273-273, 274-275, 275-275, 276-277, 279-279, 280-281, 282-282, 290-291, 301-303, 310-312, 321-322, 326-327, 331-332, 335-336, 339-340, 343-344, 347-348, 364-365, 368-369, 372-373, 402-403, 406-407, 487-488, 493-496, 497-498, 499-499, 514-515, 530-531, 535-536, 544-545, 550-551, 556-557, 558-558, 559-560, 562-562, 566-568, 569-570, 571-576, 577-581, 582-583, 584-584, 585-588, 589-589, 590-590, 592-593, 597-597, 601-602, 605-607, 81-81, 82-84, 85-86, 88-88, 90-90, 92-92, 94-95, 97-97, 99-99

Coverage — unit

5df9ce31cd830a93ca27c2baa70c714d29aba73c..2803024b589923fc3f2d3528de0bbb5a8e0a40a8

Global

Scope Statements Covered Missed Coverage
raw 88164 48976 39188 55.55%
source-only 64611 43958 20653 68.03%

Patch

Scope Changed files Changed lines Statements Covered Missed Coverage
raw 3 760 331 137 194 41.39%
source-only 3 760 331 137 194 41.39%

Changed files

File Statements Covered Missed Coverage Missed spans
tools/coverage/main.go 331 137 194 41.39% 100-102, 106-112, 115-118, 119-120, 121-122, 123-124, 125-125, 126-126, 127-128, 130-131, 132-133, 134-135, 136-139, 140-141, 142-145, 146-147, 148-148, 152-155, 156-157, 158-159, 160-161, 162-163, 164-165, 166-167, 168-169, 170-171, 172-173, 174-175, 176-177, 178-178, 179-180, 181-182, 183-183, 184-185, 186-186, 187-188, 189-189, 190-191, 192-192, 194-194, 195-196, 197-198, 200-200, 204-214, 215-216, 217-218, 219-220, 222-223, 224-225, 226-231, 232-232, 233-233, 235-235, 236-236, 237-238, 239-240, 241-242, 243-243, 244-244, 245-246, 249-250, 251-251, 252-253, 255-255, 256-257, 258-259, 260-260, 261-262, 264-264, 265-266, 268-269, 270-271, 272-272, 273-273, 274-275, 275-275, 276-277, 279-279, 280-281, 282-282, 290-291, 301-303, 310-312, 321-322, 326-327, 331-332, 335-336, 339-340, 343-344, 347-348, 364-365, 368-369, 372-373, 402-403, 406-407, 487-488, 493-496, 497-498, 499-499, 514-515, 530-531, 535-536, 544-545, 550-551, 556-557, 558-558, 559-560, 562-562, 566-568, 569-570, 571-576, 577-581, 582-583, 584-584, 585-588, 589-589, 590-590, 592-593, 597-597, 601-602, 605-607, 81-81, 82-84, 85-86, 88-88, 90-90, 92-92, 94-95, 97-97, 99-99

Coverage — integration

5df9ce31cd830a93ca27c2baa70c714d29aba73c..2803024b589923fc3f2d3528de0bbb5a8e0a40a8

Global

Scope Statements Covered Missed Coverage
raw 127907 42211 85696 33.00%
source-only 105544 37466 68078 35.50%

Patch

Scope Changed files Changed lines Statements Covered Missed Coverage
raw 3 760 0 0 0 0.00%
source-only 3 760 0 0 0 0.00%

Changed files

File Statements Covered Missed Coverage Missed spans

Coverage — e2e

5df9ce31cd830a93ca27c2baa70c714d29aba73c..2803024b589923fc3f2d3528de0bbb5a8e0a40a8

Global

Scope Statements Covered Missed Coverage
raw 55561 18235 37326 32.82%
source-only 54452 18118 36334 33.27%

Patch

Scope Changed files Changed lines Statements Covered Missed Coverage
raw 3 760 18 16 2 88.89%
source-only 3 760 18 16 2 88.89%

Changed files

File Statements Covered Missed Coverage Missed spans
core/coverage_flush.go 18 16 2 88.89% 20-21, 32-33

@trunk-io

trunk-io Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge

View Full Report ↗︎ ⋅ Docs

The PR base SHA can be ahead of the branch point. Use git merge-base so patch coverage only includes the PR's own changes.
Remove the temporary pull-request and manual dispatch triggers used to validate the collector before merge. The final collector is triggered by completed test workflows.
@007vasy

007vasy commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator Author

Coverage evidence — PR #24064

PR: #24064
Merge commit: fb2745bd0f1e7eaccccb2a1a2bebd8f4bc5dc543
PR head: 09b37a3be282053a4b578e62cdc9f12149fe4a50
PR base: cb18f172054bf59dd938fc0ed7cdf6afa83f8088

Measurement: current develop contains #24064. Coverage artifacts were collected from the GHA coverage harness commit 974cf972a8dfa7fce7d2dab829c3bd8aeb6da08a (commits after it only change the collector). PR patch coverage uses the PR's actual diff; core/services/cre/cre.go is excluded because it was modified again after the PR merged and its line numbers no longer match.

Global coverage (GHA artifacts)

Category Raw coverage Source-only
Unit 54.33% (45,972 / 84,609) 67.20% (40,941 / 60,924)
Integration 29.60% (37,327 / 126,092) 32.79% (34,636 / 105,643)
E2E 33.56% (18,258 / 54,400) 34.05% (18,144 / 53,291)
Any (union) 51.23% (72,847 / 142,208) 58.71% (66,337 / 112,986)

PR #24064 patch coverage

37 coverable changed statements after excluding cre.go:

Category Covered / coverable Patch coverage
Unit 35 / 37 94.6%
Integration 6 / 37 16.2%
E2E 6 / 37 16.2%
Any (union) 35 / 37 94.6%

Per file:

File Unit Integration E2E Any
core/config/toml/types.go 11/11 6/11 6/11 11/11
core/services/chainlink/config_capabilities.go 0/2 0/2 0/2 0/2
core/services/workflows/syncer/v2/direct_fetcher.go 24/24 0/24 0/24 24/24

The two uncovered accessors in config_capabilities.go are FetchMode() and FetchAllowedHosts() (lines 226–231).

cre.go is excluded from the patch table because it changed after #24064 merged; including it at current HEAD line numbers would produce misleading misses. Including it gives 43 statements: unit 35/43 (81.4%), integration 6/43 (13.95%), E2E 7/43 (16.28%), any 36/43 (83.72%).

Test results

Current PR collector run: 13,287 tests, 1 failure, 114 skipped across unit/integration artifacts. The collector is intentionally report-only.

Artifacts / runs

  • CI Core: 40830831879
  • Deployment Tests: 40830835572
  • Integration In-Memory Tests: 40830839304
  • Integration/E2E Tests: 40830814735
  • Collector run: 40978196763
  • Short-term artifacts: coverage-core-*, coverage-deployment-*, coverage-e2e-*, coverage-report-* (7–30 day retention).

Caveats

  • E2E coverage is from the full GHA coverage-image run, not a local E2E run.
  • The exact PR head was not re-run with the coverage harness; current develop contains the PR and line mapping is exact except for files changed after merge.
  • Thresholds shown to reviewers: project no drop >1%, patch target 80%; the collector is report-only.

Use context-aware exec, tighten output permissions, apply modernize simplifications, and mark tests parallel.
@007vasy
007vasy marked this pull request as ready for review October 30, 2026 03:02
@007vasy
007vasy requested review from a team as code owners October 30, 2026 03:02
Allow E2E coverage via the run-e2e-coverage PR label, wait for all relevant workflow runs before reporting, keep raw artifacts out of the report artifact, degrade gracefully, dedupe covdata dirs, paginate run lookups, and restrict comment updates to the collector bot.
@007vasy 007vasy added the run-e2e-coverage Build and run E2E tests with Go coverage instrumentation label Oct 30, 2026
Keep a pull_request trigger for pre-merge reporting and wait for all relevant runs before collecting. The collector remains workflow_run-driven after merge.
@007vasy
007vasy marked this pull request as draft October 30, 2026 03:43
@007vasy 007vasy removed the run-e2e-coverage Build and run E2E tests with Go coverage instrumentation label Oct 30, 2026
Avoid blocking the collector on unrelated PR workflows such as CodeQL, UI, and vault checks.
@007vasy 007vasy added the run-e2e-coverage Build and run E2E tests with Go coverage instrumentation label Oct 30, 2026
@007vasy
007vasy marked this pull request as ready for review October 30, 2026 04:13
Comment thread GNUmakefile Outdated
.PHONY: install-chainlink-cover
install-chainlink-cover: operator-ui ## Install the chainlink binary with cover flag.
go install -cover $(GOFLAGS) .
go install -tags coverage -cover -covermode=atomic $(GOFLAGS) .

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we may need to introduce the dev tag here too.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or do something like my original proposal with a debug image using dev, -cover, and -race. e.g. make install-chainlink-debug.

Enable coverage only on scheduled (or manually forced) race runs and
upload coverage-core-go_core_race_tests for 7 days. The collector skips
that artifact so -race instrumentation cannot skew project or patch
coverage. Add a workflow_dispatch hook to recalculate coverage for a SHA.
install-chainlink-cover now includes the dev tag, matching the normal
E2E image. Add install-chainlink-debug (dev + coverage + -race) and a
GO_DEBUG_FLAG Docker build path. Wire a debug input and run-e2e-debug
label into integration-tests so the debug image is selectable.
@007vasy

007vasy commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

Nightly race coverage + jmank88 follow-up

What changed

  • Scheduled Core Tests (go_core_race_tests) now runs with coverage and uploads
    coverage-core-go_core_race_tests-<run_id>-<run_attempt> with 7-day retention.
  • tools/coverage/collect.sh skips that artifact for the unit/any/patch
    coverage calculation, so -race instrumentation cannot skew the coverage trend.
  • install-chainlink-cover now includes the dev tag, matching the normal E2E image.
  • Added install-chainlink-debug (dev + coverage + -cover + -race) and a
    GO_DEBUG_FLAG Docker build path, wired through a debug input and
    run-e2e-debug label in integration-tests.yml.
  • Coverage Collector now has a workflow_dispatch hook so a specific SHA can be
    recalculated on demand.

Nightly race artifact proof

  • Artifact: coverage-core-go_core_race_tests-42173945351-1
    • Created: 2026-11-06T06:26:35Z
    • Expires: 2026-11-13T06:26:33Z (7 days)
    • Size: 14,491,651 bytes; contains covmeta.* and covcounters.*
    • Job: Core Tests (go_core_race_tests)
    • This was produced with the manual race_coverage=true hook, which uses the
      same code path as the schedule. The scheduled run uses COUNT=50; the manual
      run used the workflow-dispatch default of COUNT=5.

Excluded from coverage

The collector output for this head contains only unit, integration, e2e,
and any profiles — there is no race profile:

any.json  any.md  any.txt
e2e.json  e2e.md  e2e.txt
integration.json  integration.md  integration.txt
unit.json  unit.md  unit.txt
summary.md  test-results.json

tools/coverage/collect.sh explicitly filters
coverage-core-go_core_race_tests-* out before merging.

Coverage recalculation at a286ce01d

The bot coverage comment was updated in place:
#24111 (comment)

Metric Before (9866db13) After (a286ce01d)
Global raw 54.08% 54.27%
Global source-only 60.65% 60.92%
Patch 43.84% 43.84%
  • Test results: 13,615 tests, 0 failures, 0 errors, 114 skipped, 91 files
  • The E2E coverage suite passed with the dev-tagged coverage image:
    Integration Tests run — success, 0 failures.
  • Recalculation run:
    Coverage Collector — success.

Build evidence

  • go test ./tools/coverage/... — pass
  • go version -m for the cover binary:
    -cover=true, -tags=dev,coverage
  • go version -m for the debug binary:
    -cover=true, -race=true, -tags=dev,coverage
  • docker buildx build --check -f core/chainlink.Dockerfile . — no warnings

One E2E coverage run failed before the final fix because env.E2E_DEBUG is the
string "false" (truthy in GitHub expressions), so the build tagged the image
-debug while the tests looked for -coverage. The tag comparison was fixed in
a286ce01d; the green E2E run above is the result.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Quick patch coverage

Affected-package coverage for 2803024b589923fc3f2d3528de0bbb5a8e0a40a8, using -short.
This is a fast PR signal, not the global coverage number; the nightly full report is authoritative.

Coverage — patch (affected packages)

5df9ce31cd830a93ca27c2baa70c714d29aba73c..2803024b589923fc3f2d3528de0bbb5a8e0a40a8

Global

Scope Statements Covered Missed Coverage
raw 331 137 194 41.39%
source-only 331 137 194 41.39%

Patch

Scope Changed files Changed lines Statements Covered Missed Coverage
raw 3 760 331 137 194 41.39%
source-only 3 760 331 137 194 41.39%

Changed files

File Statements Covered Missed Coverage Missed spans
tools/coverage/main.go 331 137 194 41.39% 100-102, 106-112, 115-118, 119-120, 121-122, 123-124, 125-125, 126-126, 127-128, 130-131, 132-133, 134-135, 136-139, 140-141, 142-145, 146-147, 148-148, 152-155, 156-157, 158-159, 160-161, 162-163, 164-165, 166-167, 168-169, 170-171, 172-173, 174-175, 176-177, 178-178, 179-180, 181-182, 183-183, 184-185, 186-186, 187-188, 189-189, 190-191, 192-192, 194-194, 195-196, 197-198, 200-200, 204-214, 215-216, 217-218, 219-220, 222-223, 224-225, 226-231, 232-232, 233-233, 235-235, 236-236, 237-238, 239-240, 241-242, 243-243, 244-244, 245-246, 249-250, 251-251, 252-253, 255-255, 256-257, 258-259, 260-260, 261-262, 264-264, 265-266, 268-269, 270-271, 272-272, 273-273, 274-275, 275-275, 276-277, 279-279, 280-281, 282-282, 290-291, 301-303, 310-312, 321-322, 326-327, 331-332, 335-336, 339-340, 343-344, 347-348, 364-365, 368-369, 372-373, 402-403, 406-407, 487-488, 493-496, 497-498, 499-499, 514-515, 530-531, 535-536, 544-545, 550-551, 556-557, 558-558, 559-560, 562-562, 566-568, 569-570, 571-576, 577-581, 582-583, 584-584, 585-588, 589-589, 590-590, 592-593, 597-597, 601-602, 605-607, 81-81, 82-84, 85-86, 88-88, 90-90, 92-92, 94-95, 97-97, 99-99

Not measured

These changed files had no tests in their own package, so the quick run could not measure them:

  • core/coverage_flush.go

The nightly full report covers these.

Comment thread tools/coverage/patch.sh
# Fast patch-only coverage for the packages touched by a PR.
#
# It intentionally does NOT run the full suite or E2E. It runs the affected
# packages with -short, uses the normal Go build/module caches, computes patch

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a PR author, I am primarily interested in understanding the coverage of the test suites that execute in CI, in part and in whole. Although executing a subset of those tests (unit, package list, -short) is often useful, it represents a direct tradeoff of coverage for speed. In other words, we know we only get partial coverage and we accept that to do a quick spot check because we don't care about those particulars. Why do you think it important to support collecting and analyzing that partial coverage?

echo "Test execution results: ---------------------"
echo ""
GORACE="log_path=$PWD/race" go test -race -shuffle on -timeout "$TIMEOUT" -count "$COUNT" $1 | cat > $OUTPUT_FILE
GORACE="log_path=$PWD/race" go test -race -shuffle on -timeout "$TIMEOUT" -count "$COUNT" $COVERAGE_FLAGS "$1" $COVERAGE_ARGS | cat > $OUTPUT_FILE

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Collecting coverage from race tests can be interesting, but we must not mix this together with normal test coverage results, because it does not necessarily represent passing tests. We tolerate test failures here (and in fact induce them by soaking until timeout), and only fail for data races.

Comment on lines +4 to +6
# This is intentionally not the full coverage suite: it runs affected packages
# with -short, reuses the normal Go build/module caches, and posts a sticky PR
# comment. The nightly full report remains the authoritative number.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these the only two modes? I think it is important to be able to get full information for a single PR about what CI actually did. And in that case we already benefit from any logic for limiting tests based on affected changes, don't we? As a PR author, I'm not sure what I would do with this faster result because it is also more limited and so gaps in coverage may not actually exist. Don't we need to wait for more information to make an informed next move?

Comment thread tools/coverage/collect.sh

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why use bash and python here? Should be easy to use Go instead, right?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-e2e-coverage Build and run E2E tests with Go coverage instrumentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants