Repository navigation
Conversation
Add covdata collection to unit, integration, deployment, tools, and E2E workflows. Add a coverage collector that merges artifacts, computes global and patch coverage, and publishes a report to the PR. Add a coverage flush hook for instrumented E2E nodes.
Send SIGUSR1 to coverage-instrumented containers while tests run so counters survive container teardown. Fix coverage CLI lint findings and sanitize the CI Tools artifact name.
Coverage-instrumented E2E nodes can be SIGKILLed before graceful shutdown. Flush counters every 10 seconds in coverage-tagged builds so counter data survives container teardown.
Add merge-profiles for fast text-profile union and let the collector compute patch coverage for a separate merged-PR range. Avoids merging hundreds of covdata directories at once.
Temporary pull_request trigger so the collector can be validated before the workflow exists on the default branch. Remove the manual dispatch trigger before the final PR.
|
✅ No conflicts with other open PRs targeting |
|
I see you updated files related to
|
Coverage report for
|
| Scope | Statements | Covered | Missed | Coverage |
|---|---|---|---|---|
| raw | 144757 | 78992 | 65765 | 54.57% |
| source-only | 115422 | 70470 | 44952 | 61.05% |
Patch
| Scope | Changed files | Changed lines | Statements | Covered | Missed | Coverage |
|---|---|---|---|---|---|---|
| raw | 3 | 760 | 349 | 153 | 196 | 43.84% |
| source-only | 3 | 760 | 349 | 153 | 196 | 43.84% |
Changed files
| File | Statements | Covered | Missed | Coverage | Missed spans |
|---|---|---|---|---|---|
core/coverage_flush.go |
18 | 16 | 2 | 88.89% | 20-21, 32-33 |
tools/coverage/main.go |
331 | 137 | 194 | 41.39% | 100-102, 106-112, 115-118, 119-120, 121-122, 123-124, 125-125, 126-126, 127-128, 130-131, 132-133, 134-135, 136-139, 140-141, 142-145, 146-147, 148-148, 152-155, 156-157, 158-159, 160-161, 162-163, 164-165, 166-167, 168-169, 170-171, 172-173, 174-175, 176-177, 178-178, 179-180, 181-182, 183-183, 184-185, 186-186, 187-188, 189-189, 190-191, 192-192, 194-194, 195-196, 197-198, 200-200, 204-214, 215-216, 217-218, 219-220, 222-223, 224-225, 226-231, 232-232, 233-233, 235-235, 236-236, 237-238, 239-240, 241-242, 243-243, 244-244, 245-246, 249-250, 251-251, 252-253, 255-255, 256-257, 258-259, 260-260, 261-262, 264-264, 265-266, 268-269, 270-271, 272-272, 273-273, 274-275, 275-275, 276-277, 279-279, 280-281, 282-282, 290-291, 301-303, 310-312, 321-322, 326-327, 331-332, 335-336, 339-340, 343-344, 347-348, 364-365, 368-369, 372-373, 402-403, 406-407, 487-488, 493-496, 497-498, 499-499, 514-515, 530-531, 535-536, 544-545, 550-551, 556-557, 558-558, 559-560, 562-562, 566-568, 569-570, 571-576, 577-581, 582-583, 584-584, 585-588, 589-589, 590-590, 592-593, 597-597, 601-602, 605-607, 81-81, 82-84, 85-86, 88-88, 90-90, 92-92, 94-95, 97-97, 99-99 |
Coverage — unit
5df9ce31cd830a93ca27c2baa70c714d29aba73c..2803024b589923fc3f2d3528de0bbb5a8e0a40a8
Global
| Scope | Statements | Covered | Missed | Coverage |
|---|---|---|---|---|
| raw | 88164 | 48976 | 39188 | 55.55% |
| source-only | 64611 | 43958 | 20653 | 68.03% |
Patch
| Scope | Changed files | Changed lines | Statements | Covered | Missed | Coverage |
|---|---|---|---|---|---|---|
| raw | 3 | 760 | 331 | 137 | 194 | 41.39% |
| source-only | 3 | 760 | 331 | 137 | 194 | 41.39% |
Changed files
| File | Statements | Covered | Missed | Coverage | Missed spans |
|---|---|---|---|---|---|
tools/coverage/main.go |
331 | 137 | 194 | 41.39% | 100-102, 106-112, 115-118, 119-120, 121-122, 123-124, 125-125, 126-126, 127-128, 130-131, 132-133, 134-135, 136-139, 140-141, 142-145, 146-147, 148-148, 152-155, 156-157, 158-159, 160-161, 162-163, 164-165, 166-167, 168-169, 170-171, 172-173, 174-175, 176-177, 178-178, 179-180, 181-182, 183-183, 184-185, 186-186, 187-188, 189-189, 190-191, 192-192, 194-194, 195-196, 197-198, 200-200, 204-214, 215-216, 217-218, 219-220, 222-223, 224-225, 226-231, 232-232, 233-233, 235-235, 236-236, 237-238, 239-240, 241-242, 243-243, 244-244, 245-246, 249-250, 251-251, 252-253, 255-255, 256-257, 258-259, 260-260, 261-262, 264-264, 265-266, 268-269, 270-271, 272-272, 273-273, 274-275, 275-275, 276-277, 279-279, 280-281, 282-282, 290-291, 301-303, 310-312, 321-322, 326-327, 331-332, 335-336, 339-340, 343-344, 347-348, 364-365, 368-369, 372-373, 402-403, 406-407, 487-488, 493-496, 497-498, 499-499, 514-515, 530-531, 535-536, 544-545, 550-551, 556-557, 558-558, 559-560, 562-562, 566-568, 569-570, 571-576, 577-581, 582-583, 584-584, 585-588, 589-589, 590-590, 592-593, 597-597, 601-602, 605-607, 81-81, 82-84, 85-86, 88-88, 90-90, 92-92, 94-95, 97-97, 99-99 |
Coverage — integration
5df9ce31cd830a93ca27c2baa70c714d29aba73c..2803024b589923fc3f2d3528de0bbb5a8e0a40a8
Global
| Scope | Statements | Covered | Missed | Coverage |
|---|---|---|---|---|
| raw | 127907 | 42211 | 85696 | 33.00% |
| source-only | 105544 | 37466 | 68078 | 35.50% |
Patch
| Scope | Changed files | Changed lines | Statements | Covered | Missed | Coverage |
|---|---|---|---|---|---|---|
| raw | 3 | 760 | 0 | 0 | 0 | 0.00% |
| source-only | 3 | 760 | 0 | 0 | 0 | 0.00% |
Changed files
| File | Statements | Covered | Missed | Coverage | Missed spans |
|---|
Coverage — e2e
5df9ce31cd830a93ca27c2baa70c714d29aba73c..2803024b589923fc3f2d3528de0bbb5a8e0a40a8
Global
| Scope | Statements | Covered | Missed | Coverage |
|---|---|---|---|---|
| raw | 55561 | 18235 | 37326 | 32.82% |
| source-only | 54452 | 18118 | 36334 | 33.27% |
Patch
| Scope | Changed files | Changed lines | Statements | Covered | Missed | Coverage |
|---|---|---|---|---|---|---|
| raw | 3 | 760 | 18 | 16 | 2 | 88.89% |
| source-only | 3 | 760 | 18 | 16 | 2 | 88.89% |
Changed files
| File | Statements | Covered | Missed | Coverage | Missed spans |
|---|---|---|---|---|---|
core/coverage_flush.go |
18 | 16 | 2 | 88.89% | 20-21, 32-33 |
The PR base SHA can be ahead of the branch point. Use git merge-base so patch coverage only includes the PR's own changes.
Remove the temporary pull-request and manual dispatch triggers used to validate the collector before merge. The final collector is triggered by completed test workflows.
Coverage evidence — PR #24064PR: #24064 Measurement: current develop contains #24064. Coverage artifacts were collected from the GHA coverage harness commit Global coverage (GHA artifacts)
PR #24064 patch coverage37 coverable changed statements after excluding
Per file:
The two uncovered accessors in
Test resultsCurrent PR collector run: 13,287 tests, 1 failure, 114 skipped across unit/integration artifacts. The collector is intentionally report-only. Artifacts / runs
Caveats
|
Use context-aware exec, tighten output permissions, apply modernize simplifications, and mark tests parallel.
Allow E2E coverage via the run-e2e-coverage PR label, wait for all relevant workflow runs before reporting, keep raw artifacts out of the report artifact, degrade gracefully, dedupe covdata dirs, paginate run lookups, and restrict comment updates to the collector bot.
Keep a pull_request trigger for pre-merge reporting and wait for all relevant runs before collecting. The collector remains workflow_run-driven after merge.
Avoid blocking the collector on unrelated PR workflows such as CodeQL, UI, and vault checks.
| .PHONY: install-chainlink-cover | ||
| install-chainlink-cover: operator-ui ## Install the chainlink binary with cover flag. | ||
| go install -cover $(GOFLAGS) . | ||
| go install -tags coverage -cover -covermode=atomic $(GOFLAGS) . |
There was a problem hiding this comment.
I think we may need to introduce the dev tag here too.
There was a problem hiding this comment.
Or do something like my original proposal with a debug image using dev, -cover, and -race. e.g. make install-chainlink-debug.
Enable coverage only on scheduled (or manually forced) race runs and upload coverage-core-go_core_race_tests for 7 days. The collector skips that artifact so -race instrumentation cannot skew project or patch coverage. Add a workflow_dispatch hook to recalculate coverage for a SHA.
install-chainlink-cover now includes the dev tag, matching the normal E2E image. Add install-chainlink-debug (dev + coverage + -race) and a GO_DEBUG_FLAG Docker build path. Wire a debug input and run-e2e-debug label into integration-tests so the debug image is selectable.
Nightly race coverage + jmank88 follow-upWhat changed
Nightly race artifact proof
Excluded from coverageThe collector output for this head contains only
Coverage recalculation at
|
| Metric | Before (9866db13) |
After (a286ce01d) |
|---|---|---|
| Global raw | 54.08% | 54.27% |
| Global source-only | 60.65% | 60.92% |
| Patch | 43.84% | 43.84% |
- Test results: 13,615 tests, 0 failures, 0 errors, 114 skipped, 91 files
- The E2E coverage suite passed with the
dev-tagged coverage image:
Integration Tests run — success, 0 failures. - Recalculation run:
Coverage Collector — success.
Build evidence
go test ./tools/coverage/...— passgo version -mfor the cover binary:
-cover=true,-tags=dev,coveragego version -mfor the debug binary:
-cover=true,-race=true,-tags=dev,coveragedocker buildx build --check -f core/chainlink.Dockerfile .— no warnings
One E2E coverage run failed before the final fix because env.E2E_DEBUG is the
string "false" (truthy in GitHub expressions), so the build tagged the image
-debug while the tests looked for -coverage. The tag comparison was fixed in
a286ce01d; the green E2E run above is the result.
Quick patch coverageAffected-package coverage for Coverage — patch (affected packages)
Global
Patch
Changed files
Not measuredThese changed files had no tests in their own package, so the quick run could not measure them:
The nightly full report covers these. |
| # Fast patch-only coverage for the packages touched by a PR. | ||
| # | ||
| # It intentionally does NOT run the full suite or E2E. It runs the affected | ||
| # packages with -short, uses the normal Go build/module caches, computes patch |
There was a problem hiding this comment.
As a PR author, I am primarily interested in understanding the coverage of the test suites that execute in CI, in part and in whole. Although executing a subset of those tests (unit, package list, -short) is often useful, it represents a direct tradeoff of coverage for speed. In other words, we know we only get partial coverage and we accept that to do a quick spot check because we don't care about those particulars. Why do you think it important to support collecting and analyzing that partial coverage?
| echo "Test execution results: ---------------------" | ||
| echo "" | ||
| GORACE="log_path=$PWD/race" go test -race -shuffle on -timeout "$TIMEOUT" -count "$COUNT" $1 | cat > $OUTPUT_FILE | ||
| GORACE="log_path=$PWD/race" go test -race -shuffle on -timeout "$TIMEOUT" -count "$COUNT" $COVERAGE_FLAGS "$1" $COVERAGE_ARGS | cat > $OUTPUT_FILE |
There was a problem hiding this comment.
Collecting coverage from race tests can be interesting, but we must not mix this together with normal test coverage results, because it does not necessarily represent passing tests. We tolerate test failures here (and in fact induce them by soaking until timeout), and only fail for data races.
| # This is intentionally not the full coverage suite: it runs affected packages | ||
| # with -short, reuses the normal Go build/module caches, and posts a sticky PR | ||
| # comment. The nightly full report remains the authoritative number. |
There was a problem hiding this comment.
Are these the only two modes? I think it is important to be able to get full information for a single PR about what CI actually did. And in that case we already benefit from any logic for limiting tests based on affected changes, don't we? As a PR author, I'm not sure what I would do with this faster result because it is also more limited and so gaps in coverage may not actually exist. Don't we need to wait for more information to make an informed next move?
There was a problem hiding this comment.
Why use bash and python here? Should be easy to use Go instead, right?
Summary
Adds Go coverage collection to unit, integration, deployment, tools, and E2E test workflows, with a collector workflow that merges artifacts and publishes raw coverage to the PR.
Validation
Local unit/integration coverage and GHA coverage runs are being collected. Evidence for merged PR #24064 will be added as a comment.
Notes
This is a draft for CI validation.