Skip to content

chore: tag integration tests - #20541

Closed
erikburt wants to merge 22 commits into
developfrom
test/disable-unit-test-setup
Closed

erikburt wants to merge 22 commits into
developfrom
test/disable-unit-test-setup

Conversation

@erikburt

@erikburt erikburt commented Dec 5, 2025 •

Copy link
Copy Markdown
Collaborator

Trying to properly differentiate between tests that require no external dependencies, and tests that do.

  • I believe unit tests are tests that contain no environmental dependencies or mocked environmental dependencies.
  • Integration tests are typically those which require some sort of external environment dependency, as it tests the integration.

In /chainlink, we have grouped unit tests and integration tests together in our main validation workflow. Even though we also have other integration and e2e tests run in separate workflows.

This PR attempts to organize tests so unit tests are run independently of integration tests.

Methodology

  • Tests which can be run successfully with only go test ./... and no other application specific environment setup are unit tests.
    • Unit tests will be placed into files name *_unit_test.go, with a build flag of //go:build !integration (or should it be //go:build unit?)
  • Tests which require environment setup will be placed into files name _integ_test.go, and have at least the build flag of //go:build integration.
    • If specific integrations are required then it will have //go:build integration && solana for example
    • Or should this be //go:build integration for generic tests, and //go:build integration-solana for more specific ones

Motivation

  • We have a lot of integration-style tests running alongside our unit tests. These integration tests delay results from unit tests due to environment setup overhead, more test overhead, and longer runtime.
  • I think we should have clearer division between tests with no external dependencies (typically unit tests), and tests that have external dependencies (typically integration tests).
  • I hope dividing these will encourage sophisticated interfaces/mocks allowing for more efficient testing through unit tests

Notes / Side-effects

  • Using build flags seems like the best way to do this. However, as some files contained both unit and integration tests, i've had to split them into different files.
    • Right now I've chosen _unit_test.go and _integ_test.go, this can be changed.
    • One issue is that this will reset branch-out test histories and quarantining as file paths have changed, and the path is part of a test's unique key.

@erikburt erikburt self-assigned this Dec 5, 2025
trunk-io[bot]

This comment was marked as outdated.

@github-actions

github-actions Bot commented Dec 5, 2025 •

Copy link
Copy Markdown
Contributor

CORA - Pending Reviewers

Codeowners Entry Overall Num Files Owners
* ⏳ 1 @smartcontractkit/foundations, @smartcontractkit/core
/deployment ⏳ 5 @smartcontractkit/ccip-tooling, @smartcontractkit/ccip-offchain, @smartcontractkit/keystone, @smartcontractkit/operations-platform, @smartcontractkit/core
/deployment/ccip ⏳ 62 @smartcontractkit/ccip-tooling, @smartcontractkit/ccip-offchain, @smartcontractkit/operations-platform, @smartcontractkit/core
/deployment/ccip/changeset/globals ⏳ 1 @smartcontractkit/ccip-offchain
/deployment/common ⏳ 2 @smartcontractkit/operations-platform
/deployment/data-feeds ⏳ 4 @smartcontractkit/data-feeds-engineers, @smartcontractkit/operations-platform, @smartcontractkit/core
/.github/** ⏳ 4 @smartcontractkit/devex-cicd, @smartcontractkit/devex-tooling, @smartcontractkit/core

Legend: ✅ Approved | ❌ Changes Requested | 💬 Commented | 🚫 Dismissed | ⏳ Pending | ❓ Unknown

For more details, see the full review summary.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

@erikburt erikburt changed the title test: disable unit test setup chore: tag integration tests Dec 5, 2025
trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

Comment on lines +20 to +53
name: List all tag combinations
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 1

- name: List all tags (./core/)
working-directory: ./core
run: |
find . -name '*.go' -print0 \
| xargs -0 grep -hE '^[[:space:]]*//(go:build|\+build)' 2>/dev/null \
| sed -E '
s|^[[:space:]]*//go:build[[:space:]]+||;
s|^[[:space:]]*//\+build[[:space:]]+||;
s/[[:space:]]+$//
' \
| sed -E '/^[[:space:]]*$/d' \
| sort -u

- name: List all tags (./deployment/)
working-directory: ./deployment
run: |
find . -name '*.go' -print0 \
| xargs -0 grep -hE '^[[:space:]]*//(go:build|\+build)' 2>/dev/null \
| sed -E '
s|^[[:space:]]*//go:build[[:space:]]+||;
s|^[[:space:]]*//\+build[[:space:]]+||;
s/[[:space:]]+$//
' \
| sed -E '/^[[:space:]]*$/d' \
| sort -u

tagged-tests:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI 12 months ago

The best way to fix the problem is to explicitly set the permissions for the list-all-tag-combinations job. According to least privilege principles and since this job only checks out code and runs a few shell commands, it only requires read access to repository contents. Therefore, add a permissions block to the job with contents: read as the only permission.
To implement, insert the block under list-all-tag-combinations: (indented to match the other keys), before runs-on: .... No other code changes are needed.


Suggested changeset 1
.github/workflows/ci-core-integration.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/ci-core-integration.yml b/.github/workflows/ci-core-integration.yml
--- a/.github/workflows/ci-core-integration.yml
+++ b/.github/workflows/ci-core-integration.yml
@@ -18,6 +18,8 @@
 jobs:
   list-all-tag-combinations:
     name: List all tag combinations
+    permissions:
+      contents: read
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v5
EOF
@@ -18,6 +18,8 @@
jobs:
list-all-tag-combinations:
name: List all tag combinations
permissions:
contents: read
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
Copilot is powered by AI and may make mistakes. Always verify output.
trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

@trunk-io

trunk-io Bot commented Dec 5, 2025 •

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge   Static Badge

Failed Test Failure Summary Logs
Test_Service_CreateChainConfig/Solana_Chain_Type The test failed because it could not connect to the local PostgreSQL database, causing a failure to load key states. Logs ↗︎
Test_Service_EnableFeedsManager The test failed because it could not connect to the PostgreSQL database, causing key loading to fail. Logs ↗︎
Test_UpsertWorkflowSpec/workflow_is_unique_by_workflow_ID The test failed because it could not connect to the database on localhost, indicating the database server was not running or accessible. Logs ↗︎
TestTerminalAPIInitializer_InitializeWithoutAPIUser The test failed during the initialization of the terminal API without an API user. Logs ↗︎

... and 1484 more

View Full Report ↗︎ ⋅ Docs

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

@trunk-io trunk-io Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Test Results: Incorrect Test Setup

Affected tests:

  1. Workflow Run: Core Tests (go_core_ccip_deployment_tests)

What Broke

The test TestConfigureCCTPMessageTransmitterProxy was miscategorized as a unit test, but its implementation requires an integration test environment.

Proposed Fixes

Recategorize TestConfigureCCTPMessageTransmitterProxy as an integration test by updating its build tag and file name, and add a new entry to the ci-core-integration.yml workflow to ensure it runs in the correct environment with integration && db tags.

git mv deployment/ccip/changeset/v1_6_2/cs_configure_cctp_message_transmitter_proxy_unit_test.go deployment/ccip/changeset/v1_6_2/cs_configure_cctp_message_transmitter_proxy_integ_test.go

In cs_configure_cctp_message_transmitter_proxy_unit_test.go:1

- //go:build !integration
+ //go:build integration && db

In ci-core-integration.yml:70

             setup-plugins: "true"
+ 
+           - module: Deployment
+             cmd: go_core_ccip_deployment_tests
+             os: cpu=32/ram=64/family=c5ad/spot=false/image=ubuntu24-full-x64/extras=s3-cache
+             tags: "integration db"
+             go-mod-directory: "deployment/"
+             test-file-filter: "deployment/ccip/changeset/v1_6_2/cs_configure_cctp_message_transmitter_proxy_integ_test.go"
+             setup-plugins: "true"
Autofix Options

You can use our MCP server to get AI assistance with debugging and fixing these failures.

  • Use MCP in your IDE to debug the issue. Try Help me fix CI failures from YyF8G1uD to get started.

View all test uploads

@trunk-io trunk-io Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Test Results: CI Postgres Misconfiguration

Affected tests:

  1. TestConfigPoller/happy_path_(with_config_store) (Workflow Run: Core Tests (go_core_tests))
  2. TestConfigPoller (Workflow Run: Core Tests (go_core_tests))
  3. TestSequencesHaveTxHash (Workflow Run: Core Tests (go_core_tests))
  4. TestConfigPoller/LatestConfig_errors_if_there_is_no_config_in_logs_and_config_store_is_unconfigured (Workflow Run: Core Tests (go_core_tests))

What Broke

These failures stem from pipeline misconfigurations related to PostgreSQL database setup in the CI environment. Specifically, the ci-core.yml workflow either had its PostgreSQL database setup inadvertently removed or commented out, or certain tests that depend on a PostgreSQL database were not correctly tagged as integration tests, causing them to execute in an environment lacking the necessary database infrastructure.

Proposed Fixes

Re-enable the PostgreSQL database setup in the ci-core.yml workflow and add the //go:build integration && db tag to core/services/relay/evm/config_poller_test.go to ensure it runs with the required database.

In ci-core.yml:291

-       # - name: Setup Postgres
+       - name: Setup Postgres
-       #   if: ${{ matrix.type.should-run == 'true' }}
+         if: ${{ matrix.type.should-run == 'true' }}
-       #   uses: smartcontractkit/.github/actions/setup-postgres@setup-postgres/v1
+         uses: smartcontractkit/.github/actions/setup-postgres@setup-postgres/v1
-       #   with:
+         with:
-       #     tmpfs: "true"
+           tmpfs: "true"
-       #     image-tag: "16-alpine"
+           image-tag: "16-alpine"
 
       - name: Touching core/web/assets/index.html
         if: ${{ matrix.type.should-run == 'true' }}
         run: mkdir -p core/web/assets && touch core/web/assets/index.html
 
       - name: Download Go vendor packages
         if: ${{ matrix.type.should-run == 'true' }}
         run: go mod download
 
-       # - name: Setup DB
+       - name: Setup DB
-       #   if: ${{ matrix.type.should-run == 'true' }}
+         if: ${{ matrix.type.should-run == 'true' }}
-       #   run: go run ./core/store/cmd/preparetest
+         run: go run ./core/store/cmd/preparetest
-       #   env:
+         env:
-       #     CL_DATABASE_URL: ${{ env.DB_URL }}
+           CL_DATABASE_URL: ${{ env.DB_URL }}지를

In config_poller_test.go:1

+ //go:build integration && db
package evm
Autofix Options

You can apply the proposed fixes directly to your branch. Try the following:

  • Comment /trunk stack-fix UN1DIsaD to generate a stacked PR with the proposed fixes.
  • Use MCP in your IDE to fix the issue. Try Help me fix CI failures from UN1DIsaD to get started.

View all test uploads

@trunk-io trunk-io Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Test Results: Missing Integration Tag

Affected tests:

  1. TestEvmWrite (Workflow Run: Core Tests (go_core_tests))

What Broke

The test TestEvmWrite was not correctly tagged as an integration test, leading to its execution in a CI environment without the necessary PostgreSQL database setup.

Proposed Fixes

Tag TestEvmWrite in core/services/relay/evm/write_target_test.go with //go:build integration to ensure it runs in the ci-core-integration.yml workflow, which includes the necessary database setup.

In write_target_test.go:1

+ //go:build integration
 package evm_test
Autofix Options

You can apply the proposed fixes directly to your branch. Try the following:

  • Comment /trunk stack-fix pJUii1LI to generate a stacked PR with the proposed fixes.
  • Use MCP in your IDE to fix the issue. Try Help me fix CI failures from pJUii1LI to get started.

View all test uploads

@cl-sonarqube-production

Copy link
Copy Markdown

@trunk-io trunk-io Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Test Results: Incorrect Test Setup

Affected tests:

  1. TestDeployChainContractsChangesetPreload (Workflow Run: Core Tests (go_core_ccip_deployment_tests))

What Broke

The PR incorrectly re-categorized an integration test as a unit test, causing it to run in CI without the necessary Solana environment setup. This led to a Solana RPC error during contract deployment.

Proposed Fixes

Move the TestDeployChainContractsChangesetPreload test from cs_deploy_chain_unit_test.go to cs_deploy_chain_integ_test.go and add the quarantine import to the latter file.

In cs_deploy_chain_unit_test.go:21

- // use this for a quick deploy test
- func TestDeployChainContractsChangesetPreload(t *testing.T) {
- 	quarantine.Flaky(t, "DX-1729")
- 	t.Parallel()
- 
- 	homeChainSel := chain_selectors.TEST_90000001.Selector
- 	solSelector := chain_selectors.TEST_22222222222222222222222222222222222222222222.Selector
- 	programsPath := t.TempDir()
- 	progIDs := soltestutils.LoadCCIPPrograms(t, programsPath)
- 	env, err := environment.New(t.Context(),
- 		environment.WithEVMSimulated(t, []uint64{homeChainSel}),
- 		environment.WithSolanaContainer(t, []uint64{solSelector}, programsPath, progIDs),
- 		environment.WithLogger(logger.Test(t)),
- 	)
- 	require.NoError(t, err)
- 	testhelpers.RegisterNodes(t, env, 4, homeChainSel)
- 
- 	err = testhelpers.SavePreloadedSolAddresses(*env, solSelector)
- 	require.NoError(t, err)
- 
- 	e := *env
- 
- 	// empty build config means, if artifacts are not present, resolve the artifact from github based on go.mod version
- 	// for a simple local in memory test, they will always be present, because we need them to spin up the in memory chain
- 	e, _, err = commonchangeset.ApplyChangesets(t, e, initialDeployCS(t, e, nil))
- 	require.NoError(t, err)
- 	err = testhelpers.ValidateSolanaState(e, []uint64{solSelector})
- 	require.NoError(t, err)
- }

In cs_deploy_chain_integ_test.go:11

 \t"github.com/gagliardetto/solana-go"
 \tchain_selectors "github.com/smartcontractkit/chain-selectors"
 \t"github.com/stretchr/testify/require"
+ \t"github.com/smartcontractkit/quarantine"
 
 \tcldf_chain "github.com/smartcontractkit/chainlink-deployments-framework/chain"
 \tcldf "github.com/smartcontractkit/chainlink-deployments-framework/deployment"

In cs_deploy_chain_integ_test.go:504

 \t\t\t\tMinDelay: 1 * time.Second,
 \t\t\t},
 \t\t},
 \t),
 \t})
 \trequire.NoError(t, err);
+ }
+ 
+ // use this for a quick deploy test
+ func TestDeployChainContractsChangesetPreload(t *testing.T) {
+ \tquarantine.Flaky(t, "DX-1729")
+ \tt.Parallel()
+ 
+ \thomeChainSel := chain_selectors.TEST_90000001.Selector
+ \tsolSelector := chain_selectors.TEST_22222222222222222222222222222222222222222222.Selector
+ \tprogramsPath := t.TempDir()
+ \tprogIDs := soltestutils.LoadCCIPPrograms(t, programsPath)
+ \tenv, err := environment.New(t.Context(),
+ \t\tenvironment.WithEVMSimulated(t, []uint64{homeChainSel}),
+ \t\tenvironment.WithSolanaContainer(t, []uint64{solSelector}, programsPath, progIDs),
+ \t\tenvironment.WithLogger(logger.Test(t)),
+ \t)
+ \trequire.NoError(t, err)
+ \ttesthelpers.RegisterNodes(t, env, 4, homeChainSel)
+ 
+ \terr = testhelpers.SavePreloadedSolAddresses(*env, solSelector)
+ \trequire.NoError(t, err)
+ 
+ \te := *env
+ 
+ \t// empty build config means, if artifacts are not present, resolve the artifact from github based on go.mod version
+ \t// for a simple local in memory test, they will always be present, because we need them to spin up the in memory chain
+ \te, _, err = commonchangeset.ApplyChangesets(t, e, initialDeployCS(t, e, nil))
+ \trequire.NoError(t, err)
+ \terr = testhelpers.ValidateSolanaState(e, []uint64{solSelector})
+ \trequire.NoError(t, err)
+ }
Autofix Options

You can apply the proposed fixes directly to your branch. Try the following:

  • Comment /trunk stack-fix gXymS1vd to generate a stacked PR with the proposed fixes.
  • Use MCP in your IDE to fix the issue. Try Help me fix CI failures from gXymS1vd to get started.

View all test uploads

@erikburt erikburt closed this Dec 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants