Skip to content

feat(ci): targeted test suites - #20610

Closed
erikburt wants to merge 30 commits into
developfrom
test/test-suite-membership
Closed

erikburt wants to merge 30 commits into
developfrom
test/test-suite-membership

Conversation

@erikburt

@erikburt erikburt commented Dec 11, 2025 •

Copy link
Copy Markdown
Collaborator

This PR attempts to create more targeted/focused test suites for CI. It does this by adding a specific call to each test.

Current suites:

  • unit
    • No environmental dependencies - these are true unit tests
  • with-db
    • requirers a database
  • test-scripts (core module only)
    • test scripts requires a DB
  • integration (core module only)
    • go:build integration tagged tests, requires DB, LOOPPs, and Solana CLI
  • solana (deployment/ module only)
    • targeting solana integrations, most or all tests require database and solana CLI
  • aptos(deployment/ module only)
    • targeting aptos integrations, most or all tests require database and aptos CLI

Changes

  • Tag/Add all tests to a specific test suite
  • Limit CI Core to running unit only
    • Remove all environment setup from existing matrix job
  • Create new workflow/matrix for the test suites requiring environmental dependencies

Notes

  • I've made sure to introduce no new linting issues, but the changeset is too big for golangci-lint to only show new issues

DX-2467

@erikburt erikburt changed the title feat(ci: test Suite feat(ci): targeted test suites Dec 11, 2025
@github-actions

Copy link
Copy Markdown
Contributor

I see you updated files related to core. Please run pnpm changeset in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

@erikburt
erikburt force-pushed the test/test-suite-membership branch from 584a79a to be5df1a Compare December 11, 2025 19:48
trunk-io[bot]

This comment was marked as outdated.

@github-actions

github-actions Bot commented Dec 11, 2025 •

Copy link
Copy Markdown
Contributor

CORA - Pending Reviewers

Codeowners Entry Overall Num Files Owners
* ⏳ 266 @smartcontractkit/foundations, @smartcontractkit/core
/core/services/directrequest ⏳ 2 @smartcontractkit/foundations
/core/services/feeds ⏳ 5 @smartcontractkit/core, @smartcontractkit/devex-tooling
/core/capabilities/ ⏳ 40 @smartcontractkit/keystone, @smartcontractkit/capabilities-team
/core/capabilities/ccip ⏳ 32 @smartcontractkit/ccip-offchain
/core/capabilities/ccip/ccipsolana ⏳ 6 @smartcontractkit/bix-build
/core/services/fluxmonitorv2 ⏳ 11 @smartcontractkit/foundations, @smartcontractkit/core
/core/services/job ⏳ 10 @smartcontractkit/foundations, @smartcontractkit/core
/core/services/keystore ⏳ 61 @smartcontractkit/foundations
/core/services/ocr* ⏳ 41 @smartcontractkit/foundations, @smartcontractkit/core
/core/services/periodicbackup ⏳ 1 @smartcontractkit/foundations, @smartcontractkit/core
/core/services/pg ⏳ 3 @smartcontractkit/foundations, @smartcontractkit/core
/core/services/pipeline ⏳ 45 @smartcontractkit/foundations, @smartcontractkit/bix-framework, @smartcontractkit/core
/core/services/telemetry ⏳ 4 @smartcontractkit/data-tooling, @smartcontractkit/core
/core/services/relay/evm/mercury ⏳ 19 @smartcontractkit/data-streams-engineers, @smartcontractkit/core
/core/services/webhook ⏳ 4 @smartcontractkit/foundations, @smartcontractkit/bix-framework, @smartcontractkit/core
/core/services/llo ⏳ 22 @smartcontractkit/data-streams-engineers, @smartcontractkit/core
/core/services/ccip ⏳ 2 @smartcontractkit/ccip
/core/services/ocr2/plugins/ccip ⏳ 62 @smartcontractkit/ccip
/core/services/vrf ⏳ 23 @smartcontractkit/dev-services, @smartcontractkit/core
/core/services/blockhashstore ⏳ 4 @smartcontractkit/dev-services, @smartcontractkit/core
/core/services/blockheaderfeeder ⏳ 2 @smartcontractkit/dev-services, @smartcontractkit/core
/core/services/keeper ⏳ 11 @smartcontractkit/dev-services
/core/services/ocr2/plugins/ocr2keeper ⏳ 40 @smartcontractkit/dev-services
core/services/functions ⏳ 5 @smartcontractkit/dev-services
core/services/ocr2/plugins/functions ⏳ 6 @smartcontractkit/dev-services
core/services/s4 ⏳ 6 @smartcontractkit/dev-services
core/services/ocr2/plugins/threshold ⏳ 1 @smartcontractkit/dev-services
core/services/relay/evm/functions ⏳ 3 @smartcontractkit/dev-services
/core/services/registrysyncer ⏳ 6 @smartcontractkit/keystone
/core/services/workflows ⏳ 33 @smartcontractkit/keystone
/core/services/standardcapabilities ⏳ 2 @smartcontractkit/keystone
/core/web/resolver ⏳ 30 @smartcontractkit/foundations, @smartcontractkit/core
/integration-tests/**/*ccip* ⏳ 1 @smartcontractkit/ccip-offchain, @smartcontractkit/core, @smartcontractkit/ccip
/deployment ⏳ 25 @smartcontractkit/ccip-tooling, @smartcontractkit/ccip-offchain, @smartcontractkit/keystone, @smartcontractkit/operations-platform, @smartcontractkit/core
/deployment/ccip ⏳ 64 @smartcontractkit/ccip-tooling, @smartcontractkit/ccip-offchain, @smartcontractkit/operations-platform, @smartcontractkit/core
/deployment/ccip/changeset/globals ⏳ 1 @smartcontractkit/ccip-offchain
/deployment/common ⏳ 32 @smartcontractkit/operations-platform
/deployment/cre ⏳ 36 @smartcontractkit/keystone, @smartcontractkit/operations-platform
/deployment/data-feeds ⏳ 28 @smartcontractkit/data-feeds-engineers, @smartcontractkit/operations-platform, @smartcontractkit/core
/deployment/keystone ⏳ 35 @smartcontractkit/keystone, @smartcontractkit/operations-platform, @smartcontractkit/core
/deployment/vault ⏳ 4 @smartcontractkit/cld-vault, @smartcontractkit/operations-platform, @smartcontractkit/core
/.github/** ⏳ 4 @smartcontractkit/devex-cicd, @smartcontractkit/devex-tooling, @smartcontractkit/core
/tools/plugout ⏳ 1 @smartcontractkit/devex-cicd
go.mod ⏳ 6 @smartcontractkit/core, @smartcontractkit/foundations
go.sum ⏳ 6 @smartcontractkit/core, @smartcontractkit/foundations
integration-tests/go.mod ⏳ 1 @smartcontractkit/core, @smartcontractkit/devex-tooling, @smartcontractkit/foundations
integration-tests/go.sum ⏳ 1 @smartcontractkit/core, @smartcontractkit/devex-tooling, @smartcontractkit/foundations

Legend: ✅ Approved | ❌ Changes Requested | 💬 Commented | 🚫 Dismissed | ⏳ Pending | ❓ Unknown

For more details, see the full review summary.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

@erikburt
erikburt force-pushed the test/test-suite-membership branch from b88a8ff to dee1156 Compare December 11, 2025 22:25
trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

@erikburt erikburt added the allow-lint-issues Allow merging despite linter issues. Do not abuse. label Dec 11, 2025
trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

@erikburt
erikburt force-pushed the test/test-suite-membership branch from d2690b1 to a9db8f3 Compare December 12, 2025 17:21
trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

trunk-io[bot]

This comment was marked as outdated.

@trunk-io trunk-io Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Test Results: Pipeline Env Var Missing

Affected failures:

  1. Workflow Run: Core Tests (go_core_fuzz)

What Broke

The CL_CURRENT_TEST_SUITE environment variable was not correctly propagated to the go_core_fuzz job, leading to the execution of an unfiltered fuzz test that exceeded its time limit.

Proposed Fixes

Update the ci-core.yml workflow to correctly set the CL_CURRENT_TEST_SUITE environment variable for the go_core_fuzz job and annotate the FuzzStarknetKeyring_Marshal test with tests.BelongsToCISuite("unit") for proper test suite filtering.

In ci-core.yml:212

           - cmd: go_core_fuzz
             os: ${{ needs.runner-config.outputs.core-fuzz-tests-runner }}
             should-run: ${{ needs.filter.outputs.should-run-core-fuzz-tests }}
+             test-suite: "unit"
             # trunk-auto-quarantine: "true"

In ocr2key_test.go:201

- func FuzzStarknetKeyring_Marshal(f *testing.F) {
+ func FuzzStarknetKeyring_Marshal(f *testing.F) {
+ 	tests.BelongsToCISuite(t, "unit")
Autofix Options

You can apply the proposed fixes directly to your branch. Try the following:

  • Comment /trunk stack-fix yA1glUKU to generate a stacked PR with the proposed fixes.
  • Use MCP in your IDE to fix the issue. Try Help me fix CI failures from yA1glUKU to get started.

Tip

Get Better Results: This CI job is not uploading test reports. Adding structured test reports enables more precise, test-level analysis with better root cause identification and more targeted fix recommendations.
👉🏻 Learn how to upload test results.

@cl-sonarqube-production

Copy link
Copy Markdown

@erikburt erikburt closed this Dec 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

allow-lint-issues Allow merging despite linter issues. Do not abuse.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant