Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/gold-turtles-chew.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"chainlink": minor
---

postgres: adding support for v18, removing support for v14 #nops
4 changes: 2 additions & 2 deletions .github/workflows/cre-system-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ jobs:
# http://docs.github.com/en/actions/how-tos/deploy/configure-and-manage-deployments/control-deployments#using-environments-without-deployments
name: integration
deployment: false
timeout-minutes: 10
timeout-minutes: 13
env:
ENABLE_AUTO_QUARANTINE: "true"
BILLING_PLATFORM_SERVICE_IMAGE:
Expand Down Expand Up @@ -415,7 +415,7 @@ jobs:
continue-on-error: ${{ env.ENABLE_AUTO_QUARANTINE == 'true' }}
env:
TEST_NAME: ${{ matrix.tests.test_name }}
TEST_TIMEOUT: 7m # let's leave 3 minutes for other steps (the whole job times out after 10 minutes)
TEST_TIMEOUT: 10m # let's leave 3 minutes for other steps (the whole job times out after 13 minutes)
RUN_QUARANTINED_TESTS: "true" # always run quarantined tests in CI
TOPOLOGY_NAME: ${{ matrix.tests.topology }}
CONFIDENTIAL_COMPUTE_ROOT: ${{ github.workspace }}/chainlink-confidential-compute
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ regarding Chainlink social accounts, news, and networking.
- Example Path for macOS `export PATH=$GOPATH/bin:$PATH` & `export GOPATH=/Users/$USER/go`
2. Install [NodeJS v20](https://nodejs.org/en/download/package-manager/) & [pnpm v10 via npm](https://pnpm.io/installation#using-npm).
- It might be easier long term to use [nvm](https://nodejs.org/en/download/package-manager/#nvm) to switch between node versions for different projects. For example, assuming $NODE_VERSION was set to a valid version of NodeJS, you could run: `nvm install $NODE_VERSION && nvm use $NODE_VERSION`
3. Install [Postgres (>= 12.x)](https://wiki.postgresql.org/wiki/Detailed_installation_guides). It is recommended to run the latest major version of postgres.
- Note if you are running the official Chainlink docker image, the highest supported Postgres version is 17.x due to the bundled client.
3. Install [Postgres (>= 15.x)](https://wiki.postgresql.org/wiki/Detailed_installation_guides). It is recommended to run the latest major version of postgres.
- Note if you are running the official Chainlink docker image, the highest supported Postgres version is 18.x due to the bundled client.
- You should [configure Postgres](https://www.postgresql.org/docs/current/ssl-tcp.html) to use SSL connection (or for testing you can set `?sslmode=disable` in your Postgres query string).
4. Download Chainlink: `git clone https://github.com/smartcontractkit/chainlink && cd chainlink`
5. Build and install Chainlink: `make install`
Expand Down
2 changes: 1 addition & 1 deletion core/chainlink.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \
&& echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] \
https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" \
>/etc/apt/sources.list.d/pgdg.list \
&& apt-get update && apt-get install -y --no-install-recommends postgresql-client-17=17.10-1.pgdg24.04+1 \
&& apt-get update && apt-get install -y --no-install-recommends postgresql-client-18 \
&& rm -rf /var/lib/apt/lists/*

# Prod images (CHAINLINK_USER=chainlink) run as UID:GID 14933:14933 for deterministic
Expand Down
2 changes: 1 addition & 1 deletion core/config/docs/secrets.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[Database]
# URL is the PostgreSQL URI to connect to your database. Chainlink nodes require Postgres versions >= 11. See
# URL is the PostgreSQL URI to connect to your database. Chainlink nodes require Postgres versions >= 15. See
# [Running a Chainlink Node](https://docs.chain.link/docs/running-a-chainlink-node/#set-the-remote-database_url-config) for an example.
#
# Environment variable: `CL_DATABASE_URL`
Expand Down
43 changes: 43 additions & 0 deletions core/scripts/cre/environment/docs/cresettings-override-README.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,49 @@ say *what* to change; the helper makes sure it lands everywhere it could be read
(Internally it still merges each DON's own boot baseline, since DONs can boot with
different `CL_CRE_SETTINGS` — but that's handled for you.)

## Mixed-env caveat: a NEW flag will diverge (guard the test)

[Mixed-env](./mixed-env.md) runs each multi-node DON as **2 nodes on the PR image + 2 on
the develop/baseline image** and fails the run if the two halves disagree. That interacts
with this helper in one specific case:

- **Overriding a flag that already exists in the baseline image** → fine. Both halves know
the key and behave the same; no divergence.
- **Overriding a flag that is NEW in your PR** (you added the `cresettings` key *and* the
code that reads it in the same PR) → **the 2 PR nodes act on it, the 2 baseline nodes
cannot** (their binary has neither the key nor a reader — the override is validated only
in the test process against the PR schema, delivered to every node, then silently ignored
on the baseline nodes). The DON diverges 2-vs-2, which trips the mixed-env non-determinism
gate (a consensus stall and/or the post-run log-marker scan) — a **false failure**, since
your PR is correct.

For that second case, guard the test so it does not run under mixed-env:

```go
func Test_CRE_MyNewFlag(t *testing.T) {
// Skips ONLY under mixed-env; the test still runs in full under the normal
// single-image suite. Call it before deploy/register/ApplyCRESettings.
t_helpers.SkipIfMixedEnv(t, "enables PerWorkflow.<NewFlag>, new in this PR; baseline "+
"nodes lack it and would diverge. Remove once <NewFlag> is in the release baseline.")

testEnv := t_helpers.SetupTestEnvironmentWithPerTestKeys(t, t_helpers.GetDefaultTestConfig(t))
t_helpers.ApplyCRESettings(t, testEnv, t_helpers.Workflow(myWorkflowID, "..."))
// ...
}
```

> **⚠️ Do not reach for `SkipIfMixedEnv` unless you know precisely why *this* test must skip
> mixed-env.** Mixed-env is a required merge gate for cross-version compatibility; every skip
> is a hole in it. It is justified **only** when the test deliberately exercises behavior
> that exists in the PR image but not the baseline (a brand-new flag or capability), so a
> 2-vs-2 split is unavoidable and expected — **never** to silence a real divergence a
> reviewer should see. The `reason` is mandatory; note removing the guard once the flag
> reaches the baseline image. See [mixed-env.md](./mixed-env.md#skipping-a-single-test).

(Enabling a new flag via a topology's boot `CL_CRE_SETTINGS` rather than this helper diverges
from boot regardless of any test — this guard won't help there; keep the new flag out of the
mixed-env topology instead.)

## Run these serially — the override guard

Because overrides mutate settings on the shared environment, **only one override may be
Expand Down
34 changes: 34 additions & 0 deletions core/scripts/cre/environment/docs/mixed-env.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,40 @@ These lines are **silent** when every node runs the same code, so a single occur
- It flags **any** divergence from `develop` — including **intentional** report/payload changes. Those are genuine incompatibilities: if the change is deliberate and will be rolled out safely, bypass the check with the [`skip-mixed-env` label](#required-check--emergency-bypass) rather than "fixing" it.
- It compares against the `develop` commit your PR is based on — not the very latest `develop`. So a change that only conflicts with `develop` commits landed **after** you branched is caught by the nightly full-matrix sweep (which pins the latest `develop`), not by the per-PR run. Keep your branch reasonably current for the tightest signal.

## Skipping a single test

Sometimes a test *legitimately* cannot pass under mixed-env because it deliberately
exercises behavior that exists in the PR image but **not** the baseline image — most commonly
a **brand-new `cresettings` flag** (added, and enabled via `ApplyCRESettings`, in the same PR)
or a **newly added capability**. The 2 PR nodes act on it while the 2 baseline nodes cannot,
so the DON diverges 2-vs-2 and the check fires — even though the PR is correct. (See the
[CRE settings override caveat](./cresettings-override-README.md#mixed-env-caveat-a-new-flag-will-diverge-guard-the-test).)

For that case, guard the individual test with the helper in the test-helpers package:

```go
func Test_CRE_MyNewThing(t *testing.T) {
t_helpers.SkipIfMixedEnv(t, "enables <NewFlag>, new in this PR; baseline nodes lack it "+
"and would diverge. Remove once <NewFlag> is in the release baseline.")
// ... rest of the test ...
}
```

It skips **only** under mixed-env (it keys on `TOPOLOGY_NAME`); the test still runs in full
under the normal single-image suite, and mixed-env covers it again automatically once the
flag/behavior lands in the baseline image. Place the call at the top of the test/scenario,
before any deploy/register/`ApplyCRESettings`, so the skipped test emits no divergent traffic.
For one scenario inside a bucket suite, put it inside that scenario's `t.Run(...)` closure.

> **⚠️ Do not reach for `SkipIfMixedEnv` unless you know precisely why *this* test must skip
> mixed-env.** Every skip is a hole in a required cross-version gate. Use it **only** when a
> 2-vs-2 split is unavoidable and expected (new-in-PR flag or capability) — **never** to
> silence a real divergence a reviewer should see. If the divergence is an *intentional but
> already-in-baseline* report/payload change affecting a whole run, prefer the
> [`skip-mixed-env` label](#required-check--emergency-bypass) instead. To drop a test from
> mixed-env entirely (not just skip at runtime), remove its name from `defaultCREMixedEnvTests`
> in `tools/ci/internal/matrix/mixedenv.go`.

## When it runs

- **CI:** automatically on CRE-affecting PRs, in its **own** workflow `.github/workflows/cre-mixed-env-tests.yaml` (kept separate from `cre-system-tests.yaml` so that file stays simple). It runs the OCR3/DON2DON-heavy tests — `Test_CRE_V2_Suite_Bucket_A`, `Test_CRE_V2_Suite_Bucket_B`, and the `Test_CRE_V2_EVM_Read_*` suite — under mixed-env. The PR image and the develop image are both already built (per-PR and nightly), so no extra image builds are added. A dedicated **Check for non-determinism** step scans the live node containers after the suite and fails the job on any marker — kept separate from the auto-quarantined test step so the failure can't be swallowed.
Expand Down
2 changes: 1 addition & 1 deletion docs/SECRETS.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ AllowSimplePasswords = false # Default
```toml
URL = "postgresql://user:pass@localhost:5432/dbname?sslmode=disable" # Example
```
URL is the PostgreSQL URI to connect to your database. Chainlink nodes require Postgres versions >= 11. See
URL is the PostgreSQL URI to connect to your database. Chainlink nodes require Postgres versions >= 15. See
[Running a Chainlink Node](https://docs.chain.link/docs/running-a-chainlink-node/#set-the-remote-database_url-config) for an example.

Environment variable: `CL_DATABASE_URL`
Expand Down
6 changes: 3 additions & 3 deletions plugins/chainlink.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -126,9 +126,9 @@ RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \
--keyring /usr/share/keyrings/postgresql-archive-keyring.gpg \
--fingerprint B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8 \
&& echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] \
https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" \
>/etc/apt/sources.list.d/pgdg.list \
&& apt-get update && apt-get install -y --no-install-recommends postgresql-client-17=17.10-1.pgdg24.04+1 \
https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" \
>/etc/apt/sources.list.d/pgdg.list \
&& apt-get update && apt-get install -y --no-install-recommends postgresql-client-18 \
&& rm -rf /var/lib/apt/lists/*

# Prod images (CHAINLINK_USER=chainlink) run as UID:GID 14933:14933 for deterministic ownership.
Expand Down
4 changes: 1 addition & 3 deletions system-tests/tests/smoke/cre/cre_suite_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -164,9 +164,7 @@ func runSuiteScenario(t *testing.T, topology string, scenario suite_config.Suite
})
case suite_config.SuiteScenarioDONTime:
t.Run("DON Time - "+topology, func(t *testing.T) {
if nonDeterminismCheckEnabled() {
t.Skip("DONTime registry launch requires all nodes to support jobless capability startup")
}
t_helpers.SkipIfMixedEnv(t, "DONTime registry launch requires all nodes to support jobless capability startup")
if parallelEnabled {
t.Parallel()
}
Expand Down
2 changes: 1 addition & 1 deletion system-tests/tests/smoke/cre/nondeterminism_check_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ func nonDeterminismCheckEnabled() bool {
if strings.EqualFold(os.Getenv("CRE_NONDETERMINISM_CHECK"), "true") {
return true
}
return strings.Contains(strings.ToLower(os.Getenv("TOPOLOGY_NAME")), "mixed-env")
return t_helpers.IsMixedEnvTopology()
}

// reportNonDeterminism scans all container logs for the markers and prints any
Expand Down
25 changes: 25 additions & 0 deletions system-tests/tests/test-helpers/mixed_env_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package helpers

import "testing"

func TestIsMixedEnvTopology(t *testing.T) {
cases := []struct {
name string
topologyName string
want bool
}{
{name: "empty", topologyName: "", want: false},
{name: "single-image topology", topologyName: "workflow-gateway-capabilities-don", want: false},
{name: "exact mixed-env", topologyName: "mixed-env", want: true},
{name: "mixed-env confidential variant", topologyName: "mixed-env-confidential-workflows", want: true},
{name: "case-insensitive", topologyName: "MIXED-ENV", want: true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Setenv("TOPOLOGY_NAME", tc.topologyName)
if got := IsMixedEnvTopology(); got != tc.want {
t.Fatalf("IsMixedEnvTopology() with TOPOLOGY_NAME=%q = %v, want %v", tc.topologyName, got, tc.want)
}
})
}
}
33 changes: 33 additions & 0 deletions system-tests/tests/test-helpers/t_helpers.go
Original file line number Diff line number Diff line change
Expand Up @@ -967,6 +967,39 @@ func ParallelEnabled() bool {
return v == "1" || v == "true" || v == "yes"
}

// IsMixedEnvTopology reports whether the suite is running under a mixed-env topology,
// i.e. each multi-node DON runs 2 nodes on the PR image and 2 on the develop/baseline
// image. CI sets TOPOLOGY_NAME=mixed-env for these runs. This is the signal that a DON
// can legitimately behave two ways at once, so it is the correct gate for
// SkipIfMixedEnv. See core/scripts/cre/environment/docs/mixed-env.md.
func IsMixedEnvTopology() bool {
return strings.Contains(strings.ToLower(os.Getenv("TOPOLOGY_NAME")), "mixed-env")
}

// SkipIfMixedEnv skips the current test when running under a mixed-env topology (see
// IsMixedEnvTopology). Call it at the very top of the test/scenario body, BEFORE any
// deploy/register/ApplyCRESettings, so the skipped test emits no divergent traffic.
//
// WARNING: do NOT use this unless you know precisely why THIS test must skip mixed-env.
// Mixed-env is a required merge gate that catches cross-version consensus/DON2DON
// divergence between PR and develop nodes; every skip is a hole in that coverage. It is
// justified ONLY when the test deliberately exercises behavior that exists in the PR
// image but not the baseline image — e.g. a brand-new CRE settings flag enabled via
// ApplyCRESettings, or a newly added capability — so a 2-vs-2 split is unavoidable and
// expected. It is NOT a way to silence a real divergence a reviewer should see. The test
// still runs in full under the normal single-image suite; only the (impossible)
// cross-version check is skipped, and mixed-env covers it again automatically once the
// flag/behavior lands in the baseline image.
//
// reason is mandatory: state why the divergence is expected, and note removing this guard
// once the flag/behavior reaches the baseline image.
func SkipIfMixedEnv(t *testing.T, reason string) {
t.Helper()
if IsMixedEnvTopology() {
t.Skipf("skipping in mixed-env: %s", reason)
}
}

// ─── Stellar ReadContract test helpers (shared by smoke + regression) ───
//
// These are the generic, chain-family-agnostic bits reused by both the smoke
Expand Down
Loading