NO-JIRA: fix perf sno timeout failures by gangwgr · Pull Request #2425 · openshift/library-go

gangwgr · 2026-08-13T12:15:43Z

fix perf sno timeout failure
https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-kube-apiserver-operator/2256/pull-ci-openshift-cluster-kube-apiserver-operator-main-e2e-gcp-operator-encryption-perf-single-node/2087793680185298944

Summary by CodeRabbit

Bug Fixes
- Improved handling of temporary service-unavailable errors during encryption operations.
- Polling now logs and retries transient errors instead of stopping immediately.
- Non-transient errors continue to fail promptly.

coderabbitai · 2026-08-13T12:16:02Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3f650fc7-68d5-4f0a-b75c-4e7a72f5c53e

📥 Commits

Reviewing files that changed from the base of the PR and between afd9404 and 4be779e.

📒 Files selected for processing (2)

test/library/encryption/errors.go
test/library/encryption/perf_helpers.go

Walkthrough

The encryption test helpers now classify service-unavailable errors as transient. Operator-condition polling logs and retries transient errors while preserving failure behavior for non-transient errors.

Changes

Transient API retry handling

Layer / File(s)	Summary
Error classification and polling retry `test/library/encryption/errors.go`, `test/library/encryption/perf_helpers.go`	Service-unavailable errors are classified as transient. Polling logs and retries transient operator-condition errors. Non-transient errors still terminate polling.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Mergeability Score: ⚪ Minimal · up to 4be77

This PR makes a localized test-library change, and no actionable merge-blocking risk remains; it is merge-ready after normal checks and review.

Possibly related PRs

openshift/library-go#2424: Directly matches the transient error classification and polling retry changes.

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name	Status	Explanation	Resolution
No-Sensitive-Data-In-Logs	❌ Error	The PR adds t.Logf with raw transient errors; network timeout and connection errors can include internal API hostnames or addresses.	Log only a fixed error category, or sanitize and redact endpoint details before writing the retry message.

✅ Passed checks (14 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The patch changes only transient error handling and polling retries. It adds or alters no Ginkgo test title and introduces no dynamic test-name value.
Test Structure And Quality	✅ Passed	The PR changes only retry classification and transient polling logs; the existing wait.Poll retains a 79-minute timeout, and no new resource setup, assertions, or test blocks were added.
Microshift Test Compatibility	✅ Passed	The patch only changes existing error handling and polling helpers; the diff adds no Ginkgo tests or new MicroShift-incompatible API references.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The patch adds no Ginkgo tests or multi-node assumptions. It only broadens transient error handling and retries operator-condition polling.
Topology-Aware Scheduling Compatibility	✅ Passed	The commit changes only transient API error handling and test polling in test/library/encryption; it adds no manifests, controllers, replicas, affinities, selectors, tolerations, topology spreads,...
Ote Binary Stdout Contract	✅ Passed	The diff adds only transient-error handling and t.Logf inside a polling test helper; it adds no stdout writes or process-level suite setup code.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The PR changes only existing error and polling helpers; the diff adds no Ginkgo tests, IPv4 assumptions, or external connectivity requirements.
No-Weak-Crypto	✅ Passed	The diff only adds service-unavailable retry handling and logging; it introduces no weak cipher, custom crypto, or secret comparison.
Container-Privileges	✅ Passed	The PR changes only two Go files for retry handling; the changed lines add no privileged, host namespace, SYS_ADMIN, or allowPrivilegeEscalation settings.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: fixing timeout failures in SNO performance tests.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

gangwgr · 2026-08-13T12:16:28Z

test run-

oc get no
NAME                                        STATUS   ROLES                         AGE   VERSION
ip-10-0-12-120.us-east-2.compute.internal   Ready    control-plane,master,worker   62m   v1.36.2
rgangwar@rgangwar-mac cluster-kube-apiserver-operator % make test-e2e-encryption-perf-aescbc
ENCRYPTION_PROVIDER=aescbc /Library/Developer/CommandLineTools/usr/bin/make test-e2e-encryption-perf
go test -mod=vendor -race -v -timeout 2h -p 1 ./test/e2e-encryption-perf/... -args -provider=aescbc
=== RUN   TestPerfEncryption
    encryption_perf.go:52: encryption type: aescbc
Found configuration for host https://api.rgang-sno.qe.devcluster.openshift.com:6443.
Aug 13 17:15:28.661: Executing provided load function for 3 workers
Found configuration for host https://api.rgang-sno.qe.devcluster.openshift.com:6443.
Found configuration for host https://api.rgang-sno.qe.devcluster.openshift.com:6443.
Found configuration for host https://api.rgang-sno.qe.devcluster.openshift.com:6443.
Aug 13 17:16:49.424: All workers completed successfully
Aug 13 17:16:49.424: Created 27000 secrets
Aug 13 17:16:49.424: Created 15000 configmaps
Aug 13 17:16:49.424: Populating etcd took 1m20.763190625s
Aug 13 17:16:49.424: Starting encryption e2e test for "aescbc" mode
Aug 13 17:16:49.424: Waiting up to 1h19m0s for the condition "EncryptionMigrationControllerProgressing" with the reason "Migrating" to be set to true
Found configuration for host https://api.rgang-sno.qe.devcluster.openshift.com:6443.
Aug 13 17:16:49.906: Updating encryption configuration for APIServer from v1.APIServerEncryption{Type:"", KMS:v1.KMSPluginConfig{Type:"", Vault:v1.VaultKMSPluginConfig{KMSPluginImage:"", VaultAddress:"", VaultNamespace:"", VaultAuthNamespace:"", TLS:v1.VaultTLSConfig{CABundle:v1.VaultConfigMapReference{Name:""}, ServerName:""}, Authentication:v1.VaultAuthentication{Type:"", AppRole:v1.VaultAppRoleAuthentication{Secret:v1.VaultSecretReference{Name:""}}}, VaultKeyPath:""}}} to v1.APIServerEncryption{Type:"aescbc", KMS:v1.KMSPluginConfig{Type:"", Vault:v1.VaultKMSPluginConfig{KMSPluginImage:"", VaultAddress:"", VaultNamespace:"", VaultAuthNamespace:"", TLS:v1.VaultTLSConfig{CABundle:v1.VaultConfigMapReference{Name:""}, ServerName:""}, Authentication:v1.VaultAuthentication{Type:"", AppRole:v1.VaultAppRoleAuthentication{Secret:v1.VaultSecretReference{Name:""}}}, VaultKeyPath:""}}}
Aug 13 17:16:50.148: Waiting up to 1h19m0s for the next key "encryption-key-openshift-kube-apiserver-1", previous key was "no previous key"
Aug 13 17:17:05.387: Observed key "encryption-key-openshift-kube-apiserver-1", waiting up to 1h19m0s until it will be used to migrate [secrets configmaps]
Aug 13 17:19:20.935: failed to list secrets, err = Get "https://api.rgang-sno.qe.devcluster.openshift.com:6443/api/v1/namespaces/openshift-config-managed/secrets?labelSelector=encryption.apiserver.operator.openshift.io%2Fcomponent%3Dopenshift-kube-apiserver": dial tcp 3.22.193.94:6443: connect: connection refused
Aug 13 17:19:21.774: failed to list secrets, err = Get "https://api.rgang-sno.qe.devcluster.openshift.com:6443/api/v1/namespaces/openshift-config-managed/secrets?labelSelector=encryption.apiserver.operator.openshift.io%2Fcomponent%3Dopenshift-kube-apiserver": dial tcp 3.22.193.94:6443: connect: connection refused
Aug 13 17:19:22.602: failed to list secrets, err = Get "https://api.rgang-sno.qe.devcluster.openshift.com:6443/api/v1/namespaces/openshift-config-managed/secrets?labelSelector=encryption.apiserver.operator.openshift.io%2Fcomponent%3Dopenshift-kube-apiserver": dial tcp 3.22.193.94:6443: connect: connection refused
Aug 13 17:19:23.711: failed to list secrets, err = Get "https://api.rgang-sno.qe.devcluster.openshift.com:6443/api/v1/namespaces/openshift-config-managed/secrets?labelSelector=encryption.apiserver.operator.openshift.io%2Fcomponent%3Dopenshift-kube-apiserver": dial tcp 3.22.193.94:6443: connect: connection refused
Aug 13 17:19:25.757: failed to list secrets, err = Get "https://api.rgang-sno.qe.devcluster.openshift.com:6443/api/v1/namespaces/openshift-config-managed/secrets?labelSelector=encryption.apiserver.operator.openshift.io%2Fcomponent%3Dopenshift-kube-apiserver": dial tcp 3.22.193.94:6443: connect: connection refused
Aug 13 17:19:32.934: failed to list secrets, err = Get "https://api.rgang-sno.qe.devcluster.openshift.com:6443/api/v1/namespaces/openshift-config-managed/secrets?labelSelector=encryption.apiserver.operator.openshift.io%2Fcomponent%3Dopenshift-kube-apiserver": dial tcp 3.22.193.94:6443: connect: connection refused
Aug 13 17:23:20.198: failed to get operator conditions, will retry: Get "https://api.rgang-sno.qe.devcluster.openshift.com:6443/apis/operator.openshift.io/v1/kubeapiservers/cluster": dial tcp 3.22.193.94:6443: connect: connection refused
Aug 13 17:23:20.907: failed to list secrets, err = Get "https://api.rgang-sno.qe.devcluster.openshift.com:6443/api/v1/namespaces/openshift-config-managed/secrets?labelSelector=encryption.apiserver.operator.openshift.io%2Fcomponent%3Dopenshift-kube-apiserver": dial tcp 3.22.193.94:6443: connect: connection refused
Aug 13 17:23:21.665: failed to list secrets, err = Get "https://api.rgang-sno.qe.devcluster.openshift.com:6443/api/v1/namespaces/openshift-config-managed/secrets?labelSelector=encryption.apiserver.operator.openshift.io%2Fcomponent%3Dopenshift-kube-apiserver": dial tcp 3.22.193.94:6443: connect: connection refused
Aug 13 17:23:22.661: failed to list secrets, err = Get "https://api.rgang-sno.qe.devcluster.openshift.com:6443/api/v1/namespaces/openshift-config-managed/secrets?labelSelector=encryption.apiserver.operator.openshift.io%2Fcomponent%3Dopenshift-kube-apiserver": dial tcp 3.22.193.94:6443: connect: connection refused
Aug 13 17:23:23.688: failed to list secrets, err = Get "https://api.rgang-sno.qe.devcluster.openshift.com:6443/api/v1/namespaces/openshift-config-managed/secrets?labelSelector=encryption.apiserver.operator.openshift.io%2Fcomponent%3Dopenshift-kube-apiserver": dial tcp 3.22.193.94:6443: connect: connection refused
Aug 13 17:23:25.774: failed to list secrets, err = Get "https://api.rgang-sno.qe.devcluster.openshift.com:6443/api/v1/namespaces/openshift-config-managed/secrets?labelSelector=encryption.apiserver.operator.openshift.io%2Fcomponent%3Dopenshift-kube-apiserver": dial tcp 3.22.193.94:6443: connect: connection refused
Aug 13 17:24:34.935: EncryptionMigrationControllerProgressing condition observed at 2026-08-13 17:24:33 +0530 IST
Aug 13 17:42:20.333: Key "encryption-key-openshift-kube-apiserver-1" was used to migrate [configmaps secrets]
Aug 13 17:42:20.333: Checking if all Secrets where encrypted/decrypted for "aescbc" mode
W0813 17:42:25.223778   50418 logging.go:55] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:55650", ServerName: "127.0.0.1:55650", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:55650: operation was canceled"
W0813 17:42:46.256818   50418 logging.go:55] [core] [Channel #1 SubChannel #3] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:55650", ServerName: "127.0.0.1:55650", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:55650: connect: connection refused"
Aug 13 17:42:46.264: Verified 27679 Secrets
Aug 13 17:42:46.264: Checking if all ConfigMaps where encrypted/decrypted for "aescbc" mode
W0813 17:42:47.271927   50418 logging.go:55] [core] [Channel #1 SubChannel #3] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:55650", ServerName: "127.0.0.1:55650", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:55650: connect: connection refused"
W0813 17:42:48.893097   50418 logging.go:55] [core] [Channel #1 SubChannel #3] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:55650", ServerName: "127.0.0.1:55650", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:55650: connect: connection refused"
W0813 17:42:49.383727   50418 logging.go:55] [core] [Channel #5 SubChannel #6] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:55690", ServerName: "127.0.0.1:55690", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:55690: operation was canceled"
W0813 17:42:51.338162   50418 logging.go:55] [core] [Channel #1 SubChannel #3] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:55650", ServerName: "127.0.0.1:55650", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:55650: connect: connection refused"
W0813 17:42:54.668434   50418 logging.go:55] [core] [Channel #1 SubChannel #3] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:55650", ServerName: "127.0.0.1:55650", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:55650: connect: connection refused"
W0813 17:42:58.158788   50418 logging.go:55] [core] [Channel #5 SubChannel #7] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:55690", ServerName: "127.0.0.1:55690", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:55690: connect: connection refused"
Aug 13 17:42:58.162: Verified 15558 ConfigMaps
Aug 13 17:42:58.162: Checking if the last migrated key was used to encrypt [secrets configmaps]
Aug 13 17:42:58.397: AssertFunc for TestEncryption scenario with "aescbc" provider took 38.065040917s
Aug 13 17:42:58.397: Checking if "encryption-config-openshift-kube-apiserver" in "openshift-config-managed" has desired GRs [secrets configmaps]
Aug 13 17:42:58.633: migration took 17m47.333111s
--- PASS: TestPerfEncryption (1650.03s)
PASS

openshift-ci-robot · 2026-08-13T12:16:48Z

@gangwgr: This pull request explicitly references no jira issue.

Details

In response to this:

fix perf sno timeout failure
https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-kube-apiserver-operator/2256/pull-ci-openshift-cluster-kube-apiserver-operator-main-e2e-gcp-operator-encryption-perf-single-node/2087793680185298944

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2026-08-13T12:40:27Z

@gangwgr: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

gangwgr · 2026-08-13T13:48:25Z

in ci passed https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-kube-apiserver-operator/2256/pull-ci-openshift-cluster-kube-apiserver-operator-main-e2e-gcp-operator-encryption-perf-single-node/2087858142049210368

ardaguclu · 2026-08-14T12:18:51Z

/lgtm
/approve

openshift-ci · 2026-08-14T12:19:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ardaguclu, gangwgr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~test/library/encryption/OWNERS~~ [ardaguclu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fix perf sno timeout failures

4be779e

openshift-ci Bot requested review from dgrisonnet and p0lyn0mial August 13, 2026 12:16

gangwgr changed the title ~~fix perf sno timeout failures~~ NO-JIRA: fix perf sno timeout failures Aug 13, 2026

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 13, 2026

gangwgr mentioned this pull request Aug 13, 2026

add retries when apiserver down in sno parallel run #2424

Open

openshift-ci Bot assigned ardaguclu Aug 14, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 14, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 14, 2026

openshift-merge-bot Bot merged commit 7b930c0 into openshift:master Aug 14, 2026
5 checks passed

Navigation Menu

Uh oh!

NO-JIRA: fix perf sno timeout failures - #2425

NO-JIRA: fix perf sno timeout failures#2425
openshift-merge-bot[bot] merged 1 commit into
openshift:masterfrom
gangwgr:fix-sno-perf

gangwgr commented Aug 13, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Aug 13, 2026 •

edited

Loading

Pre-merge checks failed

Uh oh!

gangwgr commented Aug 13, 2026

Uh oh!

openshift-ci-robot commented Aug 13, 2026

Uh oh!

openshift-ci Bot commented Aug 13, 2026

Uh oh!

gangwgr commented Aug 13, 2026

Uh oh!

ardaguclu commented Aug 14, 2026

Uh oh!

openshift-ci Bot commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

gangwgr commented Aug 13, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Pre-merge checks failed

❌ Failed checks (1 error)

Uh oh!

gangwgr commented Aug 13, 2026

Uh oh!

openshift-ci-robot commented Aug 13, 2026

Uh oh!

openshift-ci Bot commented Aug 13, 2026

Uh oh!

gangwgr commented Aug 13, 2026

Uh oh!

ardaguclu commented Aug 14, 2026

Uh oh!

openshift-ci Bot commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gangwgr commented Aug 13, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 13, 2026 •

edited

Loading