From 2d622bb051466c097248bbdc0f78c824937f6c3e Mon Sep 17 00:00:00 2001 From: Tiger Kaovilai Date: Tue, 11 Aug 2026 19:25:24 -0400 Subject: [PATCH 1/2] gather-extra: don't let PLATFORM lookup abort entire gather step The bare 'oc get infrastructure cluster' call for PLATFORM detection is missing --insecure-skip-tls-verify/--request-timeout used by every sibling oc call in this script, and unlike them isn't tolerant of failure. Because ci-operator wraps multi-stage step commands in an ambient 'set -e', a single transient API-server hiccup on this one line aborts the entire 886-line gather-extra step immediately, losing all other diagnostic artifact gathering. Confirmed via a rehearse run on openshift/release#83282 where the raw gather-extra pod log showed only 'Gathering artifacts ...' followed by 'Unable to connect to the server: context deadline exceeded' with a ~12s total runtime -- consistent with this exact line failing before any of the backgrounded queue() calls could run. A sibling periodic job's gather-extra run succeeded cleanly in the same timeframe, confirming this is an intermittent API-server blip rather than a chronic failure. Signed-off-by: Tiger Kaovilai --- ci-operator/step-registry/gather/extra/gather-extra-commands.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci-operator/step-registry/gather/extra/gather-extra-commands.sh b/ci-operator/step-registry/gather/extra/gather-extra-commands.sh index 4d7ad29aa3b48..abfe0eb88074c 100755 --- a/ci-operator/step-registry/gather/extra/gather-extra-commands.sh +++ b/ci-operator/step-registry/gather/extra/gather-extra-commands.sh @@ -42,7 +42,7 @@ oc --insecure-skip-tls-verify --request-timeout=5s get pods -l openshift.io/comp oc --insecure-skip-tls-verify --request-timeout=5s adm inspect clusteroperators --dest-dir ${ARTIFACT_DIR}/inspect || true -PLATFORM=$(oc get infrastructure cluster -o jsonpath="{.status.platform}") +PLATFORM=$(oc --insecure-skip-tls-verify --request-timeout=5s get infrastructure cluster -o jsonpath="{.status.platform}") || true CAPI_PLATFORM=$(echo "$PLATFORM" | tr '[:upper:]' '[:lower:]') if [[ "${CAPI_PLATFORM}" == "baremetal" ]]; then From 8830df5b23fcb49fcb5336f64e5b3630f6ec7f43 Mon Sep 17 00:00:00 2001 From: Tiger Kaovilai Date: Tue, 11 Aug 2026 21:37:47 -0400 Subject: [PATCH 2/2] gather-extra: download jq into ARTIFACT_DIR, not /tmp The cluster-operator-status-to-JSON fallback path downloads a jq binary to /tmp and chmods it exec, then invokes it directly. Some CI nodes mount /tmp noexec, so the container fails with exit 126 "/tmp/jq: Permission denied" even though the script otherwise ran fine. Confirmed node-dependent rather than chronic: a rehearse run on openshift/release#83286 (pull-ci-openshift-oadp-operator-oadp-1.5-4.20-e2e-test-aws) hit this, while sibling jobs on the same commit (4.20-e2e-test-cli-aws, 4.20-e2e-test-hcp-aws) ran the identical script without issue. ARTIFACT_DIR is already used elsewhere in this script and is not subject to the same noexec mount, so download+chmod+exec jq there instead. Signed-off-by: Tiger Kaovilai --- .../step-registry/gather/extra/gather-extra-commands.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ci-operator/step-registry/gather/extra/gather-extra-commands.sh b/ci-operator/step-registry/gather/extra/gather-extra-commands.sh index abfe0eb88074c..678c2dc37f028 100755 --- a/ci-operator/step-registry/gather/extra/gather-extra-commands.sh +++ b/ci-operator/step-registry/gather/extra/gather-extra-commands.sh @@ -768,12 +768,12 @@ else source "${SHARED_DIR}/unset-proxy.sh" fi # This is a temporary conversion of cluster operator status to JSON matching the upgrade - may be moved to code in the future - curl -sL https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 >/tmp/jq && chmod ug+x /tmp/jq + curl -sL https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 >${ARTIFACT_DIR}/jq && chmod ug+x ${ARTIFACT_DIR}/jq if test -f "${SHARED_DIR}/proxy-conf.sh"; then # shellcheck disable=SC1090 source "${SHARED_DIR}/proxy-conf.sh" fi - <${ARTIFACT_DIR}/clusteroperators.json /tmp/jq -r 'def one(condition; t): t as $t | first([.[] | select(condition)] | map(.type=t)[]) // null; def msg: "Operator \(.type) (\(.reason)): \(.message)"; def xmlfailure: if .failure then "\(.failure | @html)" else "" end; def xmltest: "\( xmlfailure )"; def withconditions: map({name: "operator conditions \(.metadata.name)"} + ((.status.conditions // [{type:"Available",status: "False",message:"operator is not reporting conditions"}]) | (one(.type=="Available" and .status!="True"; "unavailable") // one(.type=="Degraded" and .status=="True"; "degraded") // one(.type=="Progressing" and .status=="True"; "progressing") // null) | if . then {failure: .|msg} else null end)); .items | withconditions | "\n\( [.[] | xmltest] | join("\n"))\n"' >${ARTIFACT_DIR}/junit/junit_install_status.xml + <${ARTIFACT_DIR}/clusteroperators.json ${ARTIFACT_DIR}/jq -r 'def one(condition; t): t as $t | first([.[] | select(condition)] | map(.type=t)[]) // null; def msg: "Operator \(.type) (\(.reason)): \(.message)"; def xmlfailure: if .failure then "\(.failure | @html)" else "" end; def xmltest: "\( xmlfailure )"; def withconditions: map({name: "operator conditions \(.metadata.name)"} + ((.status.conditions // [{type:"Available",status: "False",message:"operator is not reporting conditions"}]) | (one(.type=="Available" and .status!="True"; "unavailable") // one(.type=="Degraded" and .status=="True"; "degraded") // one(.type=="Progressing" and .status=="True"; "progressing") // null) | if . then {failure: .|msg} else null end)); .items | withconditions | "\n\( [.[] | xmltest] | join("\n"))\n"' >${ARTIFACT_DIR}/junit/junit_install_status.xml fi # This is an experimental wiring of autogenerated failure detection.