Debug ci timed out, do not merge by TristanCacqueray · Pull Request #1805 · containers/toolbox

TristanCacqueray · 2026-06-09T06:31:36Z

No description provided.

gemini-code-assist

Code Review

This pull request updates the playbooks/system-test-commands-options.yaml playbook to include additional flags (--show-output-of-passing-tests, --timing, and --trace) for the bats command when running the commands-options system tests. No review comments were provided, and there is no feedback to address.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

centosinfra-prod-github-app · 2026-06-09T08:34:22Z

Build failed.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/a428282564314caf8df44f784676a98e

debarshiray · 2026-06-11T12:56:17Z

This is where it gets stuck:

fedora-rawhide | ...
fedora-rawhide | $ _setup_docker_registry
fedora-rawhide | ...
fedora-rawhide | $$ run podman --root "${DOCKER_REG_ROOT}" run --detach --env REGISTRY_AUTH=htpasswd --env REGISTRY_AUTH_HTPASSWD_PATH="/auth/htpasswd" --env REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" --env REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt --env REGISTRY_HTTP_TLS_KEY=/certs/domain.key --name "${DOCKER_REG_NAME}" --privileged --publish 50000:5000 --rm --volume "${DOCKER_REG_AUTH_DIR}":/auth --volume "${DOCKER_REG_CERTS_DIR}":/certs "${IMAGES[docker-reg]}"
fedora-rawhide | ...
fedora-rawhide | $$ run podman login --authfile "${BATS_SUITE_TMPDIR}/authfile.json" --username user --password user "${DOCKER_REG_URI}"
fedora-rawhide | ...
fedora-rawhide | $$ run skopeo copy --dest-authfile "${BATS_SUITE_TMPDIR}/authfile.json" dir:"${IMAGE_CACHE_DIR}"/fedora-toolbox-34 docker://"${DOCKER_REG_URI}"/fedora-toolbox:34

The test suite runs its own Docker registry using podman ... run ..., and populates it with an old fedora-toolbox:34 image using skopeo copy .... The old image is already downloaded to disk in $IMAGE_CACHE_DIR, so it should be a local operation.

debarshiray

I added --show-output-of-passing-tests --timing --trace to playbooks/system-test-runtime-environment-arch-fedora.yaml and playbooks/system-test-runtime-environment-ubuntu.yaml to see why the other jobs aren't getting stuck.

centosinfra-prod-github-app · 2026-06-11T15:06:58Z

Build failed.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/cd25827a44ea4ebf843c8ae3e2fdd479

centosinfra-prod-github-app · 2026-06-12T23:00:05Z

Build failed.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/e4a149d8cf6d471ab2bb886305088204

Rolv-Apneseth · 2026-06-18T16:26:51Z

Maybe this is a race where the push starts before the registry container is actually ready. WDYT about introducing some way to wait on the registry? Maybe something like:

local -i retries
for ((retries = 0; retries < 30; retries++)); do
    if curl -fsk https://localhost:50000/v2/ > /dev/null 2>&1; then
        break
    fi
    sleep 1
done

TristanCacqueray · 2026-06-18T17:52:53Z

It looks like the job is hanging on a skopeo command, perhaps this could also be reported upstream?

Rolv-Apneseth · 2026-06-18T20:47:07Z

It looks like the job is hanging on a skopeo command, perhaps this could also be reported upstream?

Yeah probably would be a good idea.

These skopeo calls should probably use --command-timeout at the very least so they can't hang forever.

Rolv-Apneseth · 2026-06-18T21:22:19Z

podman-container-tools/skopeo#2878 is potentially related, with podman-container-tools/container-libs#777 being the proposed solution there

debarshiray · 2026-06-25T11:14:30Z

Sorry, I was away for a week at DevConf.cz and Flock.

These skopeo calls should probably use --command-timeout at the very least so they can't hang forever.

That's a great idea, @Rolv-Apneseth ! I didn't know about skopeo --command-timeout ..., and this Skopeo issue is a good opportunity to try it out. Do you want to submit a pull request for it?

I see 4 instances of skopeo copy .... Of those, I think the one in _pull_and_cache_distro_image() downloads the image over the network, and the other 3 should be doing local I/O and a timeout definitely makes sense.

For the local copies, we could go with a timeout of 25 seconds. We use it elsewhere in Toolbx because it's the default for D-Bus method calls in most implementations.

That said, I wonder if that will be enough for most storage hardware. We recommend and try not to use tmpfs for IMAGE_CACHE_DIR, and the user's home directory is usually not tmpfs.

I don't know about the one that downloads the image over the network. If we do use a timeout, then it should probably be longer. The images are big enough that 25s won't be enough without very good hardware and network bandwidth. :)

debarshiray · 2026-06-25T11:49:10Z

Maybe this is a race where the push starts before the registry container is actually ready.

Yes, once in a while on some rare occasions I have seen _setup_docker_registry() fail the CI with a complaint about port 50000 being unavailable. I never figured out if it's the podman run that failed because the port was already taken on the CI host, or if (like you said) the skopeo copy started pushing the image too soon before the registry was ready to accept it.

WDYT about introducing some way to wait on the registry? Maybe something like:

local -i retries
for ((retries = 0; retries < 30; retries++)); do
    if curl -fsk https://localhost:50000/v2/ > /dev/null 2>&1; then
        break
    fi
    sleep 1
done

Yes, it sounds like a good idea, @Rolv-Apneseth ! We already use a similar loop in _pull_and_cache_distro_image() when downloading the images. Do you want to submit a pull request for it?

I am wondering if we can:

use openssl(1) for it instead of curl(1), because we already use openssl(1) in the test suite and it can avoid the need to update the dependencies in a bunch of places upstream and downstream:

openssl s_client localhost:50000 </dev/null

avoid the curl --insecure, because we already have the TLS certificate and key in domain.crt and domain.key

In any case, I slightly prefer the longer options because aliases like -f and -s are surprisingly difficult to grep for in the manual and elsewhere. curl --fail --insecure --silent is easier for the reader. :)

Rolv-Apneseth · 2026-06-25T15:24:57Z

Thanks @debarshiray, I've opened #1811. Let's see how it goes.

TristanCacqueray requested a review from debarshiray as a code owner June 9, 2026 06:31

gemini-code-assist Bot reviewed Jun 9, 2026

View reviewed changes

TristanCacqueray mentioned this pull request Jun 9, 2026

build: Style fix #1802

Merged

debarshiray mentioned this pull request Jun 11, 2026

fix: ensure -zlazy for systems that default to -znow #1729

Merged

debarshiray force-pushed the debug-ci branch from 24544ae to cf00e86 Compare June 11, 2026 13:04

debarshiray reviewed Jun 11, 2026

View reviewed changes

TristanCacqueray and others added 2 commits June 12, 2026 22:54

Debug ci timed out, do not merge

329d58b

Debug ci timed out, do not merge

8619bd5

debarshiray force-pushed the debug-ci branch from cf00e86 to 8619bd5 Compare June 12, 2026 20:57

Rolv-Apneseth mentioned this pull request Jun 18, 2026

Add Fedora CoreOS to CI tests #1806

Open

Rolv-Apneseth mentioned this pull request Jun 25, 2026

Attempt to fix CI timeouts #1811

Open

Uh oh!

Conversation

TristanCacqueray commented Jun 9, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

centosinfra-prod-github-app Bot commented Jun 9, 2026

Uh oh!

debarshiray commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

debarshiray left a comment

Choose a reason for hiding this comment

Uh oh!

centosinfra-prod-github-app Bot commented Jun 11, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Jun 12, 2026

Uh oh!

Rolv-Apneseth commented Jun 18, 2026

Uh oh!

TristanCacqueray commented Jun 18, 2026

Uh oh!

Rolv-Apneseth commented Jun 18, 2026

Uh oh!

Rolv-Apneseth commented Jun 18, 2026

Uh oh!

debarshiray commented Jun 25, 2026

Uh oh!

debarshiray commented Jun 25, 2026

Uh oh!

Rolv-Apneseth commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

debarshiray commented Jun 11, 2026 •

edited

Loading