ci(infra): minio pull denied blocks HTTP, CLI and MCP lifecycle acceptance #207

Summary

Validate Agent memory → job HTTP, CLI and MCP lifecycle cannot start its test stack: docker compose is refused an anonymous pull of minio/minio. It is red on every attempt made since at least 2026-09-14T04:43Z, on four different pull requests, with an identical error and no in-repo change anywhere on that path. This is not a code regression — it is the repository's only end-to-end agent-memory lifecycle gate going dark underneath us.

The error, verbatim:

 minio Error pull access denied for minio/minio, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Error response from daemon: pull access denied for minio/minio, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Evidence

Observed runs of HTTP, CLI and MCP lifecycle, all on ubuntu-24.04, all failing the same step (memory-validation.yml:190-191 → scripts/acceptance_agent_memory.sh):

What those logs and configuration establish:

• It is deterministic, not transient. In each run the daemon answers denied ~0.13 s after minio Pulling (04:43:53.449 → 04:43:53.578), and postgres is reported Interrupted as a consequence, not as a second failure. Four attempts over ~11 hours behave identically.
• Docker Hub anonymous pulls still work from these runners. The PostgreSQL integration job pulls pgvector/pgvector:pg16@sha256:00ba258a… straight from Docker Hub on the same runner image at the same time (memory-validation.yml:144) and is green in all four runs above. So this is specific to the minio/* repositories, not general registry egress, and not a rate-limit toomanyrequests.
• The rest of CI is green on the identical head SHAs. CI (workflow 321876000) reports success for f464f68636, 145a987e5f, 4e1ea308a6 and 7e9ce11e51 — byte-for-byte the heads the table above fails on. Only the minio-dependent job is red.
• Nothing in this repository changed. Last green: run 34624197227 on main @ 3c13f04e at 2026-09-11T16:49Z, complete in 5 min. docker-compose.test.yml and scripts/acceptance_agent_memory.sh were last touched by b3924d1f (feat: ship portable agent memory drive #11, 2026-07-28) and memory-validation.yml by cc727db0 (chore(release): prepare v0.1.1 candidate #130, 2026-08-31) — every input predates the last green run.
• Only one job can see it, and it can't. The meta job runs docker compose -f docker-compose.test.yml config (memory-validation.yml:68-69), which validates the model without pulling. No other workflow in the repo references minio (ci.yml, bytefolk-security.yml, bytefolk-scorecard.yml, release.yml, pr-policy.yml, npm-publish.yml: zero hits).
• The same dependency is pinned three different ways, so this is the first time the divergence has cost anything: docker-compose.test.yml:25 minio/minio:latest@sha256:14cea493… and :43 minio/mc:latest@sha256:a7fe349e…; docker-compose.yml:43/:63 floating minio/minio:latest and minio/mc:latest with no digest at all; deploy/compose/compose.yaml:93/:111/:220 RELEASE.2025-04-22T22-12-26Z-style tags. The dev and self-hosted paths are exposed to the same pull but are not observed by CI.
• Onset is only bounded, not dated. No Validate Agent memory run exists anywhere between 2026-09-11T16:54Z and 2026-09-14T04:43Z, so it could have begun at any point in that ~36 h window. main has had no push since 3c13f04e, which is why this reads as "main was never re-verified" rather than "main broke".

What I could not verify

Stated explicitly so nobody re-spends this effort or over-trusts the lines above.

• Which side is at fault. Outbound network from the environment this was gathered in is limited to api.github.com, so Docker Hub could not be queried and nothing could be pulled locally. I cannot distinguish (a) the minio/minio public repository having been removed or restricted, (b) an anonymous-pull policy change, (c) a GitHub-hosted-runner egress/IP problem. Docker Hub returns the same repository does not exist or may require 'docker login' string for all of them. That is why this issue carries evidence:e2-source and not E3.
• Whether minio/mc is also denied. The run dies on the minio service first (acceptance_agent_memory.sh:200: compose up -d --wait postgres minio), so minio-init is untested. A fix that only addresses minio/minio may reveal a second failure.
• Whether HTTP, CLI and MCP lifecycle is in the enforced required set. GET /repos/bytefolk/mem/branches/main/protection returns 404 to me, and GOVERNANCE.md:41-42 predicts exactly that for a non-admin ("Contributors without admin cannot read the endpoint"). Org rulesets need admin:org. GOVERNANCE.md:44-53 lists this job among observed checks but says of itself that the enumeration is "evidence of what runs, not a substitute for the configuration". Observation consistent with it being required: chore(ci): correct CodeQL pinned-version annotations #203 and ci(security): run CodeQL on fork pull requests #205 are APPROVED with this as their only red check and have sat unmerged since 09-14. That is an inference, not a read.
• chore(release): prepare ByteFolk mem-mcp 0.1.2 #193 is not part of this. Its one red check is npm wrapper compatibility (node20-windows) from run 34496784723 (2026-09-10), a different failure predating the window.
• Whether other bytefolk repositories pull minio. roleweave has no minio or compose references, so it offers no comparison.

Direction to evaluate (no assertion that any of these is correct)

1. Authenticated pull. Add a login step before acceptance_agent_memory.sh using a Docker Hub read-only token in a repo secret. Cheapest, changes no pins. Costs: CI inherits an account's rate limit and the secret's lifetime, and it does not survive the repository actually being gone.
2. Serve the same content from a registry the org controls or that is not restricted. The current reference is digest-pinned, so the mirrored content is comparable rather than trusted — a replacement must present digest 14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e, or differ on the record with a reason.
3. Make the failure self-diagnosing (follow-up, does not unblock): a preflight that pulls the pinned images by name, so a registry-side event reads as "image unavailable" instead of a lifecycle acceptance failure on unrelated PRs.

scripts/acceptance_agent_memory.sh:200 has no skip or fallback path, and minio is not optional for this acceptance, so "stop testing it" is not among the options.

Acceptance criteria

1. HTTP, CLI and MCP lifecycle is green on a fresh commit on a PR whose head is current with main, and the mechanism that made it green is named in the PR (login, mirror, or other), with the pin diff visible.
2. The pull of both images at docker-compose.test.yml:25 and :43 is confirmed working, so the minio/mc question above is closed rather than deferred.
3. If a mirror is introduced, the digest is verified equal to the currently pinned value, or the divergence is justified in writing.
4. An admin states in a comment on this issue whether HTTP, CLI and MCP lifecycle is in the required set, replacing my inference with a read.
5. If the chosen mechanism changes what a developer must do locally, docs/DEVELOPMENT.md and DEPLOYMENT.md say so in the same PR.

Non-goals

• Unifying the three minio pinning conventions above. Real, separate, and worth its own issue once the queue moves.
• Whether minio is the right object-store dependency at all.
• fix(folders): recursive folder delete leaves every object in bucket storage #177 and every other product-level bug: unrelated to this path.

Who can act, and how this is labelled

The plausible fixes touch repository secrets or registry infrastructure, which CODEOWNERS (* @PeterGuy326 @Bindy-lbb @waterbro-8) does not make reachable for a non-admin code owner.

Applied per docs/maintainers/triage.md: type:maintenance (the taxonomy folds infrastructure here), area:infra, evidence:e2-source, status:needs-triage. No severity: severity is required for bugs and incidents and is "not normally applied to maintenance tasks", and I have not established that no workaround exists — criterion 4 above is what turns that judgement into a fact. A maintainer who reads this as an incident rather than maintenance should re-type it as type:bug and add the severity; s1-high is the honest candidate once the required-check question is answered.