Skip to content

Commit 8bb544b

Browse files
committed
workflows: run external-package purge once per workflow, not per chunk
The repository-update workflow fans out infrastructure-download-external across 4 chunks so download-time aptly operations stay under strategy.matrix's 256-entry cap, and the actual per-package download step does use CHUNK_INDEX to slice its work. preclean / postclean don't. Both build the same (release × package) matrix on every invocation and run repo.sh -c delete against the shared /publishing/repository-debs repository. At 4 chunks that meant four parallel copies of aptly's delete op touching the same database at the same time. aptly serialises via its own lockfile so it usually doesn't corrupt, but it's wasted work at best and a corruption hazard the moment a lock-wait times out or is bypassed. Gate both jobs on CHUNK_INDEX == 0 so the purge runs exactly once per workflow, independent of the download fanout. Chunks 1..3 now skip preclean/postclean entirely and jump straight to their slice of the download matrix. Download parallelism is preserved; purge goes back to being the single-threaded operation aptly's data model requires.
1 parent 719a9f7 commit 8bb544b

File tree

1 file changed

+18
-2
lines changed

1 file changed

+18
-2
lines changed

.github/workflows/infrastructure-download-external.yml

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,18 @@ jobs:
9393
preclean:
9494
name: "Purge"
9595
needs: perm
96-
if: ${{ inputs.PURGE == true }}
96+
# Purge only in chunk 0. The preclean/postclean pair doesn't use
97+
# CHUNK_INDEX to filter its matrix — it builds the same
98+
# (release × package) list on every chunk and runs repo.sh -c delete
99+
# against /publishing/repository-debs. With 4 chunks that meant 4
100+
# parallel invocations of aptly's delete op racing on the same
101+
# repository state; aptly serialises via its own lockfile but the
102+
# fanout is still pure waste at best and a corruption hazard when a
103+
# held lock times out. Package download IS parallel-friendly (each
104+
# chunk fetches its own slice of debs into /incoming/debs/external),
105+
# but the repo-side purge is a single-threaded operation on shared
106+
# state. Gate it to chunk 0 so it runs exactly once per workflow.
107+
if: ${{ inputs.PURGE == true && inputs.CHUNK_INDEX == 0 }}
97108
runs-on: ubuntu-latest
98109
outputs:
99110
matrix: ${{steps.json.outputs.JSON_CONTENT}}
@@ -124,7 +135,12 @@ jobs:
124135
125136
postclean:
126137
needs: preclean
127-
if: ${{ inputs.PURGE == true }}
138+
# Same chunk-0 gate as preclean (see its comment): this is the
139+
# repo.sh -c delete pass, which mutates shared aptly state and
140+
# MUST run single-threaded. The inner matrix already has
141+
# max-parallel: 1; the chunk gate here prevents 4 parallel
142+
# copies of that same serialised matrix from racing each other.
143+
if: ${{ inputs.PURGE == true && inputs.CHUNK_INDEX == 0 }}
128144
strategy:
129145
fail-fast: false
130146
max-parallel: 1

0 commit comments

Comments
 (0)