Skip to content

ci(data-deploy): publish on the crawler, serialize Pages deploys - #60

Merged
andrew2net merged 1 commit into
mainfrom
feat/data-deploy-trigger-concurrency
Aug 11, 2026
Merged

ci(data-deploy): publish on the crawler, serialize Pages deploys#60
andrew2net merged 1 commit into
mainfrom
feat/data-deploy-trigger-concurrency

Conversation

@andrew2net

Copy link
Copy Markdown
Contributor

Fixes two fleet-wide defects in the shared Pages plumbing, found while rolling relaton index out to relaton-data-iana. Both affect every relaton-data-* repo.

1. Callers publish stale data

The caller template's push trigger cannot fire on crawled data: crawler.yml's "Push data" step commits with the default actions/checkout GITHUB_TOKEN, and GitHub deliberately raises no workflow events for GITHUB_TOKEN pushes. So the daily cron was the only path from crawl to Pages — and it assumed a fixed one-hour gap (crawler 0 14, deploy 0 15) that scheduled dispatch does not honour. In relaton-data-iana's last 12 Crawler runs, 11 started at or after 14:58 UTC, several past 16:00, so the 15:00 deploy usually indexed the previous day's data.

cimas-config/gh-actions/data/deploy.yml now triggers on workflow_run (Crawler, completed), with the cron demoted to an 18:00 fallback clear of the crawler's observed window. Listing the same master, main, v2 set as push keeps the template repo-agnostic — no per-repo edit needed.

Safe with the existing default-branch gate: a workflow_run run's GITHUB_REF is the default branch, so the gate passes and GITHUB_REF_NAME still yields correct --base-url raw-YAML links.

2. No concurrency group

Two overlapping runs both reached actions/deploy-pages@v4 and the loser failed. data-deploy.yml now declares the group itself — a reusable workflow's own top level is where GitHub documents this to live (jobs.<id>.concurrency on the calling job "will not behave as expected") — so this fixes every caller with no per-repo edit.

The group is keyed on the deploy job's default-branch gate, so only runs that can publish share a queue. GitHub keeps one pending run per group and cancels it when the next arrives; a PR or tag-push build sharing the queue would otherwise build for ~10 minutes, skip deploy, and drop a pending run carrying fresh data.

The failed-crawler if: guard is likewise central, on build_index_page (a called workflow inherits the caller run's github context).

Not in this PR

Hand-off §3 (pinning relaton-ref off a moving relaton/relaton@main) — left as the accepted temporary posture until relaton-cli ships index.

⚠️ Before syncing the template

Cimas replaces deploy.yml wholesale in all 29 mapped repos, and four carry a hand-added with: block that nothing regenerates (configs.yml renders _config.yml only):

  • relaton-data-bipm, relaton-data-3gppsource: git
  • relaton-data-w3c, relaton-data-ianasource: git plus favicon:/description:

Syncing over them drops those back to source: gem, which fails with Could not find command "index". Re-apply after the sync, or hold those four back. Documented in data-index/README.adoc.

Tests

bundle exec rake193 examples, 0 failures. New specs pin the couplings that fail silently: the trigger's workflows: value against crawler.yml's own name:, the concurrency group against the deploy job's gate, and the fallback cron's margin over the crawler's.

Two fleet-wide defects in the shared Pages plumbing.

The caller template's `push` trigger cannot fire on crawled data: crawler.yml
commits with the default actions/checkout GITHUB_TOKEN, and GitHub raises no
workflow events for GITHUB_TOKEN pushes. That left the daily cron as the only
path from crawl to Pages, and it assumed a fixed one-hour gap that scheduled
dispatch does not honour -- 11 of relaton-data-iana's last 12 Crawler runs
started at or after 14:58 UTC despite a 14:00 cron, several past 16:00, so the
15:00 deploy usually indexed the previous day's data.

The Cimas caller template now triggers on `workflow_run` (Crawler, completed),
with the cron demoted to an 18:00 fallback clear of the crawler's observed
window. Listing the same master/main/v2 set as `push` keeps the template
repo-agnostic, so this needs no per-repo edit.

Neither workflow declared a `concurrency` group, so two overlapping runs both
reached actions/deploy-pages@v4 and the loser failed. data-deploy.yml now
declares one itself -- a reusable workflow's own top level is where GitHub
documents this to live -- fixing every caller at once. The group is keyed on
the deploy job's default-branch gate so only runs that can actually publish
share a queue: GitHub keeps one pending run per group and cancels it when the
next arrives, so a PR or tag-push build sharing the queue could otherwise drop
a run carrying fresh data.

The failed-crawler guard is likewise central, on build_index_page, since a
called workflow inherits the caller run's github context.

Specs pin the couplings that fail silently: the trigger's `workflows:` value
against crawler.yml's own `name:`, the concurrency group against the deploy
job's gate, and the fallback cron's margin over the crawler's.

Note before syncing the template: Cimas replaces deploy.yml wholesale, and
relaton-data-{bipm,3gpp,w3c,iana} carry hand-added `with:` blocks that nothing
regenerates. Documented in data-index/README.adoc.
@andrew2net
andrew2net merged commit 9b32e52 into main Aug 11, 2026
5 checks passed
@andrew2net
andrew2net deleted the feat/data-deploy-trigger-concurrency branch August 11, 2026 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant