ci(data-deploy): publish on the crawler, serialize Pages deploys - #60
Merged
Conversation
Two fleet-wide defects in the shared Pages plumbing.
The caller template's `push` trigger cannot fire on crawled data: crawler.yml
commits with the default actions/checkout GITHUB_TOKEN, and GitHub raises no
workflow events for GITHUB_TOKEN pushes. That left the daily cron as the only
path from crawl to Pages, and it assumed a fixed one-hour gap that scheduled
dispatch does not honour -- 11 of relaton-data-iana's last 12 Crawler runs
started at or after 14:58 UTC despite a 14:00 cron, several past 16:00, so the
15:00 deploy usually indexed the previous day's data.
The Cimas caller template now triggers on `workflow_run` (Crawler, completed),
with the cron demoted to an 18:00 fallback clear of the crawler's observed
window. Listing the same master/main/v2 set as `push` keeps the template
repo-agnostic, so this needs no per-repo edit.
Neither workflow declared a `concurrency` group, so two overlapping runs both
reached actions/deploy-pages@v4 and the loser failed. data-deploy.yml now
declares one itself -- a reusable workflow's own top level is where GitHub
documents this to live -- fixing every caller at once. The group is keyed on
the deploy job's default-branch gate so only runs that can actually publish
share a queue: GitHub keeps one pending run per group and cancels it when the
next arrives, so a PR or tag-push build sharing the queue could otherwise drop
a run carrying fresh data.
The failed-crawler guard is likewise central, on build_index_page, since a
called workflow inherits the caller run's github context.
Specs pin the couplings that fail silently: the trigger's `workflows:` value
against crawler.yml's own `name:`, the concurrency group against the deploy
job's gate, and the fallback cron's margin over the crawler's.
Note before syncing the template: Cimas replaces deploy.yml wholesale, and
relaton-data-{bipm,3gpp,w3c,iana} carry hand-added `with:` blocks that nothing
regenerates. Documented in data-index/README.adoc.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes two fleet-wide defects in the shared Pages plumbing, found while rolling
relaton indexout torelaton-data-iana. Both affect everyrelaton-data-*repo.1. Callers publish stale data
The caller template's
pushtrigger cannot fire on crawled data:crawler.yml's "Push data" step commits with the defaultactions/checkoutGITHUB_TOKEN, and GitHub deliberately raises no workflow events forGITHUB_TOKENpushes. So the daily cron was the only path from crawl to Pages — and it assumed a fixed one-hour gap (crawler0 14, deploy0 15) that scheduled dispatch does not honour. Inrelaton-data-iana's last 12 Crawler runs, 11 started at or after 14:58 UTC, several past 16:00, so the 15:00 deploy usually indexed the previous day's data.cimas-config/gh-actions/data/deploy.ymlnow triggers onworkflow_run(Crawler,completed), with the cron demoted to an 18:00 fallback clear of the crawler's observed window. Listing the samemaster, main, v2set aspushkeeps the template repo-agnostic — no per-repo edit needed.Safe with the existing default-branch gate: a
workflow_runrun'sGITHUB_REFis the default branch, so the gate passes andGITHUB_REF_NAMEstill yields correct--base-urlraw-YAML links.2. No
concurrencygroupTwo overlapping runs both reached
actions/deploy-pages@v4and the loser failed.data-deploy.ymlnow declares the group itself — a reusable workflow's own top level is where GitHub documents this to live (jobs.<id>.concurrencyon the calling job "will not behave as expected") — so this fixes every caller with no per-repo edit.The group is keyed on the deploy job's default-branch gate, so only runs that can publish share a queue. GitHub keeps one pending run per group and cancels it when the next arrives; a PR or tag-push build sharing the queue would otherwise build for ~10 minutes, skip
deploy, and drop a pending run carrying fresh data.The failed-crawler
if:guard is likewise central, onbuild_index_page(a called workflow inherits the caller run'sgithubcontext).Not in this PR
Hand-off §3 (pinning
relaton-refoff a movingrelaton/relaton@main) — left as the accepted temporary posture until relaton-cli shipsindex.Cimas replaces
deploy.ymlwholesale in all 29 mapped repos, and four carry a hand-addedwith:block that nothing regenerates (configs.ymlrenders_config.ymlonly):relaton-data-bipm,relaton-data-3gpp—source: gitrelaton-data-w3c,relaton-data-iana—source: gitplusfavicon:/description:Syncing over them drops those back to
source: gem, which fails withCould not find command "index". Re-apply after the sync, or hold those four back. Documented indata-index/README.adoc.Tests
bundle exec rake→ 193 examples, 0 failures. New specs pin the couplings that fail silently: the trigger'sworkflows:value againstcrawler.yml's ownname:, the concurrency group against the deploy job's gate, and the fallback cron's margin over the crawler's.