Repository navigation
Show the snapshot canary catching an induced validation failure (P4) #147
Description
Part of #118, the tracker for the wasm-dynamic-snapshots project. P4's line in PLAN.md Phase 8 and the success criteria on QuantEcon/meta#338 both ask for a canary run that catches an induced failure. The canary has fired once, on a genuine FRED fetch timeout (#115, 2026-09-01: exit 1, classified correctly), which covers the fetch class. The validation class (exit 2) has never run in CI; that is the failure where the data breaks the published contract and a person has to decide. This issue proves the validation class.
Steps
- Push a throwaway branch, for example
probe/canary-exit-2, that makesbuilders/business_cycle_fred.pyfail validation against live data: narrow one entry inBANDS, for example'UNRATE': (0, 1). - Dispatch the workflow against it:
gh workflow run refresh-snapshots.yml -R QuantEcon/data-lectures --ref probe/canary-exit-2 -f dataset=us_business_cycle_monthly.csv. Avoid the Monday 06:17 UTC schedule, which holds the workflow's concurrency group. It is safest before 2026-10-01: until then nothing is due, so the refresh job cannot run. From that date, if the induced failure misfired and the canary passed, the refresh job would run from the throwaway branch and push its builder change torefresh/us_business_cycle_monthly. - Confirm three things. The canary leg exits 2. The refresh job is skipped, even once the file is due from 2026-10-01. And
notifyopens theupstream-breakissue with "validation failed (exit 2)", or comments on one that is already open. - Close that issue with a note that points here, and delete the branch. Record the run id here, on PLAN: pilot migration — one dataset per hosting pattern to validate the datasets design meta#338, and on P4's line in
PLAN.md, either with block C's PR or on its own.
Done when the run id is recorded and the branch is deleted.
Activity
mmcky commentedon Sep 30, 2026
Done 2026-09-30: the canary catches an induced validation failure (exit 2), and a failing canary keeps the refresh job from running even when the file is due.
The throwaway branch
probe/canary-exit-2(caead9a) changed one line frommain(ab9d24d):BANDS['UNRATE']inbuilders/business_cycle_fred.py, narrowed from(0, 30)to(0, 1). The workflow file was byte-identical tomain's. The workflow was dispatched against the branch twice, withdataset=us_business_cycle_monthly.csv.forceIn both runs the canary step logged
##[error]us_business_cycle_monthly.csv: validation failed -- UNRATE: out of band [0, 1]and then##[error]Process completed with exit code 2., and itsfailure.jsonartifact reads"kind": "validation", "code": "2". The lines are quoted here because the artifacts expire on 2026-10-07 and run logs are not kept for good.Each run proves something different, so both are cited. The first shows the exit code and notify opening an issue, but its refresh skip proves nothing, because nothing was due. The second shows the gate. Its plan job succeeded and listed the file as due, yet the refresh job was skipped, so the canary's failure is what stopped it. The gate reads only the due list, not why a file is on it, so a file due on its cadence from 2026-10-01 is gated the same way. A failed canary trips two conditions at once: the job's explicit
needs.canary.result == 'success'and thesuccess()that GitHub adds implicitly. These runs cannot tell which of the two did the work.Before either dispatch, the builder ran locally against live FRED under the pinned pandas 2.3.3. Unmodified, it exited 0. With the probe's change, it exited 2 with the same message. So the failure is the induced one and nothing else.
Nothing reached git or the data: no
refresh/*branch, no pull request,mainstill atab9d24d, andlectures/us_business_cycle_monthly.csvstill hashes to its manifest'ssha256. #150 is closed as not planned, with a note pointing here. The branch is deleted, and the run ids are recorded on QuantEcon/meta#338. P4's line inPLAN.mdtakes them in block C's PR (#146), which ticks that line once the flip lands.What these runs do not show: the scheduled trigger taking this path (both runs were dispatched), a failure in the builder that writes a set of files (
builders/business_cycle.py), and several legs failing in one run. The fetch class (exit 1) was shown by a real FRED timeout on 2026-09-01 (#115).