Repository navigation
lecture-wasm dynamic snapshots: business_cycle reads the four data-lectures snapshots (P4) #118
Description
This issue is the tracker for the wasm-dynamic-snapshots project in the status-projects registry (programme: infrastructure; project page), and its sub-issues are the work list. The project moves lecture-wasm's business_cycle lecture off its live World Bank and FRED calls and onto the four business_cycle dynamic snapshots this repo serves, so the lecture runs in the reader's browser from a cached copy that refreshes by pull request. Done means three things: the lecture is re-enabled in lecture-wasm and runs end to end in the browser; the four snapshots record it as their consumer and are marked repointed; and the P4 box in PLAN.md Phase 8 is ticked. The work began on 2026-09-01 as Track E of the datasets migration (project page). On 2026-09-29 @mmcky ruled it a project of its own, with this issue as its tracker, so that the migration can close without it.
Where we stand (verified 2026-09-30)
The four snapshots landed on 2026-09-01: gdp_growth_annual.csv (as business_cycle_data.csv in #109), then unemployment_rate_annual.csv, private_credit_to_gdp.csv and us_business_cycle_monthly.csv (#114). The weekly canary and refresh-as-PR workflow landed the same day (#110), with its first refresh (#112) and its first real alarm, a FRED fetch timeout it classified correctly (#115). On 2026-09-07 block A settled the names (#113, landed in #124), so the four filenames are final, and the shared manifest-driven validate() began checking all four on every PR (#126). On 2026-09-30 the canary proof closed (#147). Two dispatched runs showed a validation failure exiting 2 and notify opening an upstream-break issue, then commenting on it. In the second run the file was due and the refresh job was still skipped. No lecture reads the files yet. The same day @mmcky took decisions 1 and 2 and ruled on FRED's caching clause (see "Decisions taken"), and blocks B and C opened as draft PRs, QuantEcon/lecture-wasm#85 and #153. Two data-lectures PRs went first and merged the same day: #151, an audit annotation that main needed regardless (intro's mle began reading yfinance on 2026-09-29), and #152, which records the licence decisions. Next the first monthly refresh merges, B's gate runs on its preview, and B and C merge on the same day.
| Fact | State on 2026-09-30 |
|---|---|
| The four snapshots | each serves HTTP 200 from raw.githubusercontent.com with access-control-allow-origin: * and bytes matching its manifest's sha256; all four retrieved 2026-09-01 |
| Their consumers | none: every manifest has consumers: [], migration.yml holds them at landed, and the audit of 2026-09-28 finds no reader |
| Refresh workflow | refresh-snapshots green on every scheduled run since 2026-09-07, the last on 2026-09-28. The two failed runs on 2026-09-30 are #147's probe, failing by design. Nothing is due before 2026-10-01 |
| lecture-wasm | lectures/business_cycle.md unchanged since 2025-10-23 (QuantEcon/lecture-wasm#31) and still commented out of lectures/myst.yml, so the page 404s on the live site; block B open as QuantEcon/lecture-wasm#85 (draft); main at ff9196d |
| lecture-wasm previews | Netlify's app check fails on every PR (QuantEcon/lecture-wasm#62), while the workflow's own preview at pr-<N>--beautiful-dodol-cb9543.netlify.app serves real pages |
| data-lectures | main at bbb4cc6 after #151 and #152, with its audit green and deployed; the audit of 2026-09-28 counted 23 lectures that call live APIs, 5 of them in lecture-wasm |
The work
Each block is a sub-issue. This table says what each one is for; the sub-issue list carries its state.
| Block | Sub-issue | Delivers | Gate |
|---|---|---|---|
| A. Naming | #113 | the naming policy; the four filenames final (landed in #124, 2026-09-07) | done |
| B. lecture-wasm adoption | QuantEcon/lecture-wasm#70, PR QuantEcon/lecture-wasm#85 | business_cycle.md reads the four files and is re-enabled in lectures/myst.yml; the lecture runs end to end in the browser on the PR's own preview |
the first monthly refresh merged (#152 merged 2026-09-30), then Run all on the preview |
| C. The flip | #146, PR #153 | lecture-wasm recorded as the consumer on all four manifests, migration.yml at repointed, CATALOG.md regenerated, the strict audit dry-run in both directions |
B merged; same day |
| Canary proof | #147 | the weekly canary shown catching an induced validation failure (exit 2), the last item on P4's line in PLAN.md and one of QuantEcon/meta#338's success criteria |
done 2026-09-30 |
B is larger than QuantEcon/lecture-wasm#70 records (measured 2026-09-29):
- The five enabled lecture-wasm lectures that read this repo in code (
long_run_growth,inflation_history,french_rev,mleandsimple_linear_regression) all callpyodide_http.patch_all()before reading. B needs the same. - lecture-wasm's copy of
business_cycle.mdlacks four later intro changes: Fix grammar and spelling errors across lecture files lecture-python-intro#688, Adding Tom's Edits to Exercises lecture-python-intro#748, [business_cycle] update made to the lecture lecture-python-intro#641 and Update business cycle data to latest available (#364) lecture-python-intro#770. The last of these moved intro to the latest available data and "to the present" prose. The two copies differ on about 446 lines. Decision 1 settled it on 2026-09-30: B ports intromain. - lecture-wasm's CI never executes a cell (
myst build --html), and itsdata-url-guardrejects only the LFS media host. So the gate is a person running "Run all" on the PR preview (JupyterLite stops at the first failing cell), and the URL form is checked by eye:raw.githubusercontent.comonly, per repoint rule 5. Agithub.com/…/raw/read is caught only after merge, by this repo's strict audit. - The README edit removes
business_cycleand the staleshort_pathfrom the unsupported list, and retires the dead "sync from intro'swasmbranch" rule. - Drafted and tested on 2026-09-30 in business_cycle: read four data-lectures snapshots and re-enable the lecture lecture-wasm#85, which covers all of the above. All 22 code cells run under Pyodide 0.27.0's pinned pandas 2.2.3 with no errors and no warnings, after passing
fill_method=Noneto the twopct_changecalls, and the site builds.
B and C land on the same day. Between B's merge and C's merge this repo's strict audit fails, because a landed record has a reader. Every push to main, the Monday 05:17 UTC run and any PR that touches a manifest go red, including the monthly refresh PR, and the Pages deploy is skipped. Avoid a Monday.
Deadlines
| When | What |
|---|---|
| 2026-10-01 | us_business_cycle_monthly.csv falls due: monthly cadence, 30 days after retrieved 2026-09-01 |
| Monday 2026-10-05, about 06:17 UTC | the first scheduled monthly refresh PR should open on refresh/us_business_cycle_monthly. This is projected from refresh-snapshots.yml and has not been observed yet; scheduled runs have started 30 to 40 minutes late |
| 2026-10-31 | the last day to merge that refresh. The committed file's newest month is 2026-07, which from 2026-11-01 is more than MAX_STALENESS_MONTHS = 3 old. From then check_committed() in builders/business_cycle_fred.py fails validate-datasets on every PR and every push to main. It is not a required check, so merges still go through. Each merge restarts the clock, so merging the refresh every month is a standing duty that outlives this project |
| 2027-09-01 | the three World Bank files fall due (annual cadence); their builder's staleness guard holds until 2028-01-01 |
Decisions taken
- 2026-09-01: the pilot is the
business_cycleset, not a lone UNRATE twin. The lecture that needs a twin is excluded from lecture-wasm for want of one, and a partial twin buys it nothing (PLAN.mdPhase 8, P4). - 2026-09-01: lecture-python-intro keeps its live API calls, because there the API is the lesson; lecture-wasm reads the snapshots instead (AGENTS.md, "Refresh, break, or schema change", and each manifest's
consumerscomment). - 2026-09-01: one builder per source, composite files where a lecture reads series together, a shared fetch library (
builders/_fred.py), and nopandas_datareaderin a builder (pandas_datareader is maintained again (0.11.0/0.11.1, June 2026) — reconsider it as the builder-side entry point for dynamic data #26). - 2026-09-07: the naming policy (Naming policy for published files: what a composite is called, and when a file may be renamed #113, landed in Record the four schema and naming decisions: rename, pattern capture groups, null placement, dtype sweep #124). The four names are final, and once a file has a consumer, renaming it means a new filename with the old one kept serving.
- 2026-09-07: the FRED composite declares
recent: 1, so a monthly refresh that adds a row with blanks in UMCSENT, CPILFESL and INDPRO is expected and is not a defect (this issue's comment of 2026-09-07). - 2026-09-29, ruled by @mmcky: this work is its own project,
wasm-dynamic-snapshots(programme infrastructure, stage active, priority next, started 2026-09-01), with this issue as its tracker. qeld is a separate proposed project,qeld-package. - 2026-09-30, ruled by @mmcky: block B ports lecture-python-intro
main'sbusiness_cycle.md(decision 1), whose "to the present" prose matches a refreshing snapshot. It picks up intro's four later changes. - 2026-09-30, ruled by @mmcky: UMCSENT stays in the composite,
permittedon the University of Michigan's FAQ for data on its main website, whilebbh_michigan_monthly.csv's portal-only series stayrestricted(decision 2; Licensing: record-and-track policy, and the per-dataset inventory for migration #35; recorded in licence: rest UMCSENT on Michigan's FAQ and record FRED's caching clause #152). - 2026-09-30, ruled by @mmcky: FRED stays a known yes, with its Terms of Use clause (p), on storing or caching FRED content, recorded once for the source in AGENTS.md and pointed to from the seven FRED-derived manifests. Asking the Bank for consent waits for the promotion gate (Licensing: record-and-track policy, and the per-dataset inventory for migration #35; recorded in licence: rest UMCSENT on Michigan's FAQ and record FRED's caching clause #152).
Decisions open
on_refreshfor lecture-wasm, set in C. Flip wave P4 to repointed — the four business_cycle snapshots #153 setsrebuild, because no prose in B quotes a number a refresh can move. It is taken when C merges, unless B's prose changes in review.- Who merges the monthly FRED refresh PRs once this project closes.
Not in scope
- lecture-python-intro's live calls (decided 2026-09-01).
- The other 22 live-API lectures in the audit of 2026-09-28, which counts 23 with this one. Among them are python.myst's
unemployment_linearandunemployment_shocks, which read UNRATE over the window the FRED composite carries and are candidates for a later adoption. - lecture-wasm's other excluded lectures and their browser shims (for example Two network reads run unshimmed under Pyodide: inequality.md imports pyodide_http but never patches, short_path.md has no shim at all lecture-wasm#59).
- qeld, which has been the proposed
qeld-packageproject since 2026-09-29. It covers the qeld decisions (qeld: the seven open decisions from PLAN-QELD-PACKAGE.md, and the conventions each one settles #66), the deferred domain (Track Y (deferred): data.quantecon.org DNS + custom domain — retired in favor of qeld (D11); reopenable, NXDOMAIN as of 2026-08-10 #37) and teaching the audit to recogniseqeld.url()calls (qeld Q1: build_audit.py must learn the qeld.url() call form before any lecture adopts it #88). - Consumer fan-out, the old block D (
PLAN.mdPhase 5). lecture-wasm runs the lecture in the reader's browser and readsraw/mainwhen it runs, so a merged refresh reaches its readers with no rebuild. Its CI never executes code and has no dispatch path that builds, so a dispatched rebuild would change nothing a reader sees. The fan-out waits for the first consumer that executes at build time. - Housekeeping carried over from the old plan, related but not tracked here: doubts_or_variability: drop the pandas-datareader install nothing imports lecture-python-advanced.myst#381 (green, awaiting a merge) and pandas_datareader is maintained again (0.11.0/0.11.1, June 2026) — reconsider it as the builder-side entry point for dynamic data #26 (decided, open only for the owner's word).
Learned along the way: kept from the session plan of 2026-09-01, with facts added on 2026-09-07, 2026-09-29 and 2026-09-30
- FRED stalls custom user agents from GitHub-hosted runners (
qeld-builder, and evenMozilla/5.0), but it answers urllib's default user agent and curl in about 50 ms. The same requests all succeed from a workstation. This is recorded inbuilders/_fred.py's docstring, and it was found by a probe run on a runner (TEMP: probe FRED endpoints from a GitHub runner (do not merge) #116), because it reproduces nowhere else. workflow_dispatchonly runs workflows that exist on the default branch, so a branch-only diagnostic has to ride apull_requesttrigger (a draft PR). A workflow that exists onmaincan be dispatched with--ref <branch>to run that branch's copy (run 35319536981, 2026-09-18).- The stamp's read-back guard earns its keep. It refused a date-shaped
date_range.endon the first multi-file stamp (YAML reads it back as a date), and it refused a foreign summary on the firstpr-bodytest. - Manifest prose must not embed a fact that a refresh can change, such as an end year, an observed range or a row count. The stamp rewrites fields, not sentences, and the first refresh PR shipped a stale column description. The rule is in AGENTS.md and
builders/_template.py. Three comments ingdp_growth_annual.csv.ymlstill break it, and C rewrites them. - The
datasetdispatch input and the per-builder canary legs were both wrong on their first outing and were set right in FRED from GitHub runners: no custom User-Agent; canary one leg per builder #117. A builder that writes a set of files now runs once. - The FRED composite's UNRATE and CPILFESL are empty for 2025-10, the month BLS never published. The 1929–42 series is the column
M0892AUSM156SNBR, and the lecture'srenametoUNRATEstill applies. - lecture-wasm's README rule "never edit lectures here; sync from intro's
wasmbranch" is dead. That branch is 130 commits behind intromain(its last commit was on 2025-04-24), and lecture-wasm PRs editlectures/directly. - lecture-wasm's
lectures/myst.ymlexcludes seven lectures, not the eight that Include business_cycle: what it reads, what data-lectures now serves, and the sequence lecture-wasm#70 counts:short_pathhas been enabled since at least 2026-06-11, though the README still lists it as unsupported. Of the lectures that issue names as reading this repo,heavy_tailsandinequalityare excluded and 404 on the live site, whilemleandsimple_linear_regression, which it does not name, do read this repo. - lecture-wasm's
ci.ymlbuilds only on a push tomainor on a pull request. Aworkflow_dispatchof it builds nothing, and it has norepository_dispatch. - The canary has fired for real once, on a genuine FRED fetch timeout (refresh-snapshots: a dynamic snapshot's builder is failing #115: exit 1, open for 22 minutes on 2026-09-01). The validation class (exit 2) was shown on 2026-09-30 by an induced failure (Show the snapshot canary catching an induced validation failure (P4) #147).
- A job-level
if:with no status function gets an implicitsuccess(). So a failed canary skips the refresh job through two conditions at once: the explicitneeds.canary.result == 'success'and the implicit one. A run cannot show which of them did the work (Show the snapshot canary catching an induced validation failure (P4) #147). - The canary's failure artifacts are kept for seven days, so a record of a canary run should quote its key log lines rather than only link to it. notify's issue text begins "The weekly sources-alive canary failed" even when the run was dispatched.
- Rewrite MLE as an elementary lecture on maximum likelihood lecture-python-intro#849 (merged 2026-09-29) added a
yfinanceread to intro'smle, and the strict audit fails on it until it is annotated (audit: annotate intro mle's yfinance read #151). A lecture repo's merge can turn this repo's audit red with no change here, so a block's dry-run needs a greenmainfirst. - Pyodide 0.27.0 ships pandas 2.2.3, whose
pct_changepads interior holes forward and warns about it. Intro builds on pandas 3, which does neither, so B passesfill_method=Noneto keep its figures the same as intro's. - FRED labels USREC and M0892AUSM156SNBR "Copyrighted: Citation Required" as well as UMCSENT, and gives M0892AUSM156SNBR as seasonally adjusted. The manifests had both wrong until licence: rest UMCSENT on Michigan's FAQ and record FRED's caching clause #152.
api.worldbank.org/v2sendsaccess-control-allow-origin: *, contrary to what Include business_cycle: what it reads, what data-lectures now serves, and the sequence lecture-wasm#70 assumed; only FRED'sfredgraph.csvsends none. Whetherwbgapiitself could run under Pyodide is untested.
Activity
mmcky commentedon Sep 6, 2026
Revision 2026-09-07: block A is done, and it went first after all. The naming policy settled on #113 and landed with the three schema decisions in #124 (merged as
c4a286c):business_cycle_data.csvis nowgdp_growth_annual.csv, the other three P4 names are confirmed final, and every manifest, the builder,migration.ymlandCATALOG.mdfollow. #120, #121, #122 and #113 are closed; #119 is the only child of #14 left. The measured table at the top is otherwise unchanged (no refresh was due; the snapshots are the 2026-09-01 vintage).Two facts for block B, measured this session rather than assumed: the Netlify deploy-preview check is red on every
lecture-wasmPR (QuantEcon/lecture-wasm#62 is still true as of #68 on 2026-09-01) whilebuild-and-deploypasses, so B's gate — the lecture runs end to end in the browser — has to use the workflow's own CLI preview URL, not the Netlify status; and the four URLs the lecture will read are now frozen:raw.githubusercontent.com/QuantEcon/data-lectures/main/lectures/{gdp_growth_annual,unemployment_rate_annual,private_credit_to_gdp,us_business_cycle_monthly}.csv.Resume pointer: block B next, as a draft PR in
../lecture-wasmfor approval. Block C's manifest flip should also add theconsumersentry to all four files in one PR, since thenulls:placement blocks and theYR(\d{4})patterns now mean a manifest edit touches lines the shared validator will read.mmcky commentedon Sep 7, 2026
Revision 2026-09-07 (later): block E's "#14's remaining half" is done. #119 landed in #126 (merged as
818811b): the shared manifest-drivenvalidate()inbuilders/_validate.py, both dynamic builders delegating to it, andvalidate-datasets.ymlrunning conformance for all 44 manifests and byte validation for every CSV on each PR. #14 is closed; nothing from it is left on this plan.One consequence for block B and C: the FRED manifest now says
recent: 1and carries no exact null counts, because the validator's first live dry-run showed UMCSENT, CPILFESL and INDPRO unpublished for the newest month. The flip in block C should therefore expect the monthly refresh PRs to add one row with three blanks in it, which is correct, not a defect.Resume pointer unchanged: block B next, the
lecture-wasmdraft PR against the four frozen URLs.[-]Work plan — P4 dynamic snapshots: lecture-wasm adoption of the business_cycle set[/-][+]lecture-wasm dynamic snapshots: business_cycle reads the four data-lectures snapshots (P4)[/+]mmcky commentedon Sep 29, 2026
Revision 2026-09-29: this plan is now the tracker for its own project. On 2026-09-29 @mmcky ruled that lecture-wasm's adoption of the
business_cyclesnapshots, which until then was Track E of the datasets migration, is a project of its own:wasm-dynamic-snapshotsin the status-projects registry (programme infrastructure, active, priority next, started 2026-09-01), with this issue as its tracker. The static migration finished on 2026-08-18 and now closes without it. qeld is registered separately, as the proposedqeld-packageproject.Why this issue. It has carried this work's plan since 2026-09-01, so the tracker dates from the day the work began.
What moved where.
Projectissue type, sub-issues as the work list, and a dated "Where we stand" stamp that is re-dated whenever the project is touched. It is no longer a session plan. Comments remain the revision log, and the old body is in the edit history (last edited 2026-09-02).PLAN.mdis new as Show the snapshot canary catching an induced validation failure (P4) #147.PLAN.mdPhase 5 for the first consumer that executes at build time.pyodide_http.patch_all(), and the four intro changes that lecture-wasm's copy lacks), the FRED refresh dates with the 2026-10-31 merge deadline, and the UMCSENT licence question from Licensing: record-and-track policy, and the per-dataset inventory for migration #35.mmcky commentedon Sep 30, 2026
Revision 2026-09-30: the canary proof is done, decisions 1 and 2 are taken, and blocks B and C are open as draft PRs.
builders/business_cycle_fred.py, and the workflow was dispatched against it twice (36654189187 and 36654357802). The canary exited 2, notify opened refresh-snapshots: a dynamic snapshot's builder is failing #150 and then commented on it, and in the second run the refresh job was skipped even though the file was due. The evidence is on Show the snapshot canary catching an induced validation failure (P4) #147 and is recorded on PLAN: pilot migration — one dataset per hosting pattern to validate the datasets design meta#338. refresh-snapshots: a dynamic snapshot's builder is failing #150 is closed as not planned.main(decision 1). UMCSENT stays,permittedon Michigan's FAQ for main-website data (decision 2). FRED stays a known yes, with its caching clause (p) recorded in AGENTS.md. licence: rest UMCSENT on Michigan's FAQ and record FRED's caching clause #152 records the two licence decisions and corrects two manifest errors the check turned up.yfinanceread inmle(Rewrite MLE as an elementary lecture on maximum likelihood lecture-python-intro#849, merged 2026-09-29), without which the strict audit fails onmainfrom its next run. Block B is business_cycle: read four data-lectures snapshots and re-enable the lecture lecture-wasm#85 and block C is Flip wave P4 to repointed — the four business_cycle snapshots #153, both drafts.The stamp is re-dated to 2026-09-30 after re-checking every row of the table. The body now carries the decisions, the new deadline for #151 and four facts learned today. Still open: decision 3 (
on_refresh, set torebuildin #153) and decision 4 (who merges the monthly refresh PRs).licence: rest UMCSENT on Michigan's FAQ and record FRED's caching cla…
mmcky commentedon Sep 30, 2026
Revision 2026-09-30 (later): #151 and #152 merged. #151 (05:02 UTC) annotated intro's
mleyfinanceread, somain's strict audit is green again. #152 (05:10 UTC) recorded the UMCSENT and FRED licence decisions. Theaudit-dashboardruns onmainafter each merge passed and deployed.Block C (#153) rebases cleanly onto the new
main. Its dry-run there gives exactly what #146 expects, with no stand-in annotation now that #151 is real: thelandedrecords with B's branch exit 1 with 4 warnings, C with B's branch exits 0, and C with lecture-wasm'smainexits 1. The two deadline rows these PRs met are removed from the body.Next: the first monthly refresh (expected Monday 2026-10-05), then Run all on QuantEcon/lecture-wasm#85's preview, then B and C on the same day.