Repository navigation
Fully static build pipeline MVP - #839
Conversation
With STATIC_BUILD set, the Astro config attaches no adapter, so a static build emits plain files and no serverless function. The three preview routes, which only make sense with a server, are prerendered with no paths in static mode by a small integration. Nothing changes when STATIC_BUILD is unset. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbHx5mB9KZKURcTMDh8B9e
Astro cannot build server islands without an adapter, and a static site has no server to render them later. With STATIC_BUILD set, a small plugin drops the server:defer directive from each component as it is loaded, before Astro compiles it, so every island renders into the page at build time. Server-rendered sites keep deferring. The plugin works at load time because Astro's own component plugin compiles in a pre-ordered transform that runs before any integration's transform. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbHx5mB9KZKURcTMDh8B9e
✅ Deploy Preview for padp-staging ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for pss-scavenger-hunt ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for marronage-staging ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for relnet-staging ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
iiif-cloud's create_static_assets endpoint now rejects requests without a storage credential (no more ENV fallback) and always uses its own server-side-configured storage endpoint/region, so this: - renames the request body key sent to iiif-cloud from r2 to storage, and drops the now-ignored endpoint sub-key from that request - renames R2_* env vars to STATIC_STORAGE_*, and config.r2 to config.storage - renames staticBuild.r2.mjs to staticBuild.storage.mjs (still used for this script's own upload of the built site, which still needs its own endpoint) - updates docs/static-build-pipeline.md to match, including removing the stale claim that iiif-cloud falls back to its own credentials, and noting that the storage endpoint/region can no longer be set per-request Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
blms
left a comment
blms
left a comment
There was a problem hiding this comment.
Looking great so far!
I think at this point for the first iteration we'll need to just run npm run static-build by hand from dev machines for each site. Not ideal but we are pretty much out of time on static builds. We can revisit once we get through some more client-facing frontend work on this site and see what time remains.
To keep the git tree from getting too twisted up this PR doesn't include the static maps piece, which is currently sitting in a separate branch; once both are merged, the static build script should be updated so that it also calls the static maps script as part of its workflow.
Oops sorry, just merged that! Once you merge develop in, a plan could be something like:
- Have the script run
npm run build:maps -- --export-dir . --out .static-mapsin parallel w/ the image step and the Astro build. It reads places fromapi/.- Write the maps to their own folder, not
dist/, because the Astro build clearsdist/when it starts.
- Write the maps to their own folder, not
- In the upload step, also upload that folder to
<destination>/_fds/maps/, if it isn't empty. - Add to the docs that this needs the
pmtilescommand (and git) .- And that each map layer needs a
static.urlin the site config:/_fds/maps/style.jsonfor the basemap, or a path in/_fds/maps/overlays/for overlays (for example/_fds/maps/overlays/historic/{z}/{x}/{y}.png). Seedocs/static-deploy.mdfor a table.
- And that each map layer needs a
| } | ||
| try { | ||
| const relatedRecords = JSON.parse(fs.readFileSync(`${DATA_BASE_PATH}/${indexName}/${uuid}/${model}.json`, 'utf-8'))[model]; | ||
| record.relatedRecords[model] = filterRelated(relatedRecords) |
There was a problem hiding this comment.
Media and taxonomies have no content collections, so I think they'll need to be stored in full. filterRelated keeps only uuid and the relationship fields, which leaves content_url undefined (so image URLs come out as /iiif/3/image/undefined/...) and taxonomy labels empty. Could we add an exception here for media_contents and taxonomies to skip filterRelated?
|
|
||
| // We need to modify the media contents to point at local records | ||
| // This process should be improved! | ||
| record.relatedRecords['media_contents'] = _.map(record.relatedRecords['media_contents'], (media: any) => { |
There was a problem hiding this comment.
The services and static search read media from relatedRecords.mediaContents, not media_contents, so this line, as well as line 52 and line 57 (when they are grabbing keys from the loop iteration) will need to be updated with the camel-case key.
|
|
||
| for (let page = 1; page <= pages; page += 1) { | ||
| logger.info(`Fetching page ${page}...`); | ||
| const listRecords = fs.existsSync('api/places/index.json') ? JSON.parse(fs.readFileSync(`api/places/index.json`, 'utf-8')).places : []; |
There was a problem hiding this comment.
I think this loader also runs for non-static builds when PRELOAD_MAP=true, so that'll be trying to hit the file system here. Those builds have no api/ directory, so those sites would lose their map geometry. Could it read api/ only when STATIC_BUILD, and use the API as before otherwise?
| * @param {Object} config - see staticBuild.config.mjs#resolveConfig | ||
| * @param {string} sourceDir - local directory to upload (e.g. "dist") | ||
| */ | ||
| export const uploadDirectory = async (config, sourceDir) => { |
There was a problem hiding this comment.
For the static maps integration: we'll need to let the same upload put a folder under a sub-path of destination, e.g. uploadDirectory(config, mapsDir, '_fds/maps'). Something like this:
| export const uploadDirectory = async (config, sourceDir) => { | |
| export const uploadDirectory = async (config, sourceDir, keyPrefix = '') => { |
And then on line 33:
const key = path.posix.join(config.destination, keyPrefix, ...relativePath.split(path.sep));| | Variable | Required | Description | | ||
| |---|---|---| | ||
| | `IIIF_CLOUD_URL` | Yes | Base URL of the iiif-cloud instance, e.g. `https://iiif.example.com`. | | ||
| | `IIIF_CLOUD_API_KEY` | Yes | API key for an admin iiif-cloud user account (required to call `create_static_assets`). | |
There was a problem hiding this comment.
IIIF_CLOUD_API_KEY doesn't need to be admin anymore right?
| ]; | ||
|
|
||
| const DEFAULT_POLL_INTERVAL_SECONDS = 10; | ||
| const DEFAULT_POLL_MAX_ATTEMPTS = 360; // 1 hour at the default interval |
There was a problem hiding this comment.
One other thought here, similar to the fairdata question about polling, should we go for a similar reset-on-progress approach here? Or maybe for MVP we just set the polling attempts really high for big projects locally? I imagine 1 hour won't be enough for larger projects but not sure.
blms
left a comment
blms
left a comment
There was a problem hiding this comment.
Looking great! Just a couple more issues with the camel case media contents thing. Once those are fixed feel free to merge.
Co-authored-by: Ben Silverman <ben.lm.silverman@gmail.com>
Co-authored-by: Ben Silverman <ben.lm.silverman@gmail.com>
In this PR
This PR revamps the
STATIC_BUILD=truebuild mode content loaders to pull from static JSON files sitting at/api/...rather than making HTTP requests to a live API. It also adds astatic-buildscript that triggers a data export from FairData, downloads and writes the created files to the local filesystem, and then triggers the Astro build process in static mode as well as calling FairImage to trigger level 0 IIIF asset generation, and writes the output to a provided R2 bucket with provided credentials. For detailed (LLM-generated, but proofread by me) documentation of the static build script, see the newdocs/static-build-pipeline.mdfile.To keep the git tree from getting too twisted up this PR doesn't include the static maps piece, which is currently sitting in a separate branch; once both are merged, the static build script should be updated so that it also calls the static maps script as part of its workflow.
Some notes and questions
pstudiothat ends up calling the script. The existing FDS site registry is a good starting point for organizing the necessary per-project parameters.galleriescontent loader; it still makes a live call to whatever link is provided in the config file. Ideally it should instead read that data from the correct path in the/apiJSON files, but that felt out of scope for this MVP given how few of the sites use the gallery feature for now... but should be flagged as something to come back to in the next batch of improvements.