Term pipeline: replace 763k per-term content files with bulk extraction and Hugo content adapters

[Robbie1977]

Summary

The nightly static-site pipeline cannot complete in a day. Both halves are dominated by per-term work over NFS, and neither is fixable by tuning:

job	measured	notes
update_vfb_terms_to_static_site #265	10 h	created=0 skipped=762,990 failed=1 — ten hours to write nothing
update_vfb_static_site (2026-08-05)	11 h 38 m, did not finish	never left content assembly; only the 66 static files were ever staged

Total >21 h for a job scheduled nightly.

This is not a regression from the theme change (#414) — that merge is what made it visible, because it forced the first full rebuild in a long time. The underlying cost has been there and growing with the corpus.

Proposal: stop materialising one markdown file per term. Extract in bulk, and generate the pages at build time with a Hugo content adapter.

Current cost, measured

Term generation — build #265, 762,991 terms across 13 segments:

segment	terms
VFB individuals	630,089
FlyBase features	69,595
FBbt	28,772
VFBexp	25,106
GO	5,322
SO	2,443
7 others	~1,664
total	762,991

The run created zero pages — every term was skipped as unchanged — and still took 10 hours. That is ~47 ms per term purely to decide nothing needed doing, i.e. the cost is per-term round trips, not page generation.

Site build. The workspace is NFS (129.215.164.35:/volume1/docker//jenkins_home/workspace/update_vfb_static_site, vers=3, soft). Hugo must enumerate and read the whole content tree before rendering anything. Measured on the live workspace:

• content/en/blog/ontologies/fbbt/ — 49,509 entries, 15.5 s just to ls (0.36 s user, 1.1 s sys; the rest is network wait)
• content/en/blog/ontologies/flybase/ — 135,577 entries
• content/en/blog/ontologies/vfb/ — on the order of 630k, consistent with Intermittent 'The client was not able to reconnect to the Backend, the page will be reloaded.' error #265

Concurrency makes it worse, not better. Two concurrent directory listings took 76 s against ~23.5 s for the same two run one after the other — 3.2× worse than serial. The NAS degrades under parallel metadata load, so Hugo's default worker count (47 on that host) is close to the worst possible configuration.

Diagnostic note for anyone who repeats this: rchar/syscr in /proc/<pid>/io count read(), not getdents64. A build walking the corpus therefore shows frozen read counters, near-zero CPU and flat RSS, which looks identical to a hang. Watch RSS and the staging file count instead.

Why the current design is expensive

One filesystem object per term means the corpus is ~763,000 files, 88% of them in two flat directories. Every build pays:

1. a full NFS walk to enumerate them,
2. an open + read + parse per file,
3. an rsync that has to stat all of them again to publish.

And the generation job pays a separate per-term API round trip to decide whether each file needs rewriting.

Proposal

1. Bulk extraction. Replace vfbterms.py's per-term queries with bulk retrieval — Solr cursor export or a single Neo4j query per segment — writing a small number of data files (one per ontology, or sharded) rather than 763k markdown files.

2. Content adapters. A _content.gotmpl per section reads those data files and emits pages at build time. No per-term files exist on disk at any point.

Benchmarked with the current vfb-nova theme and realistic term bodies, on a 2-core container:

pages	wall	peak RSS
5,132	19 s	—
20,132	49 s	2.57 GB

Linear at ~2 ms/page. 763k extrapolates to ~25 min on 2 cores, and single-digit minutes on the 47-core builder — against 21+ hours today.

Risks and open questions

• Memory. 763k pages at 52–110 kB/page marginal is 40–85 GB. The builder has ~170 GB available, so it fits, but this is the number to watch.
• Publishing does not improve. ~763k HTML files still have to reach public/ on the NAS. Content adapters remove the read cost entirely; the write cost is untouched and is probably the next bottleneck.
• The incremental skip is lost — rsync --checksum must replace it. Today only changed terms are rewritten, so public/ sees few changes. With adapters Hugo regenerates every page, so every file gets a fresh mtime and rsync -a would re-send all 763k. --checksum compares content instead. deploy.sh already notes this as an option; it becomes required.
• URL stability. Any change here must preserve existing term URLs exactly. [permalinks] is unchanged by Feature/vfb nova theme #414 and must stay that way.
• Is a static page per term still the right architecture for 763k entities, or should some of this be served dynamically? Worth asking explicitly rather than assuming, though SEO and stable deep links are real arguments for keeping it static.

Interim mitigation

Already applied to the hugo_builder Rancher service, no code change needed — deploy.sh takes both as environment overrides:

STAGE_DIR     = /build
HUGO_CACHEDIR = /hugo_cache

Both were previously on the same NFS export as the workspace, so the staged site was being written back over the network as well as read from it. These move them to container-local disk (9.5 TB free).

A further stopgap, not yet applied: keep a persistent local mirror of content/, rsync it from NFS each run (delta only), and build from the mirror. This needs a small change to deploy.sh so LIVE_DIR is independent of SRC_DIR — currently LIVE="$SRC/public", so overriding SRC_DIR would move the publish target too. It fixes the 11-hour half only; the 10-hour generation job is untouched by it.

Related

• Feature/vfb nova theme #414 — theme replacement; the merge that surfaced this
• update_vfb_terms_to_static_site #265 — the 10 h no-op run