Skip to content

Commit fbb4689

Browse files
Merge branch 'main' into arena/01a0db23-metamanifold-webui
2 parents 7aa274f + e64d74b commit fbb4689

8 files changed

Lines changed: 218 additions & 37 deletions

File tree

‎config/kyaml/drift.txt‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,16 +2,18 @@
22
# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
33
#
44
# Files that `just check-kyaml` does NOT require to be canonical KYAML, because a
5-
# tool that is not this repository writes them. One path prefix per line; `#`
6-
# starts a comment. Every entry needs a reason and an owner, and the list is
7-
# reviewed with docs/pilots/kyaml-pilot.md — an exemption without a reason is drift.
5+
# tool that is not this repository writes them. This is a check-only exemption:
6+
# `use-kyaml` and `use-yaml` still convert and roll these files back. One path
7+
# prefix per line; `#` starts a comment. Every entry needs a reason and an owner,
8+
# and the list is reviewed with docs/pilots/kyaml-pilot.md — an exemption without
9+
# a reason is drift.
810
#
911
# Owner ruling, 2026-09-26: the drift is ACCEPTED IN WRITING (standards
1012
# YAML-POLICY.adoc §5 step 6 requires either KYAML-emitting bots or an explicit
11-
# written acceptance). The files below ARE converted to KYAML — they are simply
12-
# not gated, because Dependabot and `gh actions-lock` rewrite their `uses:` pins
13-
# in block style, and a gate that goes red on a bot's schedule trains everyone to
14-
# ignore it. Reconcile after a bot PR with: just use-kyaml
13+
# written acceptance). These paths remain in the conversion scope, but are not
14+
# gated: Dependabot and `gh actions-lock` rewrite their `uses:` pins in block
15+
# style, and a gate that goes red on a bot's schedule trains everyone to ignore
16+
# it. Reconcile after a bot PR with: just use-kyaml
1517
#
1618
# .github/workflows/ci.yml Dependabot + gh actions-lock rewrite `uses:` pins
1719
# .github/workflows/ui.yml same

‎docs/integration/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,11 @@ empty and a normal merge surfaces all 159 shared‑but‑different files at once
1616
Nothing in this guide changes upstream's operational behaviour unless you choose
1717
it to; the default is "behave exactly like upstream".
1818

19+
For the separate KYAML authoring-format pilot, see the [upstream architecture and
20+
transition audit](./kyaml-upstream-adoption.md). It identifies the runtime config readers,
21+
user-editable YAML writers, GitHub Actions boundary, and the evidence required before
22+
changing any tracked files.
23+
1924
## The idea, in layers
2025

2126
0. **Re‑anchor** — replay the fork's commits one‑by‑one onto upstream so git
Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
<!--
2+
SPDX-License-Identifier: CC-BY-SA-4.0
3+
SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
4+
-->
5+
# KYAML adoption: upstream architecture and a low-risk path
6+
7+
**Purpose:** explain where an authoring-format change would touch Joshua's upstream
8+
repository, separate safe formatting from runtime behavior, and give each review a small,
9+
explicit scope. This is an architecture review, not a request to adopt KYAML or a claim that
10+
an upstream change has been tested or merged.
11+
12+
**Snapshot reviewed:** `JoshuaJewell/MetaManifold-WebUI` `main` at
13+
`ecefb1c72b3d2515e7086024b14227ef13329605` (2026-09-27). Recheck paths and tests against
14+
the target commit when preparing a PR.
15+
16+
## Bottom line
17+
18+
KYAML is a source-authoring convention, not a replacement parser or a new application data
19+
format. Upstream already reads configuration with `YAML.jl` 0.4. If a file remains valid
20+
YAML and its parsed values are unchanged, its existing consumers need no new dependency and
21+
no API or pipeline change. **Keep `YAML.jl`, existing serializers, and user configuration
22+
formats unchanged during the pilot.**
23+
24+
The boundary that matters is who owns and writes each file:
25+
26+
| Files / surface | Current consumer or writer | Safe transition rule |
27+
|---|---|---|
28+
| `config/defaults/*.yml` | Julia `YAML.load_file`; pipeline defaults feed the default → global → study → group → run cascade in `src/core/config.jl`. | Convert as repository-owned inputs; compare the values read by `YAML.jl` before/after. Do not change the cascade or hashes as part of a format-only patch. |
29+
| `config/ci/*.yml`, `bench/**.yml`, tracked sample `data/MiSeq_SOP/pipeline.yml` | CI/bootstrap, benchmark, or example-data readers. | Convert as authored inputs and run their existing consumers. Keep the sample's values and provenance intact. |
30+
| `.github/workflows/*.yml` | GitHub Actions' workflow parser—not the application YAML loader. Several files contain multiline shell `run:` blocks; Dependabot and `gh actions-lock` also rewrite workflow pins. | Convert only after an actual GitHub Actions parse/run probe. Preserve trigger, job IDs, `needs`, permissions, matrix values, environment, working directory, shell, and `GITHUB_ENV` behavior. Bot drift may exempt these files from the canonicality gate, but not from conversion or rollback. |
31+
| `config/pipeline.yml`, `config/primers.yml`, `config/databases.yml`, `config/composition.yml`, `config/tools.yml`, and per-study `pipeline.yml` overrides | Machine/user-editable overlays; routes in `src/server/routes/config.jl`, `composition.jl`, and `results.jl` write YAML with `YAML.write`. Root `.gitignore` excludes the machine-level config files. | Do not reformat user state or change save behavior in the pilot. The reader continues to accept YAML; runtime writes may remain block-style YAML. Keep generated/user files outside the tracked-file gate. |
32+
| `run_config.yml`, provenance sidecars, result/filter files | Generated from the config cascade or written as run output; `src/core/config.jl` and `src/core/provenance.jl` own these paths. | Leave generated artifacts and their writers alone. They are not repository-authored source configuration. |
33+
34+
This means Joshua does **not** have to take a KYAML dependency or immediately migrate
35+
user-generated files. A format-only change to the tracked defaults can be read by the
36+
existing YAML loader; an edited user overlay can continue to be written in ordinary YAML.
37+
38+
## Risks that deserve explicit evidence
39+
40+
1. **Semantic equivalence, not visual similarity.** Quoting is intentional: KYAML quotes
41+
strings and keys that YAML readers may otherwise interpret differently (`on`, `no`,
42+
version-like strings). The gate must compare the actual parsed data from the production
43+
`YAML.jl` reader before and after emission for every tracked configuration document.
44+
The emitter's own YAML→KYAML→YAML round-trip is not enough by itself.
45+
2. **GitHub workflow interpretation.** The quoted `"on"` key must still trigger the
46+
workflow. Unit tests cannot establish this. A real PR run must report the expected jobs
47+
under their unchanged stable names, with no missing required status context.
48+
3. **Shell extraction is a separate behavior change.** Moving `run: |` blocks to
49+
`scripts/ci/*.sh` can change the working directory, shell options, expression expansion,
50+
environment, exit behavior, permissions, or writes to `GITHUB_ENV`. Keep that work in a
51+
separate reviewed change before the workflow-format rewrite, with a shell-level test or
52+
the same real CI execution.
53+
4. **Bot rewrites.** A bot reverting a workflow to block style should not make the quality
54+
gate flaky, but operators must still be able to canonicalize and roll back those files.
55+
The exemption must apply only to the gate—not to the converter.
56+
5. **User data and serializer scope.** Applying the converter to ignored machine configs,
57+
run directories, or provenance outputs would create churn and could rewrite user state.
58+
The source-format pilot should enumerate tracked YAML only and must not alter the
59+
application's YAML serialization paths.
60+
6. **Reversibility.** Preserve a byte-exact Git revert for the format-only conversion.
61+
`just use-yaml` is a canonical re-emission, not a promise to recover every original
62+
whitespace byte; the Git revert is the exact escape hatch.
63+
64+
## Small, reviewable sequence
65+
66+
### 0. Hardening patch — no source YAML rewritten
67+
68+
Land the KYAML tool correction and its regression tests first. The drift list is a gate
69+
allowlist, not a migration exclusion. Test the converter and rollback against an exempt
70+
workflow path, and parse every tracked YAML document. Compare application-consumed
71+
configuration with the production `YAML.jl` reader; validate workflow files with GitHub
72+
Actions itself. Correct the corpus census and document the actual mutable-file boundary.
73+
This patch changes no workflow or application behavior.
74+
75+
### 1. Fix and establish the CI verdict
76+
77+
Resolve the current fork CI `startup_failure` before treating CI as evidence. A run with
78+
zero jobs has no test verdict. Then establish a passing baseline for the KYAML unit tests,
79+
workflow syntax checks, and existing full test suite on the exact review commit.
80+
81+
### 2. Extract CI shell blocks (only if still required)
82+
83+
If the project retains the pilot's script-extraction decision, do it separately from
84+
formatting. For every moved step, record the old and new `working-directory`, shell,
85+
`env`, GitHub expression, permissions, `set -e`/pipefail, and outputs. Run the workflow
86+
before changing its YAML style.
87+
88+
### 3. Format-only conversion
89+
90+
Convert only tracked, repository-authored `.yml`/`.yaml` files. Keep the migration as one
91+
format-only change (or one clearly named commit within the PR) so its purpose and revert
92+
boundary stay obvious. Its review description should report:
93+
94+
- the exact file list and parser-equivalence result;
95+
- every decision count from `just kyaml-report` (`null`, quoting, comment movement);
96+
- the GitHub Actions run IDs that parsed and executed the converted workflows;
97+
- the required status contexts observed;
98+
- files intentionally not gated because of bot rewriting; and
99+
- confirmation that `Project.toml`, serializers, APIs, config-cascade behavior, run hashes,
100+
and user data are unchanged.
101+
102+
Enable the canonicality gate in the same migration change. Do not claim the GitHub workflow
103+
proof is complete merely because the ordinary YAML parser accepts the file.
104+
105+
## Findings from this checkout that must be resolved before migration
106+
107+
- The repository has **17 tracked YAML files**, not 16. Two workflow paths are listed in
108+
`config/kyaml/drift.txt`; that makes 15 files subject to the canonicality gate after
109+
conversion, while all 17 still need parser/round-trip coverage.
110+
- Before the current hardening change, the CLI filtered the drift paths out of every mode.
111+
That meant the workflows would not be converted by `use-kyaml` and would not be restored
112+
by `use-yaml`, despite the pilot saying they are in scope. The implementation now applies
113+
those exemptions only to `--check`; a regression test covers conversion, gate exemption,
114+
and rollback.
115+
- The CLI also referred to `Stats` from outside its `KYAML` module without qualifying the
116+
name. It now uses `KYAML.Stats`; the new CLI-level regression test exercises that path.
117+
- The old parser-corpus test skipped the two workflows. It now includes every tracked file,
118+
compares both emitted forms with `YAML.jl` for application-consumed YAML, and leaves
119+
workflow acceptance to GitHub Actions itself. The current Actions workflow contains only
120+
a placeholder comment about adding the canonicality gate; it does not yet run
121+
`check-kyaml`. Add the real gate in the same change that converts the corpus.
122+
- **These Julia tests have not yet been executed in this sandbox**: Julia is absent, and the
123+
latest fork CI run ended in `startup_failure` before starting any jobs. Therefore this
124+
audit does not certify that all 17 files pass or that GitHub has accepted a converted
125+
workflow.
126+
127+
## Upstream source pointers
128+
129+
- [`Project.toml`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/Project.toml) — YAML.jl 0.4 remains a runtime dependency.
130+
- [`src/core/config.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/core/config.jl) — defaults and override cascade; generated `run_config.yml`.
131+
- [`src/server/routes/config.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/server/routes/config.jl) — config reads and atomic YAML writes for editable overlays.
132+
- [`src/core/primers_library.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/core/primers_library.jl) and [`src/core/databases_library.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/core/databases_library.jl) — user-edited library files and normalization/validation boundaries.
133+
- [`.github/workflows/ci.yml`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/.github/workflows/ci.yml) — CI parser, shell steps, Julia matrix, and tool installation.

‎docs/pilots/kyaml-pilot.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,9 @@ use-kyaml` / `just use-yaml`, and the escape hatch is one `git revert`.
99

1010
Authority: `hyperpolymath/standards`, `3-practice/YAML-POLICY.adoc` (rules Y-1, Y-2, Y-3,
1111
adoption order §5). This document is the operating manual for the pilot; the policy stays
12-
the authority, and where the two disagree the policy wins and this file is wrong.
12+
the authority, and where the two disagree the policy wins and this file is wrong. The
13+
upstream application touchpoints and the low-risk adoption sequence are mapped in
14+
[`docs/integration/kyaml-upstream-adoption.md`](../integration/kyaml-upstream-adoption.md).
1315

1416
## 1. Why this is worth doing, in one paragraph
1517

@@ -80,7 +82,7 @@ Refused, by name and line number, leaving the file untouched:
8082
| Multi-line plain scalars | Folded into one line by YAML rules, so the source bytes are not recoverable. |
8183
| An end-of-line comment on a block scalar | The scalar owns the rest of the line; there is nowhere lossless to put the comment. |
8284

83-
This repository's corpus needs none of those: a census of the 16 tracked YAML files found
85+
This repository's corpus needs none of those: a census of the 17 tracked YAML files found
8486
**zero** anchors, aliases, tags, multi-document streams or directives; block scalars in two
8587
files (`ci.yml`, 22 of them; `.github/ISSUE_TEMPLATE/bug_report.yml`, 2); 12 `~` nulls; and
8688
comment-bearing lines concentrated in `ci.yml` (313), `config/defaults/pipeline.yml` (97) and

‎scripts/kyaml/KYAML.jl‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1234,7 +1234,7 @@ function _kyaml_cli(argv::Vector{String})::Int
12341234
12351235
Options:
12361236
--report print the decisions taken per file
1237-
--skip-file PATH file listing paths to leave alone (default config/kyaml/drift.txt)
1237+
--skip-file PATH --check-only exemptions (default config/kyaml/drift.txt)
12381238
--expect kyaml|yaml style --check compares against (default kyaml)
12391239
12401240
Exit codes: 0 ok, 1 a check failed, 2 a file was refused (nothing was written).
@@ -1256,10 +1256,18 @@ function _kyaml_cli(argv::Vector{String})::Int
12561256
end
12571257
end
12581258
isempty(paths) && (paths = KYAML.git_yaml_paths())
1259-
kept = [p for p in paths if !any(s -> startswith(p, s), skip)]
1259+
# Drift exemptions are a gate policy, not a migration boundary. Bot-owned
1260+
# workflows are still converted (and can be rolled back); only --check omits
1261+
# them, so an external formatter cannot make the canonicality gate flap.
1262+
skipped = String[]
1263+
kept = paths
1264+
if mode == "check"
1265+
skipped = [p for p in paths if any(s -> startswith(p, s), skip)]
1266+
kept = [p for p in paths if !any(s -> startswith(p, s), skip)]
1267+
end
12601268

12611269
failures = String[]
1262-
reports = Dict{String,Stats}()
1270+
reports = Dict{String,KYAML.Stats}()
12631271
rendered = Dict{String,String}()
12641272
for p in kept
12651273
stats = KYAML.Stats()
@@ -1285,7 +1293,7 @@ function _kyaml_cli(argv::Vector{String})::Int
12851293
if mode == "check"
12861294
if isempty(failures)
12871295
println("kyaml check: $(length(kept)) file(s) in canonical $expect form" *
1288-
(isempty(skip) ? "" : ", $(length(skip)) exempt"))
1296+
(isempty(skipped) ? "" : ", $(length(skipped)) exempt"))
12891297
return 0
12901298
end
12911299
println(stderr, "kyaml check: $(length(failures)) file(s) are not canonical $expect:")
@@ -1302,7 +1310,7 @@ function _kyaml_cli(argv::Vector{String})::Int
13021310
written += 1
13031311
end
13041312
println("kyaml $mode: $(written) of $(length(kept)) file(s) rewritten" *
1305-
(isempty(skip) ? "" : ", $(length(skip)) exempt"))
1313+
(isempty(skipped) ? "" : ", $(length(skipped)) exempt"))
13061314
if report
13071315
for p in sort(collect(keys(reports)))
13081316
s = reports[p]

‎test/doi/fixtures.jl‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ function bundle_fixture(root; result=false, mock=false, dangerous=false)
3131
"method" => "nb_glm", "provenance" => Dict("mock" => mock),
3232
"results" => Dict("synthetic_fixture" => Dict("status" => "fixture")))))
3333
end
34-
B.write_checksums!(dir)
34+
Target.DOIBundles.write_checksums!(dir)
3535
return dir
3636
end
3737

@@ -102,13 +102,13 @@ function prepared_fixture(f; result=false, dangerous=false, environment="sandbox
102102
root = joinpath(tmp, "publications")
103103
fake = FakeZenodo(; environment)
104104
c = client(fake)
105-
prepared = P.prepare!(root, source, metadata_fixture(), c)
105+
prepared = Target.DOIPublications.prepare!(root, source, metadata_fixture(), c)
106106
f(tmp, root, source, fake, c, prepared)
107107
end
108108
end
109109

110110
function publish_fixture(root, prepared, c)
111-
P.publish!(root, prepared["id"], c; confirmation=prepared["confirmation_phrase"],
111+
Target.DOIPublications.publish!(root, prepared["id"], c; confirmation=prepared["confirmation_phrase"],
112112
bundle_sha256=prepared["bundle_sha256"], acknowledge_public=true)
113113
end
114114

0 commit comments

Comments
 (0)