|
| 1 | +<!-- |
| 2 | +SPDX-License-Identifier: CC-BY-SA-4.0 |
| 3 | +SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk> |
| 4 | +--> |
| 5 | +# KYAML adoption: upstream architecture and a low-risk path |
| 6 | + |
| 7 | +**Purpose:** explain where an authoring-format change would touch Joshua's upstream |
| 8 | +repository, separate safe formatting from runtime behavior, and give each review a small, |
| 9 | +explicit scope. This is an architecture review, not a request to adopt KYAML or a claim that |
| 10 | +an upstream change has been tested or merged. |
| 11 | + |
| 12 | +**Snapshot reviewed:** `JoshuaJewell/MetaManifold-WebUI` `main` at |
| 13 | +`ecefb1c72b3d2515e7086024b14227ef13329605` (2026-09-27). Recheck paths and tests against |
| 14 | +the target commit when preparing a PR. |
| 15 | + |
| 16 | +## Bottom line |
| 17 | + |
| 18 | +KYAML is a source-authoring convention, not a replacement parser or a new application data |
| 19 | +format. Upstream already reads configuration with `YAML.jl` 0.4. If a file remains valid |
| 20 | +YAML and its parsed values are unchanged, its existing consumers need no new dependency and |
| 21 | +no API or pipeline change. **Keep `YAML.jl`, existing serializers, and user configuration |
| 22 | +formats unchanged during the pilot.** |
| 23 | + |
| 24 | +The boundary that matters is who owns and writes each file: |
| 25 | + |
| 26 | +| Files / surface | Current consumer or writer | Safe transition rule | |
| 27 | +|---|---|---| |
| 28 | +| `config/defaults/*.yml` | Julia `YAML.load_file`; pipeline defaults feed the default → global → study → group → run cascade in `src/core/config.jl`. | Convert as repository-owned inputs; compare the values read by `YAML.jl` before/after. Do not change the cascade or hashes as part of a format-only patch. | |
| 29 | +| `config/ci/*.yml`, `bench/**.yml`, tracked sample `data/MiSeq_SOP/pipeline.yml` | CI/bootstrap, benchmark, or example-data readers. | Convert as authored inputs and run their existing consumers. Keep the sample's values and provenance intact. | |
| 30 | +| `.github/workflows/*.yml` | GitHub Actions' workflow parser—not the application YAML loader. Several files contain multiline shell `run:` blocks; Dependabot and `gh actions-lock` also rewrite workflow pins. | Convert only after an actual GitHub Actions parse/run probe. Preserve trigger, job IDs, `needs`, permissions, matrix values, environment, working directory, shell, and `GITHUB_ENV` behavior. Bot drift may exempt these files from the canonicality gate, but not from conversion or rollback. | |
| 31 | +| `config/pipeline.yml`, `config/primers.yml`, `config/databases.yml`, `config/composition.yml`, `config/tools.yml`, and per-study `pipeline.yml` overrides | Machine/user-editable overlays; routes in `src/server/routes/config.jl`, `composition.jl`, and `results.jl` write YAML with `YAML.write`. Root `.gitignore` excludes the machine-level config files. | Do not reformat user state or change save behavior in the pilot. The reader continues to accept YAML; runtime writes may remain block-style YAML. Keep generated/user files outside the tracked-file gate. | |
| 32 | +| `run_config.yml`, provenance sidecars, result/filter files | Generated from the config cascade or written as run output; `src/core/config.jl` and `src/core/provenance.jl` own these paths. | Leave generated artifacts and their writers alone. They are not repository-authored source configuration. | |
| 33 | + |
| 34 | +This means Joshua does **not** have to take a KYAML dependency or immediately migrate |
| 35 | +user-generated files. A format-only change to the tracked defaults can be read by the |
| 36 | +existing YAML loader; an edited user overlay can continue to be written in ordinary YAML. |
| 37 | + |
| 38 | +## Risks that deserve explicit evidence |
| 39 | + |
| 40 | +1. **Semantic equivalence, not visual similarity.** Quoting is intentional: KYAML quotes |
| 41 | + strings and keys that YAML readers may otherwise interpret differently (`on`, `no`, |
| 42 | + version-like strings). The gate must compare the actual parsed data from the production |
| 43 | + `YAML.jl` reader before and after emission for every tracked configuration document. |
| 44 | + The emitter's own YAML→KYAML→YAML round-trip is not enough by itself. |
| 45 | +2. **GitHub workflow interpretation.** The quoted `"on"` key must still trigger the |
| 46 | + workflow. Unit tests cannot establish this. A real PR run must report the expected jobs |
| 47 | + under their unchanged stable names, with no missing required status context. |
| 48 | +3. **Shell extraction is a separate behavior change.** Moving `run: |` blocks to |
| 49 | + `scripts/ci/*.sh` can change the working directory, shell options, expression expansion, |
| 50 | + environment, exit behavior, permissions, or writes to `GITHUB_ENV`. Keep that work in a |
| 51 | + separate reviewed change before the workflow-format rewrite, with a shell-level test or |
| 52 | + the same real CI execution. |
| 53 | +4. **Bot rewrites.** A bot reverting a workflow to block style should not make the quality |
| 54 | + gate flaky, but operators must still be able to canonicalize and roll back those files. |
| 55 | + The exemption must apply only to the gate—not to the converter. |
| 56 | +5. **User data and serializer scope.** Applying the converter to ignored machine configs, |
| 57 | + run directories, or provenance outputs would create churn and could rewrite user state. |
| 58 | + The source-format pilot should enumerate tracked YAML only and must not alter the |
| 59 | + application's YAML serialization paths. |
| 60 | +6. **Reversibility.** Preserve a byte-exact Git revert for the format-only conversion. |
| 61 | + `just use-yaml` is a canonical re-emission, not a promise to recover every original |
| 62 | + whitespace byte; the Git revert is the exact escape hatch. |
| 63 | + |
| 64 | +## Small, reviewable sequence |
| 65 | + |
| 66 | +### 0. Hardening patch — no source YAML rewritten |
| 67 | + |
| 68 | +Land the KYAML tool correction and its regression tests first. The drift list is a gate |
| 69 | +allowlist, not a migration exclusion. Test the converter and rollback against an exempt |
| 70 | +workflow path, and parse every tracked YAML document. Compare application-consumed |
| 71 | +configuration with the production `YAML.jl` reader; validate workflow files with GitHub |
| 72 | +Actions itself. Correct the corpus census and document the actual mutable-file boundary. |
| 73 | +This patch changes no workflow or application behavior. |
| 74 | + |
| 75 | +### 1. Fix and establish the CI verdict |
| 76 | + |
| 77 | +Resolve the current fork CI `startup_failure` before treating CI as evidence. A run with |
| 78 | +zero jobs has no test verdict. Then establish a passing baseline for the KYAML unit tests, |
| 79 | +workflow syntax checks, and existing full test suite on the exact review commit. |
| 80 | + |
| 81 | +### 2. Extract CI shell blocks (only if still required) |
| 82 | + |
| 83 | +If the project retains the pilot's script-extraction decision, do it separately from |
| 84 | +formatting. For every moved step, record the old and new `working-directory`, shell, |
| 85 | +`env`, GitHub expression, permissions, `set -e`/pipefail, and outputs. Run the workflow |
| 86 | +before changing its YAML style. |
| 87 | + |
| 88 | +### 3. Format-only conversion |
| 89 | + |
| 90 | +Convert only tracked, repository-authored `.yml`/`.yaml` files. Keep the migration as one |
| 91 | +format-only change (or one clearly named commit within the PR) so its purpose and revert |
| 92 | +boundary stay obvious. Its review description should report: |
| 93 | + |
| 94 | +- the exact file list and parser-equivalence result; |
| 95 | +- every decision count from `just kyaml-report` (`null`, quoting, comment movement); |
| 96 | +- the GitHub Actions run IDs that parsed and executed the converted workflows; |
| 97 | +- the required status contexts observed; |
| 98 | +- files intentionally not gated because of bot rewriting; and |
| 99 | +- confirmation that `Project.toml`, serializers, APIs, config-cascade behavior, run hashes, |
| 100 | + and user data are unchanged. |
| 101 | + |
| 102 | +Enable the canonicality gate in the same migration change. Do not claim the GitHub workflow |
| 103 | +proof is complete merely because the ordinary YAML parser accepts the file. |
| 104 | + |
| 105 | +## Findings from this checkout that must be resolved before migration |
| 106 | + |
| 107 | +- The repository has **17 tracked YAML files**, not 16. Two workflow paths are listed in |
| 108 | + `config/kyaml/drift.txt`; that makes 15 files subject to the canonicality gate after |
| 109 | + conversion, while all 17 still need parser/round-trip coverage. |
| 110 | +- Before the current hardening change, the CLI filtered the drift paths out of every mode. |
| 111 | + That meant the workflows would not be converted by `use-kyaml` and would not be restored |
| 112 | + by `use-yaml`, despite the pilot saying they are in scope. The implementation now applies |
| 113 | + those exemptions only to `--check`; a regression test covers conversion, gate exemption, |
| 114 | + and rollback. |
| 115 | +- The CLI also referred to `Stats` from outside its `KYAML` module without qualifying the |
| 116 | + name. It now uses `KYAML.Stats`; the new CLI-level regression test exercises that path. |
| 117 | +- The old parser-corpus test skipped the two workflows. It now includes every tracked file, |
| 118 | + compares both emitted forms with `YAML.jl` for application-consumed YAML, and leaves |
| 119 | + workflow acceptance to GitHub Actions itself. The current Actions workflow contains only |
| 120 | + a placeholder comment about adding the canonicality gate; it does not yet run |
| 121 | + `check-kyaml`. Add the real gate in the same change that converts the corpus. |
| 122 | +- **These Julia tests have not yet been executed in this sandbox**: Julia is absent, and the |
| 123 | + latest fork CI run ended in `startup_failure` before starting any jobs. Therefore this |
| 124 | + audit does not certify that all 17 files pass or that GitHub has accepted a converted |
| 125 | + workflow. |
| 126 | + |
| 127 | +## Upstream source pointers |
| 128 | + |
| 129 | +- [`Project.toml`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/Project.toml) — YAML.jl 0.4 remains a runtime dependency. |
| 130 | +- [`src/core/config.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/core/config.jl) — defaults and override cascade; generated `run_config.yml`. |
| 131 | +- [`src/server/routes/config.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/server/routes/config.jl) — config reads and atomic YAML writes for editable overlays. |
| 132 | +- [`src/core/primers_library.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/core/primers_library.jl) and [`src/core/databases_library.jl`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/src/core/databases_library.jl) — user-edited library files and normalization/validation boundaries. |
| 133 | +- [`.github/workflows/ci.yml`](https://github.com/JoshuaJewell/MetaManifold-WebUI/blob/main/.github/workflows/ci.yml) — CI parser, shell steps, Julia matrix, and tool installation. |
0 commit comments