Skip to content

Commit ac68bba

Browse files
fix(kyaml): escape U+2028 and U+2029 so keys stay parseable by yq (#250)
Follow-up to #248 (KYAML report output), found while drafting the upstream yq issue about its KYAML encoder. ## Problem go-yaml, the parser behind `yq` and standards' `scripts/kyaml-format.sh`, treats a raw U+2028 (LS) or U+2029 (PS) as a line break, which is YAML 1.1 behaviour. Our emitter already escaped NEL (U+0085) but wrote LS and PS raw. A report key holding either one therefore produced KYAML that `yq` could not read at all: ``` $ yq -p yaml -o json old.kyaml # key "x<LS>y" written raw Error: ... go-yaml load error in parser (while parsing a flow mapping) ... did not find expected ',' or '}' ``` Values were unaffected, and `crate::yaml` (serde-saphyr, YAML 1.2) read these files correctly before and after. In #248 I deliberately left U+2028 raw because YAML 1.2 calls it printable. Testing keys showed that was the wrong call for interop. ## Fix `write_quoted` also escapes U+2028 and U+2029 as ` ` and ` `. Those escapes are valid in YAML 1.1 and 1.2, so nothing that parsed the old output loses anything. ## Validation - `cargo fmt --check`, plus `cargo clippy --all-targets -D warnings` with and without `--all-features`: clean. - `cargo test --all-features`: all suites green (lib: 510 passed). - **New test `line_breaks_in_keys_are_escaped`** covers NEL, LS and PS in keys: exact output, and a round trip through `crate::yaml`. Run against the previous escape set, it fails with the raw separator in the output (`"x\u{2028}y"`). - `escapes_non_printables_that_yq_writes_raw` is updated: it asserted LS stays raw, and now asserts it is escaped. - **End to end with real `yq` v4.54.1:** KYAML from this branch with NEL, LS or PS in a key reads back intact (`yq -p yaml -o json`). ## One thing this PR can't fix `kyaml-format.sh --check` still **refuses** these files: `REFUSED — the KYAML rendering does not parse to the same data`. The gate re-renders through `yq -o kyaml`, and yq's encoder writes the characters raw again. That is the upstream yq bug: - With a `yq` built from my proposed upstream patch, the same three files pass (`all 3 file(s) are KYAML`). - The upstream issue is drafted but **not filed yet**; filing it is the owner's call. Until yq is fixed, a report key containing NEL, LS or PS can't pass the gate. Such keys are rare in practice, since report keys are field names and file paths. --- _Generated by [Claude Code](https://claude.ai/code/session_01PFaqv6iBXrt3W5Akd1Fq5k)_ Co-authored-by: Claude <noreply@anthropic.com>
1 parent 2285a77 commit ac68bba

2 files changed

Lines changed: 37 additions & 10 deletions

File tree

‎CHANGELOG.adoc‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,14 @@
22

33
=== [Unreleased]
44

5+
==== Fixed — KYAML keys holding a line separator
6+
7+
* KYAML output now escapes U+2028 (LS) and U+2029 (PS) as `+\uXXXX+`, as it
8+
already did for NEL (U+0085). go-yaml, the parser behind `+yq+`, reads all
9+
three as line breaks, so a raw one in a key made the document unparseable
10+
by `+yq+` and by standards' `+kyaml-format.sh+`.
11+
`+crate::yaml+` (serde-saphyr) read these files correctly before and after.
12+
513
==== Changed — attestation hashes are reproducible JCS
614

715
* `+assail --attest+` now hashes the intent and evidence records as their

‎src/kyaml.rs‎

Lines changed: 29 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,13 @@
1616
//! of the same scan differ only in syntax.
1717
//!
1818
//! One deliberate difference from `yq`: YAML's non-printable characters
19-
//! (U+007F to U+009F, U+FFFE, U+FFFF) are escaped as `\uXXXX`. `yq` writes
20-
//! them raw, which YAML 1.2 does not allow, and U+0085 (NEL) written raw
21-
//! reads back as a line break, corrupting the string. Strings holding one
22-
//! are therefore not a `yq` fixed point; `kyaml-format.sh` refuses such a
23-
//! file instead of judging it.
19+
//! (U+007F to U+009F, U+FFFE, U+FFFF) and the YAML 1.1 line breaks U+2028
20+
//! (LS) and U+2029 (PS) are escaped as `\uXXXX`. `yq` writes them raw, which
21+
//! YAML 1.2 does not allow for the non-printables. go-yaml, the parser
22+
//! behind `yq`, reads a raw U+0085 (NEL), LS or PS as a line break: in a
23+
//! value NEL is folded to a space, and in a key any of the three makes the
24+
//! document unparseable. Strings holding one are therefore not a `yq` fixed
25+
//! point; `kyaml-format.sh` refuses such a file instead of judging it.
2426
2527
use anyhow::{Context, Result};
2628
use serde::Serialize;
@@ -104,9 +106,12 @@ fn write_quoted(out: &mut String, text: &str) {
104106
'\n' => out.push_str("\\n"),
105107
'\r' => out.push_str("\\r"),
106108
'\t' => out.push_str("\\t"),
107-
'\0'..='\u{1F}' | '\u{7F}'..='\u{9F}' | '\u{FFFE}' | '\u{FFFF}' => {
108-
out.push_str(&format!("\\u{:04X}", c as u32))
109-
}
109+
'\0'..='\u{1F}'
110+
| '\u{7F}'..='\u{9F}'
111+
| '\u{2028}'
112+
| '\u{2029}'
113+
| '\u{FFFE}'
114+
| '\u{FFFF}' => out.push_str(&format!("\\u{:04X}", c as u32)),
110115
_ => out.push(c),
111116
}
112117
}
@@ -150,17 +155,31 @@ mod tests {
150155

151156
#[test]
152157
fn escapes_non_printables_that_yq_writes_raw() {
153-
let text = "a\u{7F}\u{85}\u{9F}\u{FFFE}\u{FFFF}\u{A0}\u{2028}z";
158+
let text = "a\u{7F}\u{85}\u{9F}\u{2028}\u{2029}\u{FFFE}\u{FFFF}\u{A0}z";
154159
let out = to_string(text).unwrap();
155160
assert_eq!(
156161
out,
157-
"\"a\\u007F\\u0085\\u009F\\uFFFE\\uFFFF\u{A0}\u{2028}z\"\n"
162+
"\"a\\u007F\\u0085\\u009F\\u2028\\u2029\\uFFFE\\uFFFF\u{A0}z\"\n"
158163
);
159164
// ...and they read back intact, NEL included.
160165
let back: String = crate::yaml::from_str(&out).unwrap();
161166
assert_eq!(back, text);
162167
}
163168

169+
#[test]
170+
fn line_breaks_in_keys_are_escaped() {
171+
// go-yaml reads a raw NEL, LS or PS in a key as a line break and
172+
// rejects the document, so keys are escaped like values.
173+
for c in ['\u{85}', '\u{2028}', '\u{2029}'] {
174+
let key = format!("x{c}y");
175+
let value = json!({ key.clone(): 1 });
176+
let out = to_string(&value).unwrap();
177+
assert_eq!(out, format!("{{\n \"x\\u{:04X}y\": 1,\n}}\n", c as u32));
178+
let back: serde_json::Value = crate::yaml::from_str(&out).unwrap();
179+
assert_eq!(back, value);
180+
}
181+
}
182+
164183
#[test]
165184
fn keys_follow_jcs_order() {
166185
// UTF-16 order puts U+10000 (a surrogate pair) before U+FF61;

0 commit comments

Comments
 (0)