Skip to content

Commit 29b2f8c

Browse files
claude[bot]claude
andauthored
fix(scripts): scan a template's ${...} as code so a nested backtick cannot flip js-comment-mask's parity (#10632)
* fix(scripts): scan a template's ${...} as code, so a nested backtick cannot flip the mask's parity scanSource walked a template literal's interpolation as plain literal text and stopped the span at the next backtick. An interpolation is code: it can hold a nested template (how this tree formats a list of names), a backtick inside a regex or a string, or a brace inside a string. The nested opener was read as the outer template's closer, and the phantom span ran to the next backtick anywhere in the file. Measured against @typescript-eslint/parser's comment ranges over 4,733 files: 16 files disagreed before (15 in the FABRICATES direction the module's own header calls the worse one, up to 10,252 comment bytes read as live code in one file), 0 after. The interpolation is now scanned by the same loop, with the same string, regex and comment branches, and its bytes are still reported as the enclosing template's literal content -- the documented flag is unchanged, so both consumers of `literal` (check-parse-guard, check-entry-guard) see what they saw. Seven shapes pinned in --self-test, each asserting both sides (a comment that must go, live code that must stay); all 22 cases verified byte-for-byte against the parser. The header's "cannot fabricate a lead" guarantee was measured false and is replaced by what can be re-derived. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DdCnBGcHeufjrq7drTD3wt * test(scripts): pin the interpolation brace shape with a case that a mutation can kill The `${x ? '}' : '{'}` spelling passed with the brace counting deleted AND with the pre-fix scanner -- it asserted "no error" and pinned nothing. Replaced with `${fmt({ a: 1 }, '`')}`, which carries a nested brace and a quoted backtick in one interpolation: deleting the `{` counting ends the interpolation at the object literal's `}`, the quoted backtick is then read as the template's closer, and the docblock below survives the mask. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DdCnBGcHeufjrq7drTD3wt * test(scripts): pin the interpolation whose object literal precedes a nested template Deleting the `{` counting inside `${...}` passed all 22 cases and the whole 4,733-file sweep -- the tree does not currently write that shape, so nothing held it. `${fmt({ a: 1 }, `\``)}` does: without the counting the interpolation ends at the object literal's `}`, the nested template's delimiters are then read in body position, and the docblock below survives the mask. Also records what the mutation runs showed about depth: matched backticks pair off whatever a scan believes about nesting, so nested-in-nested ALONE is green under every mutation. Only nesting that meets an escape or a quoted backtick discriminates. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DdCnBGcHeufjrq7drTD3wt * docs(scripts): spell out the sweep that measures this scanner, and what it missed The header pointed at "the sweep" without saying how to run one. It now names the corpus, the parser and the comparison, so the 16-file census is re-derivable from the file itself rather than from a PR description. Also records the result that ranks the two instruments: deleting the `{` counting inside `${...}` passed every pinned case AND the whole 4,739-file sweep, because the tree does not happen to write that shape. The case that holds it now was written from the mutation, not from the corpus. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DdCnBGcHeufjrq7drTD3wt --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent b5dbc59 commit 29b2f8c

1 file changed

Lines changed: 150 additions & 18 deletions

File tree

‎scripts/js-comment-mask.mjs‎

Lines changed: 150 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -43,14 +43,35 @@
4343
* findings against line numbers that no longer exist in the file it read, and
4444
* the drift is invisible until someone opens the file at the reported line.
4545
*
46-
* ## The direction it fails in
46+
* ## The direction it fails in -- a guarantee this module once claimed falsely
4747
*
48-
* A shape this scan gets wrong fails toward masking MORE than it should, which
49-
* costs recall (a real finding dropped) and cannot fabricate a lead. That is
50-
* the deliberate direction: a gate that over-masks under-reports loudly the
51-
* next time someone re-derives its scope, while a gate that under-masks
52-
* manufactures findings out of prose and burns a reader's afternoon proving
53-
* the sentence it quoted meant the opposite.
48+
* Over-masking is the direction to fail in: a gate that over-masks under-
49+
* reports loudly the next time someone re-derives its scope, while a gate that
50+
* under-masks manufactures findings out of prose and burns a reader's
51+
* afternoon proving the sentence it quoted meant the opposite.
52+
*
53+
* This header used to state that as a property -- "cannot fabricate a lead".
54+
* It was not one. Measured on 2026-08-21 (#10427): walk every
55+
* `.{ts,tsx,mts,cts,js,mjs,cjs,jsx}` file in the tree (minus `node_modules`,
56+
* `dist`, `.next`, `build`, `.turbo`, `coverage`), parse each with
57+
* `@typescript-eslint/parser` (`{ comment: true, range: true }`), and diff the
58+
* comment ranges it reports against this scan's `comment` array byte for byte.
59+
* Over 4,739 files, 16 disagreed -- 15 in the FABRICATES direction, up to
60+
* 10,252 comment bytes handed to a caller as live code in a single file. The
61+
* cause was the template scan above; the same sweep after the fix disagrees on
62+
* 0 files.
63+
*
64+
* The lesson is about the claim, not the bug. A failure DIRECTION is a
65+
* property of an implementation, not of an intention, and this one cannot be
66+
* read off the code -- it took an independent parser over the whole tree to
67+
* find out which way the module actually failed. So the honest statement is
68+
* the one that can be re-derived: the shapes below are pinned, the sweep just
69+
* described is the way to check the rest, and neither direction is promised by
70+
* construction. Re-run it after touching `scanSource` -- and note that the
71+
* sweep is the STRONGER instrument of the two. A mutation that deleted the
72+
* brace counting inside `${...}` passed every case below AND the whole sweep,
73+
* because the tree did not happen to write the shape; the case that now holds
74+
* it was written from the mutation, not from the corpus.
5475
*/
5576

5677
/** A character that can end an identifier -- i.e. a value, so `/` is division. */
@@ -77,9 +98,21 @@ const REGEX_AFTER_KEYWORD = new Set([
7798
*
7899
* The literal flag covers a literal's CONTENT, not its delimiters, so a caller
79100
* blanking comments still sees every string intact. Template interiors are
80-
* treated as literal through `${...}` as well: an interpolation's braces are
81-
* balanced by construction, so ignoring them is right for depth counting, and
82-
* a caller reading raw characters sees them either way.
101+
* reported as literal through `${...}` as well, so a caller reading raw
102+
* characters sees them either way.
103+
*
104+
* That is the FLAG. The SCAN of an interpolation is not the same question, and
105+
* conflating the two is the defect #10427 measured: `${...}` was walked as
106+
* plain literal text on the reasoning that its braces are balanced by
107+
* construction, which is true of the braces and false of everything else the
108+
* interpolation may hold. It is code, so it can hold a nested template
109+
* (``${xs.map((x) => `<${x}>`)}``, exactly how this tree formats a list of
110+
* names), a backtick inside a regex or a string (`packages/cli`'s `quoteIdent`
111+
* writes both), or a brace inside a string. Reading a nested opener as the
112+
* outer template's CLOSER flipped the parity of every backtick after it, and
113+
* the phantom span ran to the next backtick anywhere in the file. So the
114+
* interpolation is scanned as code here -- the same loop, with the same string,
115+
* regex and comment branches -- and its bytes are flagged literal at the end.
83116
*
84117
* @param {string} source
85118
* @returns {{ comment: Uint8Array, literal: Uint8Array }}
@@ -92,13 +125,58 @@ export function scanSource(source) {
92125
let prev = ''; // last significant CODE character
93126
let word = ''; // ...and the identifier it is the tail of, if any
94127

128+
// Open templates, innermost last. `braces` is the `{` depth inside the
129+
// frame's CURRENT interpolation: 0 means the scanner is in that template's
130+
// literal BODY, and > 0 means it is inside `${...}`, where the language says
131+
// the bytes are code. `start` is where that `${` began.
132+
const templates = [];
133+
// Closed `${...}` spans, flushed to `literal` after the pass -- see the
134+
// closing block of this function for why they are not flagged inline.
135+
const interpolations = [];
136+
95137
// A shebang is a comment to node; it is also the one line whose slashes are
96138
// neither division nor a regex.
97139
if (source.startsWith('#!')) {
98140
while (i < n && source[i] !== '\n') comment[i++] = 1;
99141
}
100142

101143
while (i < n) {
144+
const frame = templates.length ? templates[templates.length - 1] : null;
145+
146+
// A template's literal BODY: every byte is content until `${` opens an
147+
// interpolation, a backtick closes the template, or the file ends.
148+
if (frame && frame.braces === 0) {
149+
const ch = source[i];
150+
if (ch === '\\' && i + 1 < n) {
151+
literal[i] = 1;
152+
literal[++i] = 1;
153+
i++;
154+
continue;
155+
}
156+
if (ch === '`') {
157+
templates.pop();
158+
// A NESTED template's delimiters are the outer template's content.
159+
if (templates.length) literal[i] = 1;
160+
i++;
161+
prev = 'x'; // a value just ended
162+
word = '';
163+
continue;
164+
}
165+
if (ch === '$' && source[i + 1] === '{') {
166+
literal[i] = 1;
167+
literal[i + 1] = 1;
168+
frame.braces = 1;
169+
frame.start = i;
170+
i += 2;
171+
prev = ''; // `${/re/.test(x)}` -- the interpolation starts a fresh expression
172+
word = '';
173+
continue;
174+
}
175+
literal[i] = 1;
176+
i++;
177+
continue;
178+
}
179+
102180
const c = source[i];
103181
const next = source[i + 1];
104182

@@ -127,15 +205,12 @@ export function scanSource(source) {
127205
continue;
128206
}
129207
if (c === '`') {
208+
// A template OPENS here -- at the top level, or nested inside a `${...}`
209+
// this same loop is already reading as code. The body, and the matching
210+
// closer, are handled by the template-body branch above.
211+
if (templates.length) literal[i] = 1;
212+
templates.push({ braces: 0, start: -1 });
130213
i++;
131-
while (i < n && source[i] !== '`') {
132-
literal[i] = 1;
133-
if (source[i] === '\\' && i + 1 < n) literal[++i] = 1;
134-
i++;
135-
}
136-
if (i < n) i++;
137-
prev = 'x';
138-
word = '';
139214
continue;
140215
}
141216
if (c === '/' && !(IDENT_CHAR.test(prev) || prev === ')' || prev === ']')) {
@@ -166,12 +241,32 @@ export function scanSource(source) {
166241
word = '';
167242
continue; // re-read this `/` with prev cleared, as a regex
168243
}
244+
// Inside `${...}`: balance the braces, so the interpolation ends at ITS
245+
// `}` and not at one quoted inside it. A `{` or `}` in a string, regex or
246+
// comment never reaches here -- its own branch consumed it already.
247+
if (frame && frame.braces > 0) {
248+
if (c === '{') frame.braces++;
249+
else if (c === '}' && --frame.braces === 0) {
250+
interpolations.push([frame.start, i + 1]);
251+
frame.start = -1;
252+
}
253+
}
169254
if (!/\s/.test(c)) {
170255
prev = c;
171256
word = IDENT_CHAR.test(c) ? word + c : '';
172257
}
173258
i++;
174259
}
260+
261+
// `${...}` is CODE to the language, and the scan above reads it as code so a
262+
// backtick quoted inside it cannot flip the template's parity. The flag it
263+
// reports is the documented one: an interpolation's bytes are the enclosing
264+
// template's LITERAL content, marked in one pass at the end because the span
265+
// is only known once its closing brace is found. An unterminated one (EOF
266+
// inside `${`) still gets flagged, so truncated source cannot leak code
267+
// bytes into a caller's "this is not a literal" test.
268+
for (const frame of templates) if (frame.braces > 0 && frame.start >= 0) interpolations.push([frame.start, n]);
269+
for (const [start, end] of interpolations) for (let k = start; k < end; k++) literal[k] = 1;
175270
return { comment, literal };
176271
}
177272

@@ -268,6 +363,43 @@ export function selfTest() {
268363
['#!/usr/bin/env node', "/* err.code = 'GHOST' */", "err.code = 'REAL';"].join('\n')],
269364
['division after a paren, then a quote-bearing regex',
270365
['const r = (a) / b;', 'const q = /["' + BT + ']/g;', "/* err.code = 'GHOST' */", "err.code = 'REAL';"].join('\n')],
366+
// Templates whose interior puts a real backtick where a flat scan expects
367+
// the closer. Each pins BOTH sides: a genuine comment that must go, and
368+
// live code after it that must stay. A parity flip anywhere in the line
369+
// moves one of the two, so neither assertion can pass by accident.
370+
['nested template inside an interpolation',
371+
['const g = ' + BT + '${xs.map((x) => ' + BT + '\\' + BT + '${x}\\' + BT + BT + ").join(', ')} tail" + BT + ';',
372+
"/* err.code = 'GHOST' */", "err.code = 'REAL';"].join('\n')],
373+
['escaped backtick inside a template, without nesting',
374+
['const t = ' + BT + 'a \\' + BT + ' b' + BT + ';',
375+
"/* err.code = 'GHOST' */", "err.code = 'REAL';"].join('\n')],
376+
// Depth alone is NOT a defect shape: matched backticks pair off whatever a
377+
// scan believes about nesting, and this case stayed green under every
378+
// mutation of the fix that produced it (#10427). It is here for coverage of
379+
// the depth-2 path. The shapes that DO discriminate are the ones below,
380+
// where nesting meets an escape or a quoted backtick and the pairing breaks.
381+
['template nested inside a nested template',
382+
['const d = ' + BT + '${rows.map((r) => ' + BT + '${r.cells.map((c) => ' + BT + '<${c}>' + BT + ").join('')}" + BT + ").join('')}" + BT + ';',
383+
"/* err.code = 'GHOST' */", "err.code = 'REAL';"].join('\n')],
384+
['template spanning lines, carrying a nested template',
385+
['const m = ' + BT + 'head',
386+
' ${xs.map((x) => ' + BT + '\\' + BT + '${x}\\' + BT + BT + ").join(', ')}",
387+
'tail' + BT + ';', "/* err.code = 'GHOST' */", "err.code = 'REAL';"].join('\n')],
388+
// The interpolation is CODE, so a backtick or a brace QUOTED inside it is
389+
// neither a delimiter nor a nesting level. Both shapes are live in this
390+
// tree (`quoteIdent` in packages/cli writes the first one verbatim), and a
391+
// scan that only counts `${`/`}` desyncs on both.
392+
['a backtick inside a regex inside an interpolation',
393+
['const q = ' + BT + '\\' + BT + '${name.replace(/' + BT + "/g, '" + BT + BT + "')}\\" + BT + BT + ';',
394+
"/* err.code = 'GHOST' */", "err.code = 'REAL';"].join('\n')],
395+
['an object literal, then a nested template, in one interpolation',
396+
['const o = ' + BT + '${fmt({ a: 1 }, ' + BT + '\\' + BT + BT + ')} tail' + BT + ';',
397+
"/* err.code = 'GHOST' */", "err.code = 'REAL';"].join('\n')],
398+
['a brace and a backtick quoted inside an interpolation',
399+
['const b = ' + BT + "${fmt({ a: 1 }, '" + BT + "')} tail" + BT + ';',
400+
"/* err.code = 'GHOST' */", "err.code = 'REAL';"].join('\n')],
401+
['a real comment inside an interpolation',
402+
['const c = ' + BT + "${x /* err.code = 'GHOST' */} tail" + BT + ';', "err.code = 'REAL';"].join('\n')],
271403
['a genuine docblock is still removed',
272404
['/** Retired: err.code = ' + "'GHOST'" + ' must never come back. */', "err.code = 'REAL';"].join('\n')],
273405
['a genuine line comment is still removed',

0 commit comments

Comments
 (0)