Skip to content

Commit 1793fb6

Browse files
claude[bot]claude
andauthored
fix(gate): read CommonMark autolinks in check:published-readme-links (#10764)
`extractLinks` matched only `[text](dest)` and `[label]: dest`, so a bare URI in angle brackets -- the CommonMark autolink, rendered as a live link by both GitHub and npm -- was read by none of the gate's four assertions. 13 docs-site links across 6 published READMEs sat in that hole, every one of them in the "Docs" / "API Reference" footer. All 13 resolve today, so nothing was broken through it; what was missing was the population the gate's own header claims. Adds AUTOLINK as a third recognizer, transcribed from the CommonMark scheme production. Autolinks are matched last and filtered by SPAN, because a pointy-bracket destination `[d](<https://x/y>)` that contains no space is also a well-formed autolink body -- a span-blind recognizer would count it twice and inflate the printed census by one per occurrence. Claude-Session: https://claude.ai/code/session_01DdCnBGcHeufjrq7drTD3wt Co-authored-by: Claude <noreply@anthropic.com>
1 parent f06d8ed commit 1793fb6

1 file changed

Lines changed: 201 additions & 8 deletions

File tree

‎scripts/check-published-readme-links.mjs‎

Lines changed: 201 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,44 @@
128128
// is *followable* -- it just lands on raw MDX source), and whether an external
129129
// URL is alive (that is lychee's job and it needs the network).
130130
//
131+
// ## The three link shapes read, and the one that only looks like a fourth
132+
//
133+
// The header above claims "every outbound link in a PUBLISHED markdown file",
134+
// so the extractor owes that population every shape a reader can actually
135+
// follow. Three of them:
136+
//
137+
// * INLINE_LINK `[text](dest)`, `![alt](dest)`, title optional
138+
// * REFERENCE_DEFINITION `[label]: dest` at the head of a line
139+
// * AUTOLINK a bare absolute URI wrapped in angle brackets
140+
//
141+
// The autolink was missing for this gate's first three days, and the hole was
142+
// not academic: 13 docs-site links across 6 published READMEs sat in it,
143+
// every one of them in the "Docs" / "API Reference" footer -- the most
144+
// prominent link position these files have. All 13 resolved, so nothing was
145+
// broken through it; what was broken was the population. A 404 written in that
146+
// form was read by NONE of the four assertions and shipped to npm green.
147+
//
148+
// The shape that only looks like a fourth is the pointy-bracket DESTINATION,
149+
// `[text](<dest>)` -- angle brackets INSIDE an inline link's parentheses, which
150+
// is how CommonMark spells a destination containing a space. That is not an
151+
// autolink and never was; `INLINE_LINK`'s own `(<...>|...)` alternative has
152+
// always read it. The two are distinguished POSITIONALLY here: `extractLinks`
153+
// records the span each link construct consumed and refuses an autolink match
154+
// that starts inside one. Without that, `[d](<https://objectstack.ai/docs/x>)`
155+
// -- a pointy destination that happens to contain no space, and so is a
156+
// well-formed autolink body too -- would be counted twice, and the census this
157+
// gate prints would over-report by one per occurrence. The `--self-test` pins
158+
// the count, not just the presence, for exactly that reason.
159+
//
160+
// Deliberately NOT matched: CommonMark's EMAIL autolink, the address-shaped
161+
// `<foo@example.com>` with no scheme. It renders as a `mailto:` link, so it is
162+
// a link -- but it can never be a docs-site URL, so every assertion here would
163+
// classify it `external` and read no further. Matching it would widen the
164+
// recognizer and change no verdict. The scheme'd `<mailto:foo@example.com>`
165+
// form IS matched, because it costs nothing to fold into the URI pattern, and
166+
// it too classifies `external`; a pin holds that so a later reader does not
167+
// have to re-derive which of the two is in.
168+
//
131169
// ## Why fenced blocks and code spans are stripped first
132170
//
133171
// A markdown link inside a code fence is example text, not a link. This gate
@@ -196,6 +234,26 @@ const INLINE_LINK = /!?\[[^\]]*]\(\s*(<[^>\n]*>|[^)\s]*?)(?:\s+(?:"[^"]*"|'[^']*
196234
/** `[label]: dest` at the start of a line -- the reference-definition form. */
197235
const REFERENCE_DEFINITION = /^ {0,3}\[[^\]]+]:\s*(<[^>\n]*>|\S+)/;
198236

237+
/**
238+
* CommonMark's URI autolink: a bare absolute URI in angle brackets, rendered as
239+
* a live link by GitHub and by npm alike.
240+
*
241+
* Transcribed from the CommonMark scheme production rather than approximated: 2
242+
* to 32 characters, opening with an ASCII letter, continuing with letters,
243+
* digits, `+`, `.` or `-`; then a colon; then any run of characters that are
244+
* not ASCII control, space, or either angle bracket. The final class is what
245+
* keeps the pattern honest about the shapes it must NOT claim -- `<not a url>`
246+
* carries a space and no scheme, and an HTML tag like `<br>` has no colon, so
247+
* neither can match. Both are pinned.
248+
*
249+
* The absent `g`-flag twin of the pointy-destination question above: this one
250+
* IS global, and `extractLinks` filters its matches by span rather than by
251+
* pattern, because a pointy destination and an autolink are genuinely the same
252+
* characters in different positions -- no regex reading one line at a time can
253+
* separate them, and a lookbehind that tried would be guessing at nesting.
254+
*/
255+
const AUTOLINK = /<([A-Za-z][A-Za-z0-9+.-]{1,31}:[^<>\x00-\x20]*)>/g;
256+
199257
/**
200258
* Every outbound link destination in one markdown document, with the 1-based
201259
* line it was written on.
@@ -204,6 +262,13 @@ const REFERENCE_DEFINITION = /^ {0,3}\[[^\]]+]:\s*(<[^>\n]*>|\S+)/;
204262
* discrimination mechanism, and both strippers preserve newlines so the line
205263
* number points at what a human will look at.
206264
*
265+
* Autolinks are matched LAST and filtered by span, because the two bracketed
266+
* forms are the same characters in different positions: the angle brackets of
267+
* `[d](<https://x/y>)` belong to the inline link that already claimed them, and
268+
* counting them again would inflate the census by one per occurrence. So each
269+
* link construct records the range it consumed, and an autolink starting inside
270+
* one is dropped rather than re-read.
271+
*
207272
* @param {string} markdown
208273
* @returns {{ line: number, dest: string }[]}
209274
*/
@@ -212,11 +277,24 @@ export function extractLinks(markdown) {
212277
const found = [];
213278
prose.split('\n').forEach((line, idx) => {
214279
const dests = [];
280+
/** @type {[number, number][]} half-open spans already claimed on this line */
281+
const claimed = [];
215282
INLINE_LINK.lastIndex = 0;
216283
let m;
217-
while ((m = INLINE_LINK.exec(line)) !== null) dests.push(m[1]);
284+
while ((m = INLINE_LINK.exec(line)) !== null) {
285+
dests.push(m[1]);
286+
claimed.push([m.index, m.index + m[0].length]);
287+
}
218288
const ref = REFERENCE_DEFINITION.exec(line);
219-
if (ref) dests.push(ref[1]);
289+
if (ref) {
290+
dests.push(ref[1]);
291+
claimed.push([ref.index, ref.index + ref[0].length]);
292+
}
293+
AUTOLINK.lastIndex = 0;
294+
while ((m = AUTOLINK.exec(line)) !== null) {
295+
if (claimed.some(([start, end]) => m.index >= start && m.index < end)) continue;
296+
dests.push(m[1]);
297+
}
220298
for (const raw of dests) {
221299
// CommonMark's pointy-bracket destination: `[x](<a b>)`.
222300
const dest = raw.startsWith('<') && raw.endsWith('>') ? raw.slice(1, -1) : raw;
@@ -541,15 +619,26 @@ function selfTest() {
541619
'',
542620
'```ts',
543621
'const s = "[fenced](/content/docs/never.mdx)";',
622+
'const t = "<https://objectstack.ai/docs/never-fenced>";',
544623
'```',
545624
'',
546625
'A code span `[spanned](/content/docs/never.mdx)` is not a link.',
626+
'A code span `<https://objectstack.ai/docs/never-spanned>` is not one either.',
547627
'',
548628
'[ref]: /content/docs/refdef.mdx',
549629
'',
550630
'Pointy [b](</content/docs/spaced page.mdx>) form.',
551631
'Titled [c](/content/docs/titled.mdx "A title") form.',
552632
'![img](/content/docs/pic.png)',
633+
'',
634+
'Bare autolink <https://objectstack.ai/docs/autolinked> in prose.',
635+
'Mail <mailto:hi@objectstack.ai> too.',
636+
// The double-count trap: a pointy DESTINATION whose body is also a
637+
// well-formed autolink, because it happens to contain no space.
638+
'Pointy [d](<https://objectstack.ai/docs/pointy-no-space>) form.',
639+
// Angle brackets that are not links at all.
640+
'Not <not a url> and not <br> and not <a href="https://x.ai">tagged</a>.',
641+
'And not the address-shaped <foo@example.com> either.',
553642
].join('\n');
554643
const links = extractLinks(doc);
555644
const dests = links.map((l) => l.dest);
@@ -560,8 +649,48 @@ function selfTest() {
560649
ok('extractLinks drops a link title', dests.includes('/content/docs/titled.mdx'));
561650
ok('extractLinks reads an image destination', dests.includes('/content/docs/pic.png'));
562651
ok('extractLinks reports the real line number', links.find((l) => l.dest === '/content/docs/x.mdx')?.line === 1);
563-
// Five, not seven: the fenced one and the code-spanned one are the discrimination.
564-
ok('extractLinks finds exactly the five PROSE destinations', links.length === 5);
652+
653+
// ---- extractLinks: the CommonMark autolink ----------------------------
654+
// The shape this gate could not see for its first three days. Every one of
655+
// the 13 it was blind to sat in a published README's "Docs" footer, so the
656+
// ACCEPT pins below are the population and the REJECT pins are the price:
657+
// widening a recognizer is only safe if what it newly claims is exactly the
658+
// links and nothing else that wears angle brackets.
659+
ok('extractLinks reads a bare autolink', dests.includes('https://objectstack.ai/docs/autolinked'));
660+
ok('extractLinks reads a mailto autolink', dests.includes('mailto:hi@objectstack.ai'));
661+
ok(
662+
'extractLinks reports the real line number for an autolink',
663+
links.find((l) => l.dest === 'https://objectstack.ai/docs/autolinked')?.line === 17,
664+
);
665+
ok(
666+
'extractLinks SKIPS an autolink inside a fenced block',
667+
!dests.includes('https://objectstack.ai/docs/never-fenced'),
668+
);
669+
ok(
670+
'extractLinks SKIPS an autolink inside a code span',
671+
!dests.includes('https://objectstack.ai/docs/never-spanned'),
672+
);
673+
// The reject side, one pin per shape that wears the brackets without being a
674+
// link: no scheme, no colon, a tag with attributes, and the address-shaped
675+
// EMAIL autolink this recognizer deliberately leaves out (see the header).
676+
ok('extractLinks does NOT match angle-bracketed prose', !dests.some((d) => d.includes('not a url')));
677+
ok('extractLinks does NOT match an HTML tag', !dests.some((d) => d === 'br' || d.startsWith('a href')));
678+
ok('extractLinks does NOT match a bare email autolink', !dests.includes('foo@example.com'));
679+
// The double-count guard, asserted as a COUNT rather than a presence: a
680+
// pointy-bracket destination with no space is a valid autolink body too, so
681+
// a span-blind recognizer reads it twice and every census this gate prints
682+
// inflates by one per occurrence. Presence alone cannot see that.
683+
ok(
684+
'extractLinks counts a pointy-bracket destination exactly ONCE',
685+
dests.filter((d) => d === 'https://objectstack.ai/docs/pointy-no-space').length === 1,
686+
);
687+
ok(
688+
'extractLinks still unwraps the pointy destination it always read',
689+
dests.includes('/content/docs/spaced page.mdx'),
690+
);
691+
// Eight, not thirteen: two fenced, two code-spanned, and the three
692+
// bracket-wearing non-links are the discrimination.
693+
ok('extractLinks finds exactly the eight PROSE destinations', links.length === 8);
565694

566695
// ---- classify: every bucket, both directions --------------------------
567696
ok('classify: root-relative', classify('/content/docs/a.mdx') === 'root-relative');
@@ -753,6 +882,66 @@ function selfTest() {
753882
);
754883
}
755884

885+
// ---- the autolink reaches the assertions, not just the extractor ------
886+
// Extraction is necessary and not sufficient: the defect was that all four
887+
// assertions were SILENT on this shape, so each is re-observed FAILING with
888+
// the destination written as an autolink. Pinning extraction alone would
889+
// leave the gate free to read the link and then check nothing about it.
890+
// A1 is the one assertion this shape cannot reach, and that is CommonMark's
891+
// doing rather than an omission here: an autolink body must be an ABSOLUTE
892+
// URI, so a root-relative path in angle brackets is not a link on any of the
893+
// three surfaces -- GitHub and npm render `</content/docs/x.mdx>` as literal
894+
// text. Pinned in the direction that is actually true, so a later author does
895+
// not read the missing A1 pin as a hole and "fix" it by dropping the scheme
896+
// requirement -- which would start claiming every HTML tag in the tree.
897+
const auto1 = runDoc('Docs: </content/docs/automation/flows.mdx>');
898+
ok(
899+
'a root-relative path in angle brackets is NOT an autolink (no scheme)',
900+
auto1.stats.links === 0 && auto1.findings.length === 0,
901+
);
902+
const auto2 = runDoc('Docs: <https://docs.objectstack.ai/docs/automation/flows>');
903+
ok(
904+
'A2 FAILS on an alias origin written as an autolink',
905+
auto2.findings.length === 1 && auto2.findings[0].kind === 'non-canonical-host',
906+
);
907+
ok('A2 counts the autolink it read', auto2.stats['non-canonical'] === 1);
908+
const auto3 = runDoc('Docs: <https://objectstack.ai/docs/automation/no-such-page>');
909+
ok(
910+
'A3 FAILS on a dead docs page written as an autolink',
911+
auto3.findings.length === 1 && auto3.findings[0].kind === 'dead-page',
912+
);
913+
const auto3ok = runDoc('Docs: <https://objectstack.ai/docs/automation/flows>');
914+
ok(
915+
'A3 SILENT on a resolving autolink, and COUNTS it resolved',
916+
auto3ok.findings.length === 0 && auto3ok.stats.resolved === 1,
917+
);
918+
const auto4 = runDoc(
919+
'Docs: <https://objectstack.ai/docs/automation/flows#no-such-heading-anywhere>',
920+
);
921+
ok(
922+
'A4 FAILS on a dead anchor written as an autolink',
923+
auto4.stats.fragments === 1 && auto4.findings.some((f) => f.kind === 'dead-anchor'),
924+
);
925+
// The card's own positive control, kept as a pin: this exact input returned
926+
// [] from the shipped extractor, which is what made the hole measurable.
927+
const autoCard = runDoc('Docs: <https://objectstack.ai/docs/no-such-page-anywhere>');
928+
ok(
929+
'the reported 404-in-an-autolink is now a finding rather than silence',
930+
autoCard.stats.links === 1 && autoCard.findings.some((f) => f.kind === 'dead-page'),
931+
);
932+
// A mailto autolink is a LINK (it is counted) and is out of scope BY NAME
933+
// (it is classified external and yields nothing) -- both halves, because
934+
// "no findings" alone is also what a recognizer that never saw it produces.
935+
const autoMail = runDoc('Mail: <mailto:hi@objectstack.ai>');
936+
ok(
937+
'a mailto autolink is counted as a link and classified external',
938+
autoMail.stats.links === 1 && autoMail.stats.external === 1 && autoMail.findings.length === 0,
939+
);
940+
// The non-links, at document level: an angle bracket in prose must not
941+
// manufacture a link, or the empty-scan guard above stops meaning anything.
942+
const autoNone = runDoc('Text <not a url>, a tag <br>, and <foo@example.com>.');
943+
ok('bracket-wearing non-links yield no links at all', autoNone.stats.links === 0);
944+
756945
// ---- the empty-scan guard is real, not decorative ---------------------
757946
const empty = runDoc('No links here at all.\n\n```ts\nconst x = 1;\n```\n');
758947
ok('a document with no links yields no findings and no links', empty.stats.links === 0 && empty.findings.length === 0);
@@ -764,10 +953,14 @@ function selfTest() {
764953
}
765954
console.log(
766955
'✓ check:published-readme-links --self-test — extraction discrimination (fence, code span,\n'
767-
+ ' ref-def, pointy brackets, titles), all seven classify buckets, the remedy builder and its\n'
768-
+ ' refusal, the canonicaliser both ways, the pageCandidates directory/index subtlety, and\n'
769-
+ ' all four assertions observed both FAILING and SILENT — including the host split itself:\n'
770-
+ ' an alias origin is a FINDING, and is still resolved and still anchor-checked.',
956+
+ ' ref-def, pointy brackets, titles, AUTOLINKS), all seven classify buckets, the remedy\n'
957+
+ ' builder and its refusal, the canonicaliser both ways, the pageCandidates directory/index\n'
958+
+ ' subtlety, and all four assertions observed both FAILING and SILENT — including the host\n'
959+
+ ' split itself (an alias origin is a FINDING, and is still resolved and still\n'
960+
+ ' anchor-checked) and the autolink shape (assertions 2, 3 and 4 re-observed FAILING on it;\n'
961+
+ ' A1 pinned UNREACHABLE through it, since an autolink body must be an absolute URI; a\n'
962+
+ ' pointy-bracket destination counted exactly ONCE; and angle-bracketed non-links —\n'
963+
+ ' tags, prose, bare addresses — claimed by nothing).',
771964
);
772965
}
773966

0 commit comments

Comments
 (0)