Repository navigation
Speed and coverage - #2
Merged
Merged
Conversation
sieve renders a page the way a browser does and republishes it as a
structured artifact an agent can read cheaply. On linear.app that is
472,961 tokens of served HTML reduced to 1,732.
Architecture, in the order a run touches it:
fetch -> score -> [render] -> [sweep] -> [recover] -> graph -> emit
Escalation (internal/escalate, internal/static)
Most pages are answered by a plain HTTP GET and static extraction in
under a second; the browser is used only where a cheap fetch comes back
thin. Thresholds are pinned constants, and a domain that has ever
escalated stays escalated, so a page near the line is not judged
differently on different days. The tier and its reasoning are recorded
in every artifact.
Capture and sweep (internal/capture, internal/render)
One top-down walk carrying inherited state downward rather than walking
up per node. Split-text reassembly puts per-character headlines back
together. Reading order is a recursive XY-cut that competes a horizontal
against a vertical split, so two columns of prose are not read across.
Latent tier (Move B)
Hidden content is quarantined, not discarded: its own top-level key, its
own retrieval tool, a trust marker in every format, excluded from the
headline token count, labelled with the control that would reveal it.
A test asserts on every format that it never reaches a default payload.
Self-audit (Move C)
Graph retention, cross-method order agreement, and heading-cluster
separation. Named narrowly on purpose: retention measures the graph
stage, not the sweep, so it cannot be read as coverage. Confidence ships
as coarse buckets because a wrong number is worse than no number.
Canvas recovery (internal/canvas, internal/corroborate)
Five attacks, cheapest and most exact first. Anything read off pixels is
cross-checked against the text the site shipped; unconfirmed recoveries
are excluded from every default payload. The corroboration index is a
membership oracle with no way to enumerate it, which enforces the rule
that intercepted payloads may only confirm, never add. Vision is off by
default, so an artifact structurally cannot contain invented text.
Untrusted content
Rendering-grounded capture closes the DOM-text channel completely --
display:none, zero-opacity, and text matching its own background never
enter the content tier, at tier 0 as well as tier 2. Channels that
bypass visibility by design are capped, whitelisted and marked instead.
Normalisation is versioned and part of the content hash.
Determinism
Semantic content hash over the normalised graph. Locale, timezone and
viewport pinned; A/B platforms blocked. Full trace header, and snapshots
that replay the whole graph stage offline.
Four bugs the tooling found and fixed along the way:
- --in-process-gpu with SwiftShader starves frame production on every
tab after the first, which starves IntersectionObserver and silently
turns a scroll-reveal page into an empty artifact. `sieve doctor`
probes for it.
- A geometry-only settle signature declares a page settled during an
opacity fade, so text was captured mid-reveal and filed as low
confidence. Opacity and pending transitions are now in the signature.
- EverVisible looked like a visibility signal and was really a record of
where the sweep stopped; it made confidence flip between runs.
- The static tier could not resolve class-based hiding, so hidden text
leaked at tier 0 while tier 2 caught it. A bounded CSS scan closes it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five design-heavy sites took six minutes between them and still lost content. The two costs turned out to be unrelated, and both were sieve's rather than the web's. Speed: the checkpoint loop was driven from Go, so every stop cost four CDP round trips and a full re-serialisation of the DOM, and the settle wait could only ever be a fixed timeout -- the process deciding when to stop waiting sat on the far side of a socket from the frames it was waiting for. The loop now runs in the page and reports once. Settle is a count of animation frames, each checkpoint ships only what it added, and the sweep travels to where it has seen unread text instead of stepping past it. Around that: capture.js is installed at document start rather than compiled into a busy main thread after load; navigation waits for the document, not the last image; the browser warms alongside tier 0 and navigates during it on domains the memory already knows; robots.txt is bounded and cached to disk; canvases are only rasterised when OCR or vision will read the pixels; script bodies are only pulled when a canvas exists to corroborate. Every sub-budget scales from --timeout, which now covers the whole command including Chromium's shutdown. Coverage: pear.no serves four thousand characters of copy and its entire FAQ, and tier 0 called it "a shell" with 54 characters. Its FAQPage script sits inside the aria-hidden section it describes, so the walk turned back at the door -- structured data is now harvested from hidden subtrees, because being inside a hidden div is not a property JSON-LD can have. Questions and answers get their own array, and the legal identifier joins the whitelist. The deeper finding is that the two tiers held different halves of that page: the served bytes have the terms, the selectivity and the application, and the render has the hero and the pillars, which are not in the served bytes at all. sieve fetched both every run and discarded one. It now folds served-hidden text in, but only on proof -- three sentence-length fragments of that same hidden text independently witnessed, on screen during the render or published by the site as its own structured data. A page that hides text and never shows any of it proves nothing and gets nothing adopted, which is the injection case. display:none never qualifies. Everything adopted is speculative, flagged, and named in the audit. Two accuracy fixes alongside: fragments are welded together only on positive evidence that they are contiguous on one line, because a gap is not wide when it is negative and animated text overlaps constantly; and a run the page declares it will reveal, by its own transition or animation, is kept and labelled rather than dropped. Also fixed: the CLI built two distillers and deferred Close on the one that never did any work, so every error path leaked a Chromium -- that is the hundred and sixty stray processes, and they were slow enough to look like extraction bugs. A failed fetch no longer pins a domain to a browser for good on the strength of zero bytes. An entry screen is declared as a gap instead of producing a thin artifact that reads like an empty site. Temporary DNS failures are retried rather than reported as unreachable. pear.no: 3m43s to 1.1s at tier 0, with more of the site than the three-minute run produced. yomi 2.9s to 1.1s, organimo 1m52s to 4.9s, igloo 11.2s to 6.6s, all inside ten seconds.
getaddrinfo on Windows reports live domains as "no such host" under load -- definitively, with IsNotFound set, so nothing about the answer suggests retrying. It happens often enough that four different real sites hit it during this project's own testing, and it ends the run at its first stage with a message telling the operator to go and check a name that was never wrong. Retrying the same mechanism does not help, because it is not being slow, it is being confident. So a negative is now confirmed against Go's own DNS client, which talks to the configured servers directly and fails independently. Two mechanisms agreeing that a name does not resolve is an answer; one is a rumour. Costs nothing on the happy path: it runs only when the run is about to be abandoned. Weakens nothing: whatever address the second resolver returns is checked against the private ranges exactly as any other.
…itted Two faults, one artifact. A portfolio came back with thirty "content blocks" totalling five hundred and eighteen characters -- almost all of it image alt text -- while the audit reported 0% retention and 120 runs, 5,335 characters, dropped as never having reached visible opacity. Above that sat a headline claiming a 99.5% token saving. The reveal detection only understood reveals a stylesheet declares. A site that fades its text in from JavaScript writes a new opacity every frame and declares nothing, so the computed style has no transition and no animation to find, and every run the sweep did not reach in time was dropped as hidden. The evidence that works there is the reveal itself: a run seen below the visible-opacity floor at one checkpoint and above it at another was animated into view while sieve was watching. Merge now keeps the lowest opacity alongside the highest, and a page with several such runs is a page that brings its text in by animation -- so a run still below the floor is one the sweep ran out of budget before reaching, not one the page is concealing. Those are kept, marked declared, and flagged. A page that never reveals anything keeps the strict rule exactly as it was, which is the injection case and is tested. The token figure counted the prose blocks and nothing else, which is the right measure of the content and the wrong measure of the cost. The emitted document also carries the description, the links, the buttons, the media descriptions, the questions and answers and the whitelisted facts, and a reader pays for all of it. On the artifact above that was 156 tokens reported against an eight-and-a-half kilobyte file an agent would actually read -- around 2,200. Counting the emitted payload stops the headline flattering a thin artifact, and stops it understating a rich one.
Keeping runs on the evidence that a page animates its text has a predictable side effect: sites that animate text very often ship two copies of it. A masked heading, a marquee, a crossfade between an old value and a new one -- one copy is on screen and its twin waits at opacity zero. The visible copy was always captured; the hidden one used to be dropped for being invisible and is now kept for being revealable, so both reach the artifact. A portfolio came back with a hundred and six duplicates in two hundred and twenty blocks -- "HELLO, I AM", "LASISI QUADRI", "HELLO, I AM", "LASISI QUADRI" -- and a median block length of ten characters. A declared run that says nothing new is now dropped. Precedence is what makes that safe rather than arbitrary: a run sieve watched always beats one it merely believes in, and between two unobserved copies the first in reading order wins. Nothing is lost, because the words stay in the artifact exactly once.
A page was swept fifty-seven times, at sixty-five milliseconds a stop, and not one character was ever observed above the visible-opacity threshold. Every run was dropped, the audit reported nought per cent retention, and the same page read correctly at eighteen checkpoints a few minutes earlier. The sweep was outrunning the animation. The shortcut that caused it is right in its own case: a page with a permanent animation never reports itself still, so waiting the full settle at every checkpoint spends the budget on an answer that will not come, and cutting the wait buys stops for free. But "never settles" and "never reveals" are different conditions, and on a page that fades every run in they arrive together -- so the shortcut fires, the wait collapses to its floor, and the sweep captures faster than the page can animate. When text is being captured and none of it has ever been legible, the wait is now raised rather than cut. Bounded by the remaining budget, so a page that genuinely has nothing to reveal costs a few checkpoints to establish that instead of the whole sweep. The condition is also stated in the artifact, because "nothing here was ever seen on screen" is something a reader should be told outright rather than left to infer from a retention figure.
Two faults found in a spot check across six sites. techsibiti.com returned thirteen nodes from a thirty-seven kilobyte page: a loading screen and the search-engine fallback list, with the checkpoint cap reached before anything had rendered. One capture is the right amount of work for a client-rendered page whose content is all present once JavaScript has run, and the wrong amount when JavaScript has not finished running. The tier decides how much of a document to walk, not whether to wait for the document to exist -- so when the readiness wait reports the page still loading, the render tier now gets a handful of checkpoints instead of one, which is enough for the emptiness rule in the page to look again. Four of the six carried duplicated content, up to thirty-three repeats each. Carousels, marquees and infinite-scroll tracks keep two copies of their items in the DOM so the loop has somewhere to go; both are really on screen at different moments, so neither is hidden and neither can be dropped for invisibility. The artifact simply read as though the page said everything twice. An immediate echo of the preceding run is now removed. Adjacency is what makes that safe: text that recurs across a page -- a repeated call to action, a price in a table and again in a summary -- is separated by other content and is left alone.
The budget for reading a page started at navigation, so a site's own loading time was charged to the extraction. A page with an intro film or a preloader -- exactly the class of page this tool exists for -- handed the sweep whatever was left of ten seconds, and what was left was often a single capture of a loading screen. techsibiti spent five seconds booting and was read in the remaining fraction; pear.no spent three on its film and starved the sweep behind it. Waiting is not work sieve is doing. The wait now has its own allowance, LoadBudget, and it is not deducted from anything: sieve waits for the page to arrive and stop moving, and only then does the extraction budget start, in full, measured from the moment there was something to extract. --timeout means time spent reading; --load-timeout bounds the patience. A site that never finishes loading still ends. Two things fixed alongside, both found in the same spot check. The render tier took exactly one capture, which is right for a page whose content is all present once JavaScript has run and a coin toss otherwise: techsibiti returned thirteen nodes on one run and a hundred and fifty-three on the next, from an unchanged page, on nothing but whether that capture landed before or after the first paint. It gets four looks now, and the stability rule still ends it in two when the page really was ready. And the repeat collapser only compared a block with the one before it, which never fires on a cloned track: HTML, CSS, JavaScript, React, HTML, CSS, JavaScript, React contains no block immediately followed by itself, so every item survived twice. It matches sequences now, longest first.
Both were charged to one clock, so a page that loads promptly and then animates forever spent the entire loading allowance proving it would never be still: suzanne3d.com took nineteen and a half seconds to reach a page that had been ready in two, and the run took thirty-six seconds in total. They are different questions with different owners. Waiting for a document to arrive is the site's time and can legitimately be long; waiting for it to stop moving afterwards is sieve's, and should not be. The settle wait now takes both budgets and runs its stillness clock from the moment the document is complete. The render tier was also still pinning the checkpoint cap to one, so lifting that cap in the sweep had no effect -- the value had already been overwritten before it got there, and techsibiti.com stopped one checkpoint into a document it had seven unspent seconds to finish. Requiring a single stable checkpoint ends the loop just as quickly on a page that really is one screen, and lets it continue on a page that is not. And monospace stopped being proof of code. It is a typeface, and on a technically-styled site it is the whole interface: suzanne3d emitted its feature labels and its copyright line as code blocks. An author who means code says so with a tag, which was already honoured; below that the typeface now has to be corroborated by the text carrying the syntax of code rather than the shape of a sentence. techsibiti 30.3s to 12.1s, 76 to 106 blocks, and it reaches its footer now. suzanne3d 36.7s to 18.9s with no code blocks left. quadricodes 14.1s to 12.0s.
A hundred-site sweep, mixed from plain documentation to WebGL studios. Twenty-seven runs came back clean; the rest sorted into a handful of faults, two of which accounted for most of them. Forty-four sites looked full of duplicates and twenty-five looked fragmented, and it was one bug wearing two hats: navigation labels and bare punctuation were reaching the content tier. docs.python.org contributed blocks reading "«", "|" and "»"; go.dev emitted "learn more" four times and "Tour", "Docs" and "Blog" once each as prose, when every one is a link whose label and destination are already recorded a few fields away. danluu.com produced three hundred and ninety-nine blocks with a median length of seven characters, most of them the date column of an index. A run with no letter and no digit in it is a separator, and a short run that is the whole of a link's text is a menu item. Neither is content. A paragraph that contains a link is untouched, because its block is the prose rather than the link. Tier 0's fetch was priced against the budget for reading a page, which is the wrong comparison: fetching is a page load, the same bytes over the same network the browser would use. Two-fifths of the reading budget is a few seconds, and a large site behind a slow first byte exceeds that routinely. The cost is not a slow tier 0 but no tier 0 at all -- a timed-out fetch is recorded as a failure, which forbids the static fallback afterwards, so a page whose HTML was perfectly readable ends up resting on a render that may also run out of time. react.dev came back with nothing that way and returns a hundred and eighty-one blocks now. The budget follows the load allowance instead. Across the corpus: clean runs 27 to 52, duplicates 44 to 30, fragmented 25 to 12, runs over twenty-five seconds 26 to 7, slowest 69s to 32s, hard errors 2 to 0, and leaked browsers 57 to 0.
try a page tier 0 was refused A card or list layout repeats its furniture once per item, and none of it is a link, so the navigation rule never saw it. npr.org emitted "hide caption" and "toggle caption" fifty-eight times each, esa.int "views" and "likes" twenty-seven, airbnb.com "guest favorite" twenty-two. The first occurrence is kept, so nothing leaves the page's vocabulary; what goes is the repetition. Three is the threshold because three is what the corpus shows -- every duplicate finding that survived the first pass sat at exactly three. Single characters are never blocks. resend.com produced blocks reading "t", "b" and "s", which are fragments the reassembly could not place and which no artifact is improved by carrying. A response that is not HTML now hands the page to the browser instead of ending the run. patagonia.com answers this client with a bare 404 and an empty body while serving the site to a browser perfectly well; the fact sieve has is that the cheap path returned no document, and it already knows what to do with that. A genuine non-HTML resource yields nothing in the browser either, and the artifact says so. And the token line stops claiming a saving it cannot support. A page that serves a shell gives a denominator of almost nothing, so the percentage was either flattering or, when the fetch had failed outright, an apparent increase that read like a fault. It now says the served page carried no readable text and leaves the number alone. Across the corpus after these: clean 27 to 61, duplicates 44 to 13, fragmented 25 to 10, leaked browsers 57 to 0.
what went wrong when neither can A second hundred-site corpus -- news in four scripts, commerce, forums, government, agencies -- found four runs returned to the caller as outright failures. Every one was a condition of the site rather than of sieve: internetarchive.org presents a certificate issued for archive.org, north-kingdom.com uses an authority the browser does not recognise, muji.com resets the connection, uniqlo.com breaks the HTTP/2 framing. Where tier 0 has already fetched the page, discarding it because the browser could not repeat the trick is the same mistake as letting a failed fetch end a run: the artifact falls back to the served HTML and says the browser never loaded, so anything rendered with JavaScript is declared missing rather than silently absent. Where both paths fail the run still ends, because there is nothing to build from -- but it now ends legibly. "net::ERR_CERT_COMMON_NAME_INVALID" sends somebody looking for a bug in this program; "the site's TLS certificate is issued for a different hostname" sends them to the site. Ten of Chromium's network codes are translated, with the original kept in parentheses. Also from this corpus: a response that is not HTML now hands the page to the browser rather than ending the run. patagonia.com answers this client with a bare 404 and an empty body while serving browsers normally, and the fact sieve has -- the cheap path returned no document -- is one it already knows how to act on.
put a space inside CJK text Two hundred sites across both corpora. Seventeen came back empty, and the largest identifiable group was not empty at all. moma.org, ox.ac.uk and vanguardngr.com answer the Go client with 403 on their apex host while serving the identical user agent 200 on their www host -- reproducibly, same headers, same TLS settings, same everything this program controls. Freezing at tier 0 there emitted the CDN's error page as the artifact: zero blocks for sites that were refusing nothing. It also honoured nothing, because no preference had been expressed that a second client would violate. The browser now gets one attempt at a page tier 0 was refused. It is not a disguise: same identifying user agent and contact URL, same robots.txt, the tool's other ordinary client. If it is refused too, that is two independent refusals and the run stops and says so. Nothing retries a refusal, spoofs a fingerprint, or asks twice with the same client. Either outcome is written into the artifact. moma.org goes from nothing to a hundred and four blocks, vanguardngr.com to two hundred and four. And the reassembly stopped putting spaces inside Japanese and Chinese. The rule that errs towards a space is right for Latin, where a missing one invents a word, and exactly wrong for scripts that do not separate words at all -- ja.wikipedia.org came back with spaces inside its own parentheses. The test is on the two characters meeting at the join, so a Latin word abutting a CJK one still gets its space. Worth recording from this corpus: the CJK and Arabic extractions are sound. They looked fragmented only because the measure counted characters, and a ten-character Chinese block is a sentence where a ten-character English one is a fragment.
The overall deadline covered the load wait and the read but not tier 0, so the stages ate each other: a host slow to the fetch client spent its whole fetch budget failing, the browser then began a fresh twenty-second load allowance on top of that, and the deadline expired before a single checkpoint had been taken. casper.com and womp.com each burned ten seconds on a fetch timeout, waited the full twenty for the page, and returned nothing with four hundred milliseconds left. Tier 0 runs before the page is loaded, so its budget belongs in the total beside the others rather than inside one of them. Its budget is also smaller now. A host that has not answered a plain GET in eight seconds is one the browser will have to handle regardless, and every further second is taken from the stage that will actually produce the artifact. This does not rescue casper.com or womp.com, and it is worth saying so: both stall headless Chromium as well, never reaching a loaded state in twenty seconds, so neither client gets a page. The artifact records that rather than pretending otherwise. What the change fixes is the class of site where the fetch is merely slow and the browser would have worked -- which previously lost to arithmetic rather than to anything about the page.
something of the visitor A site that puts a "click to enter" screen in front of itself is not hiding. Every visitor presses it, it asks nothing of them, and behind it is the page sieve was sent to read. Declaring it as a gap was honest and useless: quadricodes.tech returned one block reading "Skip Intro" and a note explaining why there was nothing else. What may be pressed is a short allow list -- enter, start, begin, continue, explore, skip, play, go -- and what may not is a longer deny list that is checked first and wins every tie. An age gate, a cookie wall, a sign-in and a purchase button all look exactly like entry gates and none of them is one: each asks the visitor to say something on their own behalf, and no tool has standing to say it for them. Those are still declared as gaps, and the artifact now names them as such rather than merely appearing thin. Around that: never inside a form, never a link that leaves the page, never a control that is invisible or covered by something else, at most two presses, only while the page is still showing almost nothing, and every press recorded in the artifact with what it said. The press itself is a real mouse event, because a great many of these ignore element.click() and listen for a trusted one. Waiting got better in the same pass. A page that is visibly assembling itself -- hatom.com counts from zero to a hundred across five phases -- is now waited out against the load budget rather than a round count, because abandoning it at ninety-six per cent means never seeing the site at all. A page that says nothing and is doing nothing gets far less patience, since that is also what a genuinely near-empty page looks like. quadricodes.tech goes from one block to ninety, in fifteen sections, repeatably.
answer "is it the browser?" A site built with a framework attaches its listener in JavaScript, so the thing a visitor presses is very often a plain div with no role, no href and no onclick attribute -- invisible to any selector looking for semantics. The search now falls back to matching on the words when the semantic pass finds nothing. That is not a loosening of what will be pressed: the allow list and the deny list are applied to the label either way, and a candidate still has to be visible, hit-testable, big enough to aim at, outside a form, and not a link that leaves the page. Only where sieve is willing to look has changed, and only on a page that is showing almost nothing to begin with. "Enable sound" joins the allow list, which is not obvious. A browser will not play audio until someone has touched the page, so a site built around sound has no choice but to ask, and asking is then the only thing between a reader and the content. Pressing it makes no noise: Chromium is launched muted. Two diagnostics, because the question "does this site behave differently because we are headless, or because WebGL is software?" cost a rebuild each time it came up. --headed runs with a visible window; SIEVE_GL overrides the rasteriser. The second is an environment variable rather than a flag on purpose: it changes rendering, so an artifact made with it is not comparable with one made without. Checked for false presses across example.com, igloo.inc, go.dev and pear.no: none of them is touched. quadricodes.tech is, and yields ninety blocks for it.
in words Console and network capture settled what was happening on hatom.com, and it was not what either of us assumed. There is no error: no uncaught exception, no failed request, a hundred and six clean responses. The loader reaches a hundred per cent, sits unchanged for a further twenty seconds while it builds its scene, and only then prints CLICK TO ENTER. Two rules were cutting it off before that, and both were wrong for the same reason -- they treated stillness as evidence. A page that says it is loading is now waited out against the operator's load budget rather than a count of unchanging looks. Twelve identical looks is about fourteen seconds of patience, so sieve gave up a few seconds before the door appeared, every time, and concluded the site was broken. And a page too busy to answer the probe at all was treated as a refusal to proceed. That silence is the busyness: hatom stops answering entirely while it builds, which is precisely the interval it is about to finish. Silence now buys patience, not a verdict. With those, CLICK TO ENTER is seen. It has no control to aim at -- the words sit on a layer with pointer events disabled while the handler listens on the window -- so a page that asks in plain language, and offers nothing to press, gets a press in the middle, with the surface checked first as ever. Two bounds came out of this, both from mistakes made getting here. The whole gate loop now has one wall-clock ceiling: composing patient rules with patient rules is how a loop stops terminating, and an earlier version ran for a quarter of an hour. And a press that changes nothing is not repeated -- quadricodes.tech was being clicked twice in seven hundred milliseconds, before it could react. hatom.com still does not open. It is pressed, exactly where and when it asks to be, and stays on its loading screen. Whatever it is waiting for is not this gesture, and the artifact says so instead of guessing.
The rule for pressing an entry screen was "little text, nothing changing,
press the middle". Across twenty sites that pressed a maintenance notice, a
"your browser is not supported" page, a canvas homepage and an ordinary
studio site -- six presses, no gates. Silence is not a door.
A door announces itself, in words or by covering the page, so findCover
looks for a layer painted over the whole viewport and nothing is pressed
without one. That also fixes the opposite failure: a splash laid over a
rendered page reports the entire site through innerText while showing a
visitor one word, so the character count that decided "not gated" was
reading a page nobody could see.
What was findable widened to match how these are actually built. A label
split into one span per letter is how every site that animates its entry
text does it, and a cap on child count rejected exactly those; asking
whether a child repeats the whole label rejects only true wrappers.
Ornamented labels ("-> Enter site") now match, and a page that asks for a
key gets one -- quadricodes.tech says "SKIP INTRO OR PRESS ENTER" and means
both.
Three bugs behind the ones above:
- A press was judged by looking again 400ms later. quadricodes.tech was
pressed correctly, found unchanged, written off, and its artifact fell
from ninety-odd blocks to one. The press was never the problem.
- The wait for a loader was bounded by a round count, not the clock, and
the rounds ran out in five seconds because the loop probed as fast as
the page could answer.
- The post-sweep probe overwrote the gate recorded on the way in, so an
age-gated page came back looking like a page with nothing on it.
sieve still refuses what asks something of the visitor. Age gates, consent
walls and sign-in are declared, not answered.
hatom.com is unchanged at two blocks. It is now pressed, exactly when and
where it asks to be, and does not open; what the artifact gained is a note
saying the page was still loading when the budget ran out, and which flag
to raise. That is the honest version of the same result.
igloo.inc came back empty with "its text was captured but none of it became content", which is the artifact admitting it does not know. Reading the snapshot back, what had been captured was twenty-four frames of an animated ASCII progress bar -- "++==------", "=++==-----", "==++==----" -- drawn dead centre of the viewport and correctly discarded as punctuation. The site serves an empty <body>, boots a WebGL scene into a bare div, and renders everything it has as geometry. Watched for sixty seconds in a plain browser it never produces a canvas element, a character of text, or an error, and hardware GL makes no difference. There is no text on igloo.inc. Zero blocks was the right answer given badly. So the punctuation-only exclusion is now recorded rather than silent, and when it accounts for a whole page the artifact says what that means: the content is on a canvas, not in the document, and a vision pass is the only thing that would describe it. Two gate bugs surfaced on the way. A page showing zero characters was treated as a quiet page and left immediately, when zero characters is a page that has not drawn yet -- igloo.inc is blank for a second and a half before it even puts up its loader, so the whole read happened against the loading screen. And a progress indicator drawn out of punctuation was not recognised as one, because looksLikeLoader only knew the word "loading" and a percentage. A run with no letter and no digit in it is not content in any language, which is tested by Unicode class so that a short CJK page is not mistaken for a spinner.
A 403 on robots.txt was treated as a refusal of the entire site, on the reasoning that a site which will not show its rules has not invited us in. That sounds careful and is not. RFC 9309 puts the whole 4xx range under "unavailable", meaning the rules do not exist for us and a crawler may access what it likes; 5xx is the range that means stop, because a server that cannot answer might be concealing a Disallow. sieve had the two backwards, and it cost five of a hundred sites -- sciencedirect.com, noaa.gov, ox.ac.uk, zara.com and stackoverflow.com all answer 403 to robots.txt from an unfamiliar user agent while serving their pages to anyone who asks. Every one of them now reads: ox.ac.uk alone goes from a refusal to 121 blocks across 39 sections. 429 stays a refusal, being the one 4xx that is about this client rather than about the file, and 5xx becomes one. Separately: the browser's HTTP status was collected and never looked at, so a page the server answered with an error was distilled as though it were the page requested. patagonia.com returns 404 to this client at both tiers and the artifact was a single block reading "Not found", with a hundred per cent retention and nothing to say the site had refused. An agent reading that would take it for the content of patagonia.com. Any 4xx or 5xx now marks the artifact as describing the error page it actually got.
Retention answers "how much of what the browser showed reached this artifact", and it counted blocks alone. A link's label reaches the artifact as an action, with its destination attached, which is more than a block would have carried -- so every page that is mostly an index reported a dismal score while losing nothing whatsoever. arxiv.org read 25.8%. Its 5439 visible characters are 1432 of prose and 4118 of labels across 257 links, every one of them present and addressable. kubernetes.io read 32.6%, news.ycombinator.com 44.8%, go.dev 79.8%; all four are now 100%, which is the true answer. The number itself was never the damage. An audit that cries wolf on the pages it handled perfectly is worse than no audit, because the figure it reports on a page that really did drop something no longer stands out. The metric still discriminates -- canada.ca sits at 93.4% and gov.uk at 92.4% against a hundred for the rest -- which is what it is for.
Two independent paths were breaking prose around inline elements, and both produced text that reads perfectly and is wrong. Missing text announces itself; reordered and welded text does not. In the static tier, an element's direct text children were gathered into one string and emitted before any recursion. That welds the sentence shut across the hole an inline element leaves in it, and emits the element's own words somewhere else entirely. "Several community <a>translations</a> are also available." came out as "Several community are also available." -- fluent, grammatical, and not what the page says. An agent quoting it is quoting something nobody wrote. The walk now interleaves an element's text with its children in document order, and lays a fragment's synthetic geometry out along one line so the reassembler joins it back into the sentence. A line is one element's own text flow, not any container. An element holding no words itself is only a container and its children each start fresh: danluu.com lists posts as <li><d>08/26</d><a>How do programming…</a></li>, and treating a bare container as a line welded the date onto the title. A space too many is a blemish; a space too few invents a word. In the rendered tier the same paragraphs came apart for a different reason. The footer band was computed as one document height minus three quarters of a viewport, which goes negative on a short page -- so the band covered the whole document and every short linked run anywhere on it was filed as footer furniture, pulled out of the reading order and appended after the content. A sentence ending "...and see Appendix E for information on editions." stopped mid-clause, with its conclusion emitted several paragraphs later and nothing saying it had moved. The band is now never more than the last quarter of the document, and a page that fits in one viewport has no footer band at all. Region is also decided per block-level container rather than per run. Every rule there judges a run alone -- its length, whether it links somewhere, where it sits -- so a paragraph with a link in it could be filed in two places at once. The container is the page's own statement about what belongs together, and it only ever rescues runs into the flow.
CI caught a real bug, and the failure was only ever hidden here by which fonts this machine happens to have. A run of text that wraps gets a bounding box that is the union of its lines, so its left edge is the left margin -- because some later line starts there -- and not the point on the first line where the words actually begin. Reassembly ordered fragments by that edge, so a wrapped run sorted as though it began at the margin, ahead of everything that really precedes it on its own first line. The effect is prose that reads as if a word had been deleted: "See the for instructions on installing or updating Rust, and see Installation section of Chapter 1" The link is torn out of the sentence and re-emitted after it. Two adjacent code spans came back in the opposite order the same way. Whether it happens at all depends on where the lines break, which depends on the font: the fixture was set in Georgia, this machine has Georgia, an Ubuntu runner does not, and DejaVu wraps the same paragraph differently. The first client rect is the first line box, and its left edge is where the run starts reading. That is now carried alongside the line top and is what ordering uses. The union stays as the box, because it is the honest extent and everything downstream wants that. The fixture is set in a generic family so the two machines agree on roughly where it breaks, and the assertions no longer depend on any particular wrapping: they anchor on single words and check the order of them, plus the thing that actually went wrong -- that the following paragraph must not appear inside this one. Verified against the paragraph rendered at four widths, from twenty rem to forty-six, which is a wider spread than any font difference produces.
igloo.inc has a full page of prose on it. sieve reported it as a page with no words, and I twice concluded that was the right answer -- the second time in a commit message, in as many words. It was not. The site serves an empty <body>, boots three.js inside a bundled ES module, and renders every paragraph as MSDF glyph geometry. It never attaches a canvas element at any point in its life. So every DOM-shaped question sieve knew how to ask returned nothing, and nothing contradicted that: no canvas, no text, no error. The absence of a canvas was the tell I walked past. A page really drawing to WebGL would have one; a page with neither has not booted for you, or is keeping its content somewhere you have not looked. The words were in the bundle all along, and in memory at runtime: _options.text = "Pudgy Penguins, a creative venture founded in 2021, ..." Three things stood between sieve and them. The scene could not be found. The walk scanned window for something scene-shaped, and a modern bundle puts nothing there. three.js checks for __THREE_DEVTOOLS__ inside the Scene constructor and dispatches the scene to it, so bootstrap.js now installs that hook before any page script runs -- which is what bootstrap.js is for. igloo announces seventeen scenes. The walk was gated on the document containing a canvas, which asks whether a surface is in the document rather than whether a scene exists. It now runs on every rendered page; a page with no scene returns immediately, and the global scan that was the old fallback is still gated, so the cost of asking is one evaluate against an empty hook. And the walk read only geometry.parameters.text, which is TextGeometry's spelling. Troika uses .text; the MSDF renderer igloo uses hangs _options off the mesh. Reading the handful of places implementations actually use is exact -- these are the site's own strings, not a guess about pixels -- so the runs need no corroboration and are emitted as paragraphs rather than welded into one summary blob. What they lack is layout: a scene has no line boxes, so the order is the order the page assembled them in, and the artifact says so rather than implying a reading order it did not measure. Two things this uncovered. The scene walk was handed an already-expired deadline whenever the sweep overran -- which on these pages it always does, because the sweep scrolls a blank document to the end of its allowance while the content sits unread in memory. It now has a floor. And anything appended after Build left the stats describing the document sieve started from: igloo reported twenty-eight tokens for twenty-three paragraphs, and zero per cent retention for a complete extraction. Both figures are recomputed now, which fixes the same staleness in canvas recovery. igloo.inc: 0 blocks -> 29, retention 100%, the whole site -- Summary, Discover, Visit, Manifesto, copyright.
…hat exist Three bugs, all found by driving the MCP server the way a host drives it rather than by distilling one page at a time. None of them can appear in the CLI, which is why none of them had. The redirect budget was per process, not per page. One guard was created for the server's lifetime and every hop of every page was counted against it, so the server answered four pages and then refused a hundred and eighty with "more than 8 redirects" -- including pages that redirect exactly once. Each distillation now takes its own copy of the guard: a copy rather than a reset because jobs run concurrently, and clearing a shared counter mid-chain would spend a budget another page was still using. Sections were built before the non-content prune, so a section whose only block was then dropped stayed in the manifest pointing at a block that no longer existed. moma.org advertised s_00, its one block was punctuation and was removed, and get_content answered "no section s_00" for a section the manifest had just described. A human shrugs at that; an agent following the manifest cannot tell an empty section from a request it got wrong, and stops. Sections are now built from the blocks that survive. And the keepalive dropped the session on a single missed pong, because the failure threshold was left at its zero value, which the SDK reads as one. Every content tool is keyed by job_id and the jobs live in this process, so one missed ping discarded every distillation the caller had already paid for -- to notice a dead peer fifteen seconds sooner. Three misses is ninety seconds of real silence, which is a dead peer rather than a busy one. Two of the three carry regression tests: a guard that survives twenty pages of three hops each while still refusing twenty hops on one page, and a check across four fixtures that every advertised section has blocks a caller can actually fetch. The second one fails on the old ordering, which is what makes it worth having.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.