Repository navigation
Catch more changes between builds, and date files by their last change - #3
Merged
Merged
Conversation
A URL built by hand could put a space, an accent or a control character in a file, which made it unreadable. Rails locales such as pt_BR went out as hreflang="pt_BR", which search engines ignore. A lastmod string was written whatever it held, and the index took the greatest string rather than the latest time. x_default was missed when the locales block returned strings, and two languages sharing a URL listed it twice. URLs are now percent-encoded outside RFC 3986's characters, locales are written with a hyphen, lastmods are read as W3C dates and written in UTC, and a page lists each of its URLs once. The check reports URLs over 2,048 characters, hreflang values that are not language codes, lastmods that are not dates, and an index with no file or more than 50,000. Assisted-by: Claude Code
An impossible date such as 2026-13-01 crashed the build, and 2026-02-30 was written out. A date alone counted from UTC midnight, so today's date looked like the future east of UTC. A url block that forgot the locale no longer warned once its URLs were listed once per page. A protocol-relative URL got the host in front, and a % not followed by two hex digits stayed. Dates are validated, a date alone starts at UTC+14, a page whose languages share one URL is reported once per source, as is a lastmod that is not a date, and the lastmod is read once per page. An empty site no longer fails the check. Assisted-by: Claude Code
A record deleted and another inserted with an older timestamp left a shard's count and latest change alone, so its file kept the deleted page. A new shard size kept files cut for the old ranges. update_columns, a change in another table or a new url block went unseen until someone ran a rebuild. In the index, a file that lost a record kept its date or went back in time, and a build that died after uploading a new file left it there for good. The fingerprint adds the sum of the ids and the shard size. A file that comes out the same is not uploaded and keeps its date; one that changed without a later lastmod takes the build's time. A full build comes every week by itself (config.rebuild_every). The names a build is about to add are recorded in the manifest first, so the next build can remove them. Assisted-by: Claude Code
…builds SQLite's SUM overflows on ids near 2^63, so the fingerprint sums offsets within each range instead. A lastmod from an older manifest is compared as a time, not as text. A build that turned full because rebuild_every had passed says so in the result and in the build.mapitaine payload. Assisted-by: Claude Code
# Conflicts: # lib/mapitaine/builder.rb
On ruby-head, Active Support 8.0's encoder passes json an option it no longer takes (quirks_mode), so the configuration digest and the IndexNow body raised. Both hold plain values: JSON.generate writes the same bytes. Assisted-by: Claude Code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A shard's fingerprint was its count and latest change, so a record deleted while another was inserted with an older timestamp left the file listing the deleted page. A new
shard_sizekept files cut for the old ranges.update_columns, a change in another table or a newurlblock went unseen until someone ran a rebuild. In the index, a file that lost a record kept its date or went back in time, and a build that died after uploading a new file left that file there for good.The fingerprint now also holds the sum of the ids (as offsets within the range) and the shard size, files are compared by content before upload, the index dates a changed file by its change, and a full build comes once a week by itself.
size/count/sum/latestper range catches a record replaced by an older one and a new shard size; SQLite cannot overflow on large ids.lastmodtakes the build's time, so crawlers read it again.config.rebuild_every(a week by default,nilto turn off) catchesupdate_columns, other tables, newurlblocks and files lost from the storage;Result#fulland thebuild.mapitainepayload say when it happened.pendingin one write, and the next build removes them if this one dies.JSON.generate, which writes the same bytes.Proof
64 runs, 183 assertions, green on Rails 8.1 and 8.0; RuboCop clean. The new tests fail on
main: a record replaced by an older one, a new shard size, a deletion moving the index date forward, a re-rendered file not uploaded, a full build restoring a deleted file without changing dates, the weekly full build, a build dying before its index, ids near 9×10^18, and a lastmod from an older manifest compared by time.The ruby-head failure on #2 was
ArgumentError: unknown keyword: quirks_modefromActiveSupport::JSON::Encoding::JSONGemEncoder#stringify, raised byConfiguration#digest.JSON.generategives the same string asto_jsonfor these values, so the digest does not change.bench/build.rb(30,000 records × 24 languages, Ruby 3.3.6 without YJIT): full build 29.9 s and 30.7 s, nothing new 0.014 s, three records changed 1.06 s, the same as onmainwithin its spread.Generated by Claude Code