Repository navigation
Send IndexNow what it refused again, and pages that appear or go away - #4
Merged
Merged
Conversation
A URL built by hand could put a space, an accent or a control character in a file, which made it unreadable. Rails locales such as pt_BR went out as hreflang="pt_BR", which search engines ignore. A lastmod string was written whatever it held, and the index took the greatest string rather than the latest time. x_default was missed when the locales block returned strings, and two languages sharing a URL listed it twice. URLs are now percent-encoded outside RFC 3986's characters, locales are written with a hyphen, lastmods are read as W3C dates and written in UTC, and a page lists each of its URLs once. The check reports URLs over 2,048 characters, hreflang values that are not language codes, lastmods that are not dates, and an index with no file or more than 50,000. Assisted-by: Claude Code
An impossible date such as 2026-13-01 crashed the build, and 2026-02-30 was written out. A date alone counted from UTC midnight, so today's date looked like the future east of UTC. A url block that forgot the locale no longer warned once its URLs were listed once per page. A protocol-relative URL got the host in front, and a % not followed by two hex digits stayed. Dates are validated, a date alone starts at UTC+14, a page whose languages share one URL is reported once per source, as is a lastmod that is not a date, and the lastmod is read once per page. An empty site no longer fails the check. Assisted-by: Claude Code
A record deleted and another inserted with an older timestamp left a shard's count and latest change alone, so its file kept the deleted page. A new shard size kept files cut for the old ranges. update_columns, a change in another table or a new url block went unseen until someone ran a rebuild. In the index, a file that lost a record kept its date or went back in time, and a build that died after uploading a new file left it there for good. The fingerprint adds the sum of the ids and the shard size. A file that comes out the same is not uploaded and keeps its date; one that changed without a later lastmod takes the build's time. A full build comes every week by itself (config.rebuild_every). The names a build is about to add are recorded in the manifest first, so the next build can remove them. Assisted-by: Claude Code
…builds SQLite's SUM overflows on ids near 2^63, so the fingerprint sums offsets within each range instead. A lastmod from an older manifest is compared as a time, not as text. A build that turned full because rebuild_every had passed says so in the result and in the build.mapitaine payload. Assisted-by: Claude Code
# Conflicts: # lib/mapitaine/builder.rb
On ruby-head, Active Support 8.0's encoder passes json an option it no longer takes (quirks_mode), so the configuration digest and the IndexNow body raised. Both hold plain values: JSON.generate writes the same bytes. Assisted-by: Claude Code
A refusal (429, 403) or an unreachable endpoint lost the URLs for good: the next build compared lastmods with its own start. Pages that went away were never sent, nor new pages whose lastmod predates the last build, nor pages dated by a day that was still today. One URL on another host made the endpoint refuse its whole batch, and with sitemaps on a CDN the key went to the CDN, where IndexNow does not read it. URLs the endpoint did not take stay in the manifest and go out with the next build, with a warning. For each rewritten file, the URLs of the file it replaces are compared with the new ones; removed files count too. A date alone counts as changed while it may still be the day of the last build. Only URLs on the host are sent, and the key is published only where IndexNow reads it, with a warning otherwise. Assisted-by: Claude Code
Submitting before the manifest meant an unexpected error or a slow endpoint held back the manifest after files were already published. The manifest is now written with the URLs still to send, then again once IndexNow answered, so a build that dies while submitting loses none. A new configuration or new ranges no longer read every previous file to send the whole site: only the lastmods count then. URLs are compared right after each shard renders, so no list of every URL waits for the others, a month or a year alone no longer counts as a change, the check reports a key the site does not serve, and Storage::Local reads files as bytes. Assisted-by: Claude Code
# Conflicts: # AGENTS.md # lib/mapitaine/builder.rb # lib/mapitaine/index_now.rb # lib/mapitaine/manifest.rb
The ruby-head job still failed on the two tests that rewrite a manifest with Hash#to_json, which Active Support 8.0 routes through its encoder. Assisted-by: Claude Code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A refusal (429, 403) or an unreachable endpoint lost its URLs for good, because the next build compared lastmods with its own start. Pages that went away were never sent, nor new pages whose lastmod predates the last build, nor pages dated by a day that was still today somewhere. One URL on another host made the endpoint refuse its whole batch, and with the sitemaps on a CDN the key went to the CDN, where IndexNow does not read it.
The manifest now keeps the URLs IndexNow has yet to take, each rewritten file is compared with the one it replaces, and the key is published only where IndexNow reads it.
config.hostare sent; withsitemaps_urlelsewhere, or a key location on another host, builds andmapitaine:checkwarn instead of uploading the key where IndexNow never looks.Proof
73 runs, 202 assertions, green on Rails 8.1 and 8.0; RuboCop clean. The new tests fail on
main: a 429 reported and sent again with the next build, a removed and an added page with old lastmods submitted, a date of the day of the last build, a foreign host filtered out, a key the site does not serve reported by builds and by the check, one reply per batch, a build dying while submitting, a new configuration and new ranges sending only edited pages, and a year-only lastmod.The ruby-head · Rails 8.0 failure on #3 was
ArgumentError: unknown keyword: quirks_modeattest/build_test.rb:166and:202, fromHash#to_json; those lines now callJSON.generate.Generated by Claude Code