Send IndexNow what it refused again, and pages that appear or go away

AGENTS.md

@@ -29,8 +29,9 @@ joins CI's `rails` matrix.
   `Configuration#digest` (the file name), in each shard's fingerprint (the shard size), or rewrites everything by
   itself (the slug).
 - A build renders every shard, records the names it is about to add as `pending` in the manifest, publishes the
-  sitemap files, then the index, then deletes, then writes the manifest. Keep that order: it is what lets the next
-  build repair one that died midway.
+  sitemap files, then the index, then deletes, then writes the manifest with the URLs IndexNow has yet to take,
+  submits them, and writes the manifest again. Keep that order: it is what lets the next build repair one that died
+  midway.
 - `priority`, `changefreq` and the ping endpoints are left out on purpose.
 - The README is the user guide and `docs/` the GitHub Pages site: change them with the code. The README's table and
   `docs/benchmarks.html` come from the same `bench/build.rb` run.

README.md

@@ -220,10 +220,17 @@ A storage is any object with `read(name)`, `write(name, path)` and `delete(name)
 config.index_now_key = "c1f0e2b4a6d84f3e9b7a5c3d1e0f2a4b"   # yours: 8 to 128 characters, a-z, A-Z, 0-9 and -
 ```
 
-Each build then publishes the key as `<key>.txt` next to the sitemaps, and sends the URLs of the rewritten files
-whose `lastmod` is later than the previous build. The first build sends nothing. When the key cannot live at
-the root of the site, host it yourself and set `config.index_now_key_location` to its URL. An endpoint that
-cannot be reached is a warning, never a failed build.
+Each build then publishes the key as `<key>.txt` at the root of the site, and sends, from the files it rewrote,
+the URLs whose `lastmod` is later than the previous build, and those that appeared or went away whatever their
+`lastmod`. The first build sends nothing, nor does a new configuration beyond the pages whose `lastmod` moved,
+and only URLs on `config.host` are sent.
+
+When the sitemaps are served from elsewhere (`config.sitemaps_url`), the key cannot follow them: IndexNow reads
+it on the site. Put `<key>.txt` in `public/` and set `config.index_now_key_location` to its URL; a build warns
+until you do.
+
+A refusal or an endpoint that cannot be reached is a warning, never a failed build: the URLs it did not take are
+kept in the manifest and sent again with the next build.
 
 ## Checking
 

docs/guide.html

@@ -180,8 +180,9 @@ <h2 id="indexnow-title">IndexNow</h2>
       <div class="prose">
         <p><a href="https://www.indexnow.org">IndexNow</a> tells Bing, Yandex, Seznam and Naver that a URL changed. Google does not take part. Choose a key of 8 to 128 characters among a-z, A-Z, 0-9 and -:</p>
 <pre><code class="language-ruby"><span class="n">config</span><span class="p">.</span><span class="nf">index_now_key</span> <span class="o">=</span> <span class="s2">"c1f0e2b4a6d84f3e9b7a5c3d1e0f2a4b"</span></code></pre>
-        <p>Each build then publishes the key as <code>&lt;key&gt;.txt</code> next to the sitemaps, and sends the URLs of the rewritten files whose <code>lastmod</code> is later than the previous build. The first build sends nothing.</p>
-        <p>When the key cannot live at the root of the site, host it yourself and set <code>config.index_now_key_location</code> to its URL. An endpoint that cannot be reached is a warning, never a failed build.</p>
+        <p>Each build then publishes the key as <code>&lt;key&gt;.txt</code> at the root of the site, and sends, from the files it rewrote, the URLs whose <code>lastmod</code> is later than the previous build, and those that appeared or went away whatever their <code>lastmod</code>. The first build sends nothing, nor does a new configuration beyond the pages whose <code>lastmod</code> moved, and only URLs on <code>config.host</code> are sent.</p>
+        <p>When the sitemaps are served from elsewhere (<code>config.sitemaps_url</code>), the key cannot follow them: IndexNow reads it on the site. Put <code>&lt;key&gt;.txt</code> in <code>public/</code> and set <code>config.index_now_key_location</code> to its URL; a build warns until you do.</p>
+        <p>A refusal or an endpoint that cannot be reached is a warning, never a failed build: the URLs it did not take are kept in the manifest and sent again with the next build.</p>
       </div>
     </section>
 

docs/index.html

@@ -108,7 +108,7 @@ <h3>Any bucket Active Storage reaches</h3>
         </div>
         <div class="card">
           <h3>IndexNow</h3>
-          <p>Each build sends Bing, Yandex, Seznam and Naver the URLs that changed since the previous one. The first build sends nothing.</p>
+          <p>Each build sends Bing, Yandex, Seznam and Naver the URLs that changed, appeared or went away since the previous one, and sends again what they refused. The first build sends nothing.</p>
           <p><a href="guide.html#indexnow">IndexNow</a></p>
         </div>
         <div class="card">

lib/mapitaine/builder.rb

@@ -12,6 +12,10 @@ def changed?
     # https://www.sitemaps.org/protocol.html
     MAX_URL_CHARACTERS = 2_048
     MAX_FILES_PER_INDEX = 50_000
+    LAST_TIME_ZONE_OFFSET_SECONDS = 12 * 60 * 60
+    DAY_LENGTH = "2026-01-01".length
+    # One request's worth: what a refusing endpoint leaves in the manifest stays small.
+    MAX_UNSUBMITTED_URLS = IndexNow::MAX_URLS_PER_REQUEST
     # A language, then maybe a script and a region: en, pt-BR, zh-Hant-TW, es-419.
     HREFLANG = /\A[a-z]{2,3}(-[a-z]{4})?(-([a-z]{2}|\d{3}))?\z/i
 
@@ -56,16 +60,27 @@ def build_in(directory, full:)
       @x_default = Xml.hreflang(@configuration.x_default)
 
       @rendered = []
+      # The first build has no "before": submitting a whole site at once is not what IndexNow is for. Nor is a
+      # new configuration, which rewrites every file: then only the lastmods tell what changed.
+      @tracking = @submit && @configuration.index_now_key.present? && @previous.built_at.present?
+      @diffing = @tracking && @previous.digest == @configuration.digest
+      @moved_sources = Set.new
 
       @configuration.sources.each { |source_class| build_source(source_class.new(@configuration)) }
       publish_rendered
       publish_index
       deleted = (@previous.file_names | @previous.pending) - @manifest.file_names
+      @changed_urls.concat(urls_in(deleted.reject { |name| @moved_sources.include?(previous_source_of(name)) })) if @diffing
       deleted.each { |name| @storage.delete(name) }
+      inspect_index_now_key
+      urls = urls_to_submit
+      # Written before submitting, with the URLs still to send: a build that dies while submitting loses none.
+      @manifest.unsubmitted.replace(urls.last(MAX_UNSUBMITTED_URLS))
       upload(manifest_name, @manifest.to_json)
+      submitted = submit(urls)
 
       Result.new(full: @full, written: @written, kept: @kept, deleted:, url_count: @manifest.files.sum { |file| file["urls"] },
-        submitted: submit_changed_urls, warnings: @warnings)
+        submitted:, warnings: @warnings)
     end
 
     def build_source(source)
@@ -88,32 +103,69 @@ def keep(source, shard, stored)
     def render(source, shard, fingerprint, seen_urls)
       writer = ShardWriter.new(directory: @directory, basename: source.basename(shard))
       changed_urls = []
+      urls = [] if @diffing
       ActiveSupport::Notifications.instrument("shard.mapitaine", source: source.slug, shard:) do
         source.each_page(shard) do |page|
           inspect_page(source, page, seen_urls)
           writer.add(Xml.url_entries(page, x_default: @x_default), page.lastmod)
-          changed_urls.concat(page.urls) if changed_since_last_build?(page)
+          changed_urls.concat(page.urls) if @tracking && changed_since_last_build?(page)
+          urls&.concat(page.urls)
         end
         writer.close
       end
 
+      if writer.files.all? { |file| same_as_before?(file) }
+        changed_urls.clear
+      elsif urls
+        changed_urls.concat(appeared_or_gone(source.slug, shard, fingerprint, urls))
+      end
       fingerprint ||= writer.content_digest
       # Recorded now, published once every shard is rendered: the index keeps the order of the sources.
       @manifest.record(source.slug, shard, fingerprint:, files: [])
       @rendered << Rendered.new(source_slug: source.slug, shard:, fingerprint:, files: writer.files, changed_urls:)
     end
 
+    # Read while the previous files are still in the storage: a page that appeared or went away is a change too,
+    # whatever its lastmod says. Not when the ranges moved, which only carries pages from one file to another.
+    def appeared_or_gone(source_slug, shard, fingerprint, urls)
+      stored = @previous.shard(source_slug, shard)
+      if stored && fingerprint && stored["fingerprint"].to_s.split("/").first != fingerprint.split("/").first
+        @moved_sources << source_slug
+        return []
+      end
+
+      previous_urls = urls_in((stored&.fetch("files") || []).map { |file| file["name"] }).to_set
+      urls = urls.to_set
+      (urls - previous_urls).to_a + (previous_urls - urls).to_a
+    end
+
     def publish_rendered
       changed_names = @rendered.flat_map(&:files).reject { |file| same_as_before?(file) }.to_set(&:name)
       announce_new_files(changed_names)
       @rendered.each do |rendered|
+        @changed_urls.concat(rendered.changed_urls)
         entries = rendered.files.map do |file|
           changed = changed_names.include?(file.name)
           publish(file, changed)
           manifest_entry(file, changed)
         end
         @manifest.record(rendered.source_slug, rendered.shard, fingerprint: rendered.fingerprint, files: entries)
-        @changed_urls.concat(rendered.changed_urls) if rendered.files.any? { |file| changed_names.include?(file.name) }
+      end
+    end
+
+    def previous_source_of(file_name)
+      @previous_source_by_file_name ||= @previous.shards_by_source.each_with_object({}) do |(source_slug, shards), source_by_name|
+        shards.each_value { |stored| stored["files"].each { |file| source_by_name[file["name"]] = source_slug } }
+      end
+      @previous_source_by_file_name[file_name]
+    end
+
+    def urls_in(names)
+      names.flat_map do |name|
+        content = @storage.read(name)
+        content ? Xml.locations(Zlib.gunzip(content.b)) : []
+      rescue Zlib::Error
+        []
       end
     end
 
@@ -189,7 +241,7 @@ def manifest_name
 
     def inspect_page(source, page, seen_urls)
       page.urls.each do |url|
-        warn(source, url, "is outside #{@configuration.host!}") unless url.start_with?("#{@configuration.host!}/") || url == @configuration.host!
+        warn(source, url, "is outside #{@configuration.host!}") unless on_host?(url)
         warn(source, url, "is listed twice") unless seen_urls.add?(url)
         warn(source, url, "is longer than #{MAX_URL_CHARACTERS} characters") if url.length > MAX_URL_CHARACTERS
       end
@@ -225,23 +277,57 @@ def warn(source, url, message)
       @warnings << Warning.new(source: source.is_a?(Source) ? source.slug : source, url:, message:)
     end
 
-    # The first build has no "before": submitting a whole site at once is not what IndexNow is for.
+    def on_host?(url)
+      url.start_with?("#{@configuration.host!}/") || url == @configuration.host!
+    end
+
+    # A day may still be the day of the last build somewhere: it changed if it is not older than the date at
+    # UTC-12, the last time zone to leave a day. A month or a year alone says too little to tell.
     def changed_since_last_build?(page)
-      @previous.built_at && page.modified_at && page.modified_at >= @previous.built_at
+      return false unless page.lastmod
+      return page.modified_at >= @previous.built_at if page.lastmod.include?("T")
+
+      page.lastmod.length == DAY_LENGTH && page.lastmod >= (@previous.built_at - LAST_TIME_ZONE_OFFSET_SECONDS).utc.to_date.iso8601
     end
 
-    def submit_changed_urls
-      key = @configuration.index_now_key
-      return 0 unless @submit && key
+    def urls_to_submit
+      return [] unless @submit && @configuration.index_now_key
+
+      (@previous.unsubmitted + @changed_urls).uniq.select { |url| on_host?(url) }
+    end
+
+    # Returns how many URLs the endpoint took. The others, refused or unreachable, stay in the manifest for the
+    # next build: the sitemaps are published, so a refusal never fails the build.
+    def submit(urls)
+      return 0 if urls.empty?
+
+      replies = IndexNow.new(host: @configuration.host!, key: @configuration.index_now_key,
+        key_location: @configuration.index_now_key_location).submit(urls)
+      refused = replies.reject(&:accepted?)
+      refused.group_by(&:status).each do |status, replies_with_status|
+        count = replies_with_status.sum { |reply| reply.urls.size }
+        answer = status.is_a?(Integer) ? "answered #{status}" : "could not be reached (#{status})"
+        warn("indexnow", IndexNow::ENDPOINT, "#{answer} for #{count} URLs, sent again with the next build")
+      end
+      @manifest.unsubmitted.replace(refused.flat_map(&:urls).last(MAX_UNSUBMITTED_URLS))
+      upload(manifest_name, @manifest.to_json)
+      replies.select(&:accepted?).sum { |reply| reply.urls.size }
+    end
 
-      upload("#{key}.txt", key) unless @configuration.index_now_key_location
-      return 0 if @changed_urls.empty?
+    # IndexNow reads the key at the root of the site unless told where it is, and only on the site's host.
+    def inspect_index_now_key
+      key = @configuration.index_now_key
+      location = @configuration.index_now_key_location
+      return unless key
 
-      IndexNow.new(host: @configuration.host!, key:, key_location: @configuration.index_now_key_location).submit(@changed_urls)
-    rescue *IndexNow::NETWORK_ERRORS => exception
-      # The sitemaps are published: a search engine that cannot be reached must not fail the build.
-      @warnings << Warning.new(source: "indexnow", url: IndexNow::ENDPOINT.to_s, message: "could not be reached (#{exception.class})")
-      0
+      if location
+        warn("indexnow", location, "is not on #{@configuration.host!}: IndexNow only reads a key there") unless on_host?(location)
+      elsif @configuration.sitemaps_url == @configuration.host!
+        upload("#{key}.txt", key)
+      else
+        warn("indexnow", "#{@configuration.host!}/#{key}.txt", "must hold the key: sitemaps go to #{@configuration.sitemaps_url}, " \
+          "so put #{key}.txt in public/ and set index_now_key_location to this URL")
+      end
     end
   end
 end

lib/mapitaine/index_now.rb

@@ -15,12 +15,20 @@ def initialize(host:, key:, key_location: nil, transport: Net::HTTP.method(:post
       @transport = transport
     end
 
-    # Returns how many URLs the endpoint took.
+    # status: the HTTP status, or the class of the network error.
+    Reply = Data.define(:urls, :status) do
+      def accepted?
+        status.is_a?(Integer) && status.between?(200, 299)
+      end
+    end
+
+    # One reply per request of up to 10,000 URLs.
     def submit(urls)
-      urls.each_slice(MAX_URLS_PER_REQUEST).sum do |batch|
+      urls.each_slice(MAX_URLS_PER_REQUEST).map do |batch|
         body = {host: @host, key: @key, keyLocation: @key_location, urlList: batch}.compact
-        response = @transport.call(ENDPOINT, JSON.generate(body), HEADERS)
-        response.code.to_i.between?(200, 299) ? batch.size : 0
+        Reply.new(urls: batch, status: @transport.call(ENDPOINT, JSON.generate(body), HEADERS).code.to_i)
+      rescue *NETWORK_ERRORS => exception
+        Reply.new(urls: batch, status: exception.class.name)
       end
     end
   end

lib/mapitaine/manifest.rb

@@ -2,17 +2,18 @@
 
 module Mapitaine
   # What the last build wrote: for each shard of each source, its fingerprint and its files. pending lists the
-  # files a build started to publish before it could record them, so that the next one can remove them.
+  # files a build started to publish before it could record them, so that the next one can remove them;
+  # unsubmitted the URLs IndexNow did not take, for the next one to send again.
   class Manifest
-    attr_reader :built_at, :rebuilt_at, :digest, :shards_by_source, :pending
+    attr_reader :built_at, :rebuilt_at, :digest, :shards_by_source, :pending, :unsubmitted
 
     def self.read(storage, name)
       json = storage.read(name)
       return new unless json
 
       stored = JSON.parse(json)
       new(digest: stored["digest"], built_at: time(stored["built_at"]), rebuilt_at: time(stored["rebuilt_at"]),
-        shards_by_source: stored["sources"] || {}, pending: stored["pending"] || [])
+        shards_by_source: stored["sources"] || {}, pending: stored["pending"] || [], unsubmitted: stored["unsubmitted"] || [])
     rescue JSON::ParserError, ArgumentError, TypeError
       # A manifest we cannot read is a manifest we do not have: the build starts over and replaces it.
       new
@@ -22,12 +23,13 @@ def self.time(text)
       Time.iso8601(text) if text
     end
 
-    def initialize(digest: nil, built_at: nil, rebuilt_at: nil, shards_by_source: {}, pending: [])
+    def initialize(digest: nil, built_at: nil, rebuilt_at: nil, shards_by_source: {}, pending: [], unsubmitted: [])
       @digest = digest
       @built_at = built_at
       @rebuilt_at = rebuilt_at
       @shards_by_source = shards_by_source
       @pending = pending
+      @unsubmitted = unsubmitted
     end
 
     def shard(source, key)
@@ -50,7 +52,7 @@ def to_json(*)
       # Whole seconds, like the lastmods compared to built_at: a change in the second of a build is sent again
       # rather than never.
       JSON.pretty_generate("digest" => digest, "built_at" => built_at&.utc&.iso8601,
-        "rebuilt_at" => rebuilt_at&.utc&.iso8601, "pending" => pending, "sources" => shards_by_source)
+        "rebuilt_at" => rebuilt_at&.utc&.iso8601, "pending" => pending, "unsubmitted" => unsubmitted, "sources" => shards_by_source)
     end
   end
 end

lib/mapitaine/storage/local.rb

@@ -13,7 +13,7 @@ def initialize(directory)
 
       def read(name)
         path = File.join(@directory, name)
-        File.read(path) if File.exist?(path)
+        File.binread(path) if File.exist?(path)
       end
 
       # Copied beside the target, then renamed: a crawler never downloads half a file.

lib/mapitaine/xml.rb

@@ -15,6 +15,9 @@ module Xml
     W3C_DATE = /\A(\d{4})(?:-(\d{2})(?:-(\d{2}))?)?\z/
     W3C_TIME = /\A(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2})(?::(\d{2})(?:\.\d+)?)?(Z|[+-]\d{2}:\d{2})\z/
 
+    LOCATION = %r{<loc>([^<]*)</loc>}
+    UNESCAPED_BY_ENTITY = {"&amp;" => "&", "&lt;" => "<", "&gt;" => ">", "&quot;" => '"', "&apos;" => "'"}.freeze
+
     module_function
 
     # One <url> per language. Each lists every language of the page, itself included: search engines drop
@@ -66,6 +69,11 @@ def hreflang(locale)
       locale.to_s.tr("_", "-")
     end
 
+    # The <loc> of each <url> of a sitemap file.
+    def locations(xml)
+      xml.scan(LOCATION).map { |(location)| location.gsub(/&(?:amp|lt|gt|quot|apos);/, UNESCAPED_BY_ENTITY) }
+    end
+
     def escape(text)
       text.to_s.encode(xml: :text).gsub('"', "&quot;")
     end

test/build_test.rb

@@ -163,7 +163,7 @@ def test_a_file_dated_by_an_older_manifest_is_compared_by_time
     build
     manifest = JSON.parse(storage.content_by_name["sitemap-manifest.json"])
     manifest["sources"]["post"]["0"]["files"][0]["lastmod"] = "2026-01-15T10:00:00+09:00"
-    storage.content_by_name["sitemap-manifest.json"] = manifest.to_json
+    storage.content_by_name["sitemap-manifest.json"] = JSON.generate(manifest)
     Post.where(id: 1).update_all(title: "Edited", updated_at: Time.utc(2026, 1, 15, 6))
     build
 
@@ -199,15 +199,15 @@ def test_a_full_build_comes_every_week_by_itself
     build
     manifest = JSON.parse(storage.content_by_name["sitemap-manifest.json"])
     manifest["rebuilt_at"] = (Time.now - 8 * 86_400).utc.iso8601
-    storage.content_by_name["sitemap-manifest.json"] = manifest.to_json
+    storage.content_by_name["sitemap-manifest.json"] = JSON.generate(manifest)
 
     built = build
 
     assert_predicate built, :full
     assert_equal %w[sitemap-post-00000.xml.gz sitemap-page.xml.gz], built.written
     refute_predicate build, :full
     Mapitaine.configuration.rebuild_every = nil
-    storage.content_by_name["sitemap-manifest.json"] = manifest.to_json
+    storage.content_by_name["sitemap-manifest.json"] = JSON.generate(manifest)
 
     assert_empty build.written
   end