diff --git a/AGENTS.md b/AGENTS.md index 46b1f4e..0d80dc6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,8 +29,9 @@ joins CI's `rails` matrix. `Configuration#digest` (the file name), in each shard's fingerprint (the shard size), or rewrites everything by itself (the slug). - A build renders every shard, records the names it is about to add as `pending` in the manifest, publishes the - sitemap files, then the index, then deletes, then writes the manifest. Keep that order: it is what lets the next - build repair one that died midway. + sitemap files, then the index, then deletes, then writes the manifest with the URLs IndexNow has yet to take, + submits them, and writes the manifest again. Keep that order: it is what lets the next build repair one that died + midway. - `priority`, `changefreq` and the ping endpoints are left out on purpose. - The README is the user guide and `docs/` the GitHub Pages site: change them with the code. The README's table and `docs/benchmarks.html` come from the same `bench/build.rb` run. diff --git a/README.md b/README.md index 56cca6a..4982485 100644 --- a/README.md +++ b/README.md @@ -220,10 +220,17 @@ A storage is any object with `read(name)`, `write(name, path)` and `delete(name) config.index_now_key = "c1f0e2b4a6d84f3e9b7a5c3d1e0f2a4b" # yours: 8 to 128 characters, a-z, A-Z, 0-9 and - ``` -Each build then publishes the key as `.txt` next to the sitemaps, and sends the URLs of the rewritten files -whose `lastmod` is later than the previous build. The first build sends nothing. When the key cannot live at -the root of the site, host it yourself and set `config.index_now_key_location` to its URL. An endpoint that -cannot be reached is a warning, never a failed build. +Each build then publishes the key as `.txt` at the root of the site, and sends, from the files it rewrote, +the URLs whose `lastmod` is later than the previous build, and those that appeared or went away whatever their +`lastmod`. The first build sends nothing, nor does a new configuration beyond the pages whose `lastmod` moved, +and only URLs on `config.host` are sent. + +When the sitemaps are served from elsewhere (`config.sitemaps_url`), the key cannot follow them: IndexNow reads +it on the site. Put `.txt` in `public/` and set `config.index_now_key_location` to its URL; a build warns +until you do. + +A refusal or an endpoint that cannot be reached is a warning, never a failed build: the URLs it did not take are +kept in the manifest and sent again with the next build. ## Checking diff --git a/docs/guide.html b/docs/guide.html index 34770ff..0b6e1de 100644 --- a/docs/guide.html +++ b/docs/guide.html @@ -180,8 +180,9 @@

IndexNow

IndexNow tells Bing, Yandex, Seznam and Naver that a URL changed. Google does not take part. Choose a key of 8 to 128 characters among a-z, A-Z, 0-9 and -:

config.index_now_key = "c1f0e2b4a6d84f3e9b7a5c3d1e0f2a4b"
-

Each build then publishes the key as <key>.txt next to the sitemaps, and sends the URLs of the rewritten files whose lastmod is later than the previous build. The first build sends nothing.

-

When the key cannot live at the root of the site, host it yourself and set config.index_now_key_location to its URL. An endpoint that cannot be reached is a warning, never a failed build.

+

Each build then publishes the key as <key>.txt at the root of the site, and sends, from the files it rewrote, the URLs whose lastmod is later than the previous build, and those that appeared or went away whatever their lastmod. The first build sends nothing, nor does a new configuration beyond the pages whose lastmod moved, and only URLs on config.host are sent.

+

When the sitemaps are served from elsewhere (config.sitemaps_url), the key cannot follow them: IndexNow reads it on the site. Put <key>.txt in public/ and set config.index_now_key_location to its URL; a build warns until you do.

+

A refusal or an endpoint that cannot be reached is a warning, never a failed build: the URLs it did not take are kept in the manifest and sent again with the next build.

diff --git a/docs/index.html b/docs/index.html index 7730555..3cf4883 100644 --- a/docs/index.html +++ b/docs/index.html @@ -108,7 +108,7 @@

Any bucket Active Storage reaches

IndexNow

-

Each build sends Bing, Yandex, Seznam and Naver the URLs that changed since the previous one. The first build sends nothing.

+

Each build sends Bing, Yandex, Seznam and Naver the URLs that changed, appeared or went away since the previous one, and sends again what they refused. The first build sends nothing.

IndexNow

diff --git a/lib/mapitaine/builder.rb b/lib/mapitaine/builder.rb index 55c3743..e5464a4 100644 --- a/lib/mapitaine/builder.rb +++ b/lib/mapitaine/builder.rb @@ -12,6 +12,10 @@ def changed? # https://www.sitemaps.org/protocol.html MAX_URL_CHARACTERS = 2_048 MAX_FILES_PER_INDEX = 50_000 + LAST_TIME_ZONE_OFFSET_SECONDS = 12 * 60 * 60 + DAY_LENGTH = "2026-01-01".length + # One request's worth: what a refusing endpoint leaves in the manifest stays small. + MAX_UNSUBMITTED_URLS = IndexNow::MAX_URLS_PER_REQUEST # A language, then maybe a script and a region: en, pt-BR, zh-Hant-TW, es-419. HREFLANG = /\A[a-z]{2,3}(-[a-z]{4})?(-([a-z]{2}|\d{3}))?\z/i @@ -56,16 +60,27 @@ def build_in(directory, full:) @x_default = Xml.hreflang(@configuration.x_default) @rendered = [] + # The first build has no "before": submitting a whole site at once is not what IndexNow is for. Nor is a + # new configuration, which rewrites every file: then only the lastmods tell what changed. + @tracking = @submit && @configuration.index_now_key.present? && @previous.built_at.present? + @diffing = @tracking && @previous.digest == @configuration.digest + @moved_sources = Set.new @configuration.sources.each { |source_class| build_source(source_class.new(@configuration)) } publish_rendered publish_index deleted = (@previous.file_names | @previous.pending) - @manifest.file_names + @changed_urls.concat(urls_in(deleted.reject { |name| @moved_sources.include?(previous_source_of(name)) })) if @diffing deleted.each { |name| @storage.delete(name) } + inspect_index_now_key + urls = urls_to_submit + # Written before submitting, with the URLs still to send: a build that dies while submitting loses none. + @manifest.unsubmitted.replace(urls.last(MAX_UNSUBMITTED_URLS)) upload(manifest_name, @manifest.to_json) + submitted = submit(urls) Result.new(full: @full, written: @written, kept: @kept, deleted:, url_count: @manifest.files.sum { |file| file["urls"] }, - submitted: submit_changed_urls, warnings: @warnings) + submitted:, warnings: @warnings) end def build_source(source) @@ -88,32 +103,69 @@ def keep(source, shard, stored) def render(source, shard, fingerprint, seen_urls) writer = ShardWriter.new(directory: @directory, basename: source.basename(shard)) changed_urls = [] + urls = [] if @diffing ActiveSupport::Notifications.instrument("shard.mapitaine", source: source.slug, shard:) do source.each_page(shard) do |page| inspect_page(source, page, seen_urls) writer.add(Xml.url_entries(page, x_default: @x_default), page.lastmod) - changed_urls.concat(page.urls) if changed_since_last_build?(page) + changed_urls.concat(page.urls) if @tracking && changed_since_last_build?(page) + urls&.concat(page.urls) end writer.close end + if writer.files.all? { |file| same_as_before?(file) } + changed_urls.clear + elsif urls + changed_urls.concat(appeared_or_gone(source.slug, shard, fingerprint, urls)) + end fingerprint ||= writer.content_digest # Recorded now, published once every shard is rendered: the index keeps the order of the sources. @manifest.record(source.slug, shard, fingerprint:, files: []) @rendered << Rendered.new(source_slug: source.slug, shard:, fingerprint:, files: writer.files, changed_urls:) end + # Read while the previous files are still in the storage: a page that appeared or went away is a change too, + # whatever its lastmod says. Not when the ranges moved, which only carries pages from one file to another. + def appeared_or_gone(source_slug, shard, fingerprint, urls) + stored = @previous.shard(source_slug, shard) + if stored && fingerprint && stored["fingerprint"].to_s.split("/").first != fingerprint.split("/").first + @moved_sources << source_slug + return [] + end + + previous_urls = urls_in((stored&.fetch("files") || []).map { |file| file["name"] }).to_set + urls = urls.to_set + (urls - previous_urls).to_a + (previous_urls - urls).to_a + end + def publish_rendered changed_names = @rendered.flat_map(&:files).reject { |file| same_as_before?(file) }.to_set(&:name) announce_new_files(changed_names) @rendered.each do |rendered| + @changed_urls.concat(rendered.changed_urls) entries = rendered.files.map do |file| changed = changed_names.include?(file.name) publish(file, changed) manifest_entry(file, changed) end @manifest.record(rendered.source_slug, rendered.shard, fingerprint: rendered.fingerprint, files: entries) - @changed_urls.concat(rendered.changed_urls) if rendered.files.any? { |file| changed_names.include?(file.name) } + end + end + + def previous_source_of(file_name) + @previous_source_by_file_name ||= @previous.shards_by_source.each_with_object({}) do |(source_slug, shards), source_by_name| + shards.each_value { |stored| stored["files"].each { |file| source_by_name[file["name"]] = source_slug } } + end + @previous_source_by_file_name[file_name] + end + + def urls_in(names) + names.flat_map do |name| + content = @storage.read(name) + content ? Xml.locations(Zlib.gunzip(content.b)) : [] + rescue Zlib::Error + [] end end @@ -189,7 +241,7 @@ def manifest_name def inspect_page(source, page, seen_urls) page.urls.each do |url| - warn(source, url, "is outside #{@configuration.host!}") unless url.start_with?("#{@configuration.host!}/") || url == @configuration.host! + warn(source, url, "is outside #{@configuration.host!}") unless on_host?(url) warn(source, url, "is listed twice") unless seen_urls.add?(url) warn(source, url, "is longer than #{MAX_URL_CHARACTERS} characters") if url.length > MAX_URL_CHARACTERS end @@ -225,23 +277,57 @@ def warn(source, url, message) @warnings << Warning.new(source: source.is_a?(Source) ? source.slug : source, url:, message:) end - # The first build has no "before": submitting a whole site at once is not what IndexNow is for. + def on_host?(url) + url.start_with?("#{@configuration.host!}/") || url == @configuration.host! + end + + # A day may still be the day of the last build somewhere: it changed if it is not older than the date at + # UTC-12, the last time zone to leave a day. A month or a year alone says too little to tell. def changed_since_last_build?(page) - @previous.built_at && page.modified_at && page.modified_at >= @previous.built_at + return false unless page.lastmod + return page.modified_at >= @previous.built_at if page.lastmod.include?("T") + + page.lastmod.length == DAY_LENGTH && page.lastmod >= (@previous.built_at - LAST_TIME_ZONE_OFFSET_SECONDS).utc.to_date.iso8601 end - def submit_changed_urls - key = @configuration.index_now_key - return 0 unless @submit && key + def urls_to_submit + return [] unless @submit && @configuration.index_now_key + + (@previous.unsubmitted + @changed_urls).uniq.select { |url| on_host?(url) } + end + + # Returns how many URLs the endpoint took. The others, refused or unreachable, stay in the manifest for the + # next build: the sitemaps are published, so a refusal never fails the build. + def submit(urls) + return 0 if urls.empty? + + replies = IndexNow.new(host: @configuration.host!, key: @configuration.index_now_key, + key_location: @configuration.index_now_key_location).submit(urls) + refused = replies.reject(&:accepted?) + refused.group_by(&:status).each do |status, replies_with_status| + count = replies_with_status.sum { |reply| reply.urls.size } + answer = status.is_a?(Integer) ? "answered #{status}" : "could not be reached (#{status})" + warn("indexnow", IndexNow::ENDPOINT, "#{answer} for #{count} URLs, sent again with the next build") + end + @manifest.unsubmitted.replace(refused.flat_map(&:urls).last(MAX_UNSUBMITTED_URLS)) + upload(manifest_name, @manifest.to_json) + replies.select(&:accepted?).sum { |reply| reply.urls.size } + end - upload("#{key}.txt", key) unless @configuration.index_now_key_location - return 0 if @changed_urls.empty? + # IndexNow reads the key at the root of the site unless told where it is, and only on the site's host. + def inspect_index_now_key + key = @configuration.index_now_key + location = @configuration.index_now_key_location + return unless key - IndexNow.new(host: @configuration.host!, key:, key_location: @configuration.index_now_key_location).submit(@changed_urls) - rescue *IndexNow::NETWORK_ERRORS => exception - # The sitemaps are published: a search engine that cannot be reached must not fail the build. - @warnings << Warning.new(source: "indexnow", url: IndexNow::ENDPOINT.to_s, message: "could not be reached (#{exception.class})") - 0 + if location + warn("indexnow", location, "is not on #{@configuration.host!}: IndexNow only reads a key there") unless on_host?(location) + elsif @configuration.sitemaps_url == @configuration.host! + upload("#{key}.txt", key) + else + warn("indexnow", "#{@configuration.host!}/#{key}.txt", "must hold the key: sitemaps go to #{@configuration.sitemaps_url}, " \ + "so put #{key}.txt in public/ and set index_now_key_location to this URL") + end end end end diff --git a/lib/mapitaine/index_now.rb b/lib/mapitaine/index_now.rb index faba80e..f61127c 100644 --- a/lib/mapitaine/index_now.rb +++ b/lib/mapitaine/index_now.rb @@ -15,12 +15,20 @@ def initialize(host:, key:, key_location: nil, transport: Net::HTTP.method(:post @transport = transport end - # Returns how many URLs the endpoint took. + # status: the HTTP status, or the class of the network error. + Reply = Data.define(:urls, :status) do + def accepted? + status.is_a?(Integer) && status.between?(200, 299) + end + end + + # One reply per request of up to 10,000 URLs. def submit(urls) - urls.each_slice(MAX_URLS_PER_REQUEST).sum do |batch| + urls.each_slice(MAX_URLS_PER_REQUEST).map do |batch| body = {host: @host, key: @key, keyLocation: @key_location, urlList: batch}.compact - response = @transport.call(ENDPOINT, JSON.generate(body), HEADERS) - response.code.to_i.between?(200, 299) ? batch.size : 0 + Reply.new(urls: batch, status: @transport.call(ENDPOINT, JSON.generate(body), HEADERS).code.to_i) + rescue *NETWORK_ERRORS => exception + Reply.new(urls: batch, status: exception.class.name) end end end diff --git a/lib/mapitaine/manifest.rb b/lib/mapitaine/manifest.rb index da45644..9afca93 100644 --- a/lib/mapitaine/manifest.rb +++ b/lib/mapitaine/manifest.rb @@ -2,9 +2,10 @@ module Mapitaine # What the last build wrote: for each shard of each source, its fingerprint and its files. pending lists the - # files a build started to publish before it could record them, so that the next one can remove them. + # files a build started to publish before it could record them, so that the next one can remove them; + # unsubmitted the URLs IndexNow did not take, for the next one to send again. class Manifest - attr_reader :built_at, :rebuilt_at, :digest, :shards_by_source, :pending + attr_reader :built_at, :rebuilt_at, :digest, :shards_by_source, :pending, :unsubmitted def self.read(storage, name) json = storage.read(name) @@ -12,7 +13,7 @@ def self.read(storage, name) stored = JSON.parse(json) new(digest: stored["digest"], built_at: time(stored["built_at"]), rebuilt_at: time(stored["rebuilt_at"]), - shards_by_source: stored["sources"] || {}, pending: stored["pending"] || []) + shards_by_source: stored["sources"] || {}, pending: stored["pending"] || [], unsubmitted: stored["unsubmitted"] || []) rescue JSON::ParserError, ArgumentError, TypeError # A manifest we cannot read is a manifest we do not have: the build starts over and replaces it. new @@ -22,12 +23,13 @@ def self.time(text) Time.iso8601(text) if text end - def initialize(digest: nil, built_at: nil, rebuilt_at: nil, shards_by_source: {}, pending: []) + def initialize(digest: nil, built_at: nil, rebuilt_at: nil, shards_by_source: {}, pending: [], unsubmitted: []) @digest = digest @built_at = built_at @rebuilt_at = rebuilt_at @shards_by_source = shards_by_source @pending = pending + @unsubmitted = unsubmitted end def shard(source, key) @@ -50,7 +52,7 @@ def to_json(*) # Whole seconds, like the lastmods compared to built_at: a change in the second of a build is sent again # rather than never. JSON.pretty_generate("digest" => digest, "built_at" => built_at&.utc&.iso8601, - "rebuilt_at" => rebuilt_at&.utc&.iso8601, "pending" => pending, "sources" => shards_by_source) + "rebuilt_at" => rebuilt_at&.utc&.iso8601, "pending" => pending, "unsubmitted" => unsubmitted, "sources" => shards_by_source) end end end diff --git a/lib/mapitaine/storage/local.rb b/lib/mapitaine/storage/local.rb index 29f4051..80ef510 100644 --- a/lib/mapitaine/storage/local.rb +++ b/lib/mapitaine/storage/local.rb @@ -13,7 +13,7 @@ def initialize(directory) def read(name) path = File.join(@directory, name) - File.read(path) if File.exist?(path) + File.binread(path) if File.exist?(path) end # Copied beside the target, then renamed: a crawler never downloads half a file. diff --git a/lib/mapitaine/xml.rb b/lib/mapitaine/xml.rb index 0dc52d3..cdd601e 100644 --- a/lib/mapitaine/xml.rb +++ b/lib/mapitaine/xml.rb @@ -15,6 +15,9 @@ module Xml W3C_DATE = /\A(\d{4})(?:-(\d{2})(?:-(\d{2}))?)?\z/ W3C_TIME = /\A(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2})(?::(\d{2})(?:\.\d+)?)?(Z|[+-]\d{2}:\d{2})\z/ + LOCATION = %r{([^<]*)} + UNESCAPED_BY_ENTITY = {"&" => "&", "<" => "<", ">" => ">", """ => '"', "'" => "'"}.freeze + module_function # One per language. Each lists every language of the page, itself included: search engines drop @@ -66,6 +69,11 @@ def hreflang(locale) locale.to_s.tr("_", "-") end + # The of each of a sitemap file. + def locations(xml) + xml.scan(LOCATION).map { |(location)| location.gsub(/&(?:amp|lt|gt|quot|apos);/, UNESCAPED_BY_ENTITY) } + end + def escape(text) text.to_s.encode(xml: :text).gsub('"', """) end diff --git a/test/build_test.rb b/test/build_test.rb index 73ccc9d..462ba8e 100644 --- a/test/build_test.rb +++ b/test/build_test.rb @@ -163,7 +163,7 @@ def test_a_file_dated_by_an_older_manifest_is_compared_by_time build manifest = JSON.parse(storage.content_by_name["sitemap-manifest.json"]) manifest["sources"]["post"]["0"]["files"][0]["lastmod"] = "2026-01-15T10:00:00+09:00" - storage.content_by_name["sitemap-manifest.json"] = manifest.to_json + storage.content_by_name["sitemap-manifest.json"] = JSON.generate(manifest) Post.where(id: 1).update_all(title: "Edited", updated_at: Time.utc(2026, 1, 15, 6)) build @@ -199,7 +199,7 @@ def test_a_full_build_comes_every_week_by_itself build manifest = JSON.parse(storage.content_by_name["sitemap-manifest.json"]) manifest["rebuilt_at"] = (Time.now - 8 * 86_400).utc.iso8601 - storage.content_by_name["sitemap-manifest.json"] = manifest.to_json + storage.content_by_name["sitemap-manifest.json"] = JSON.generate(manifest) built = build @@ -207,7 +207,7 @@ def test_a_full_build_comes_every_week_by_itself assert_equal %w[sitemap-post-00000.xml.gz sitemap-page.xml.gz], built.written refute_predicate build, :full Mapitaine.configuration.rebuild_every = nil - storage.content_by_name["sitemap-manifest.json"] = manifest.to_json + storage.content_by_name["sitemap-manifest.json"] = JSON.generate(manifest) assert_empty build.written end diff --git a/test/index_now_test.rb b/test/index_now_test.rb index bab81e5..80c9fae 100644 --- a/test/index_now_test.rb +++ b/test/index_now_test.rb @@ -64,13 +64,116 @@ def test_a_key_hosted_elsewhere_is_named_and_not_published assert_equal "https://example.com/keys/indexnow.txt", @requests.dig(0, 1, "keyLocation") end - def test_a_refusal_counts_no_url + def test_a_refusal_is_reported_and_sent_again_with_the_next_build create_post(1) + create_post(2) build Post.find(1).update!(title: "Edited", updated_at: Time.now) - self.response_code = "403" + self.response_code = "429" + refused = build + + assert_equal 0, refused.submitted + assert_equal ["indexnow: https://api.indexnow.org/indexnow answered 429 for 2 URLs, sent again with the next build"], + refused.warnings.map(&:to_s) + Post.find(2).update!(title: "Edited", updated_at: Time.now) + self.response_code = "202" + + assert_equal 4, build.submitted + assert_equal %w[https://example.com/en/posts/1 https://example.com/fr/posts/1 https://example.com/en/posts/2 + https://example.com/fr/posts/2], @requests.last.dig(1, "urlList") + end + + def test_pages_that_appear_or_go_away_are_submitted_whatever_their_lastmod + create_post(1) + create_post(2) + create_post(12) + build + Post.find(2).update!(published: false, updated_at: LONG_AGO) + Post.where(id: 12).delete_all + create_post(3) + + assert_equal 6, build.submitted + assert_equal %w[https://example.com/en/posts/3 https://example.com/fr/posts/3 https://example.com/en/posts/2 + https://example.com/fr/posts/2 https://example.com/en/posts/12 https://example.com/fr/posts/12], @requests.last.dig(1, "urlList") + end + + def test_a_new_configuration_or_new_ranges_send_only_what_changed + create_post(5) + create_post(15) + build + Mapitaine.configuration.locales = %i[en fr de] assert_equal 0, build.submitted + Mapitaine.configuration.sources = [Class.new(PostSitemap) { shard_size 20 }, PageSitemap] + Post.find(5).update!(title: "Edited", updated_at: Time.now) + + build + + assert_equal %w[https://example.com/en/posts/5 https://example.com/fr/posts/5], @requests.last.dig(1, "urlList") + end + + def test_a_build_that_dies_while_submitting_leaves_its_urls_for_the_next + create_post(1) + build + Post.find(1).update!(title: "Edited", updated_at: Time.now) + self.network_error = Net::HTTPBadResponse + + assert_raises(Net::HTTPBadResponse) { build } + self.network_error = nil + + assert_equal 2, build.submitted + end + + def test_a_year_alone_is_not_a_change + create_post(1) + create_post(2) + yearly = Class.new(PostSitemap) { lastmod { "2026" } } + Mapitaine.configuration.sources = [yearly] + build + Post.where(id: 2).delete_all + build + + assert_equal %w[https://example.com/en/posts/2 https://example.com/fr/posts/2], @requests.last.dig(1, "urlList") + end + + def test_check_reports_a_key_the_site_does_not_serve + Mapitaine.configuration.sitemaps_url = "https://cdn.example.com/sitemaps" + + assert_equal 1, Mapitaine.check.size + end + + def test_a_date_of_the_day_of_the_last_build_is_a_change + create_post(1) + dated = Class.new(PostSitemap) { lastmod { |post| post.reviewed_at&.to_date || Date.new(2026, 1, 1) } } + Mapitaine.configuration.sources = [dated] + build + Post.find(1).update!(reviewed_at: Time.now) + + assert_equal 2, build.submitted + end + + def test_only_urls_of_the_host_are_submitted + Mapitaine.configuration.sources = [Class.new(PostSitemap) { url { |post, locale| (locale == :fr) ? "https://example.fr/posts/#{post.id}" : "/en/posts/#{post.id}" } }] + create_post(1) + build + Post.find(1).update!(title: "Edited", updated_at: Time.now) + build + + assert_equal %w[https://example.com/en/posts/1], @requests.last.dig(1, "urlList") + end + + def test_a_key_the_site_does_not_serve_is_reported + Mapitaine.configuration.sitemaps_url = "https://cdn.example.com/sitemaps" + create_post(1) + built = build + + refute storage.content_by_name.key?("0123456789abcdef.txt") + assert_equal ["indexnow: https://example.com/0123456789abcdef.txt must hold the key: sitemaps go to https://cdn.example.com/sitemaps, " \ + "so put 0123456789abcdef.txt in public/ and set index_now_key_location to this URL"], built.warnings.map(&:to_s) + Mapitaine.configuration.index_now_key_location = "https://cdn.example.com/sitemaps/0123456789abcdef.txt" + + assert_equal ["indexnow: https://cdn.example.com/sitemaps/0123456789abcdef.txt is not on https://example.com: IndexNow only reads a key there"], + build.warnings.map(&:to_s) end def test_an_unreachable_endpoint_is_a_warning_and_the_sitemaps_are_published @@ -80,7 +183,8 @@ def test_an_unreachable_endpoint_is_a_warning_and_the_sitemaps_are_published self.network_error = Errno::ECONNREFUSED built = build - assert_equal ["indexnow: https://api.indexnow.org/indexnow could not be reached (Errno::ECONNREFUSED)"], built.warnings.map(&:to_s) + assert_equal ["indexnow: https://api.indexnow.org/indexnow could not be reached (Errno::ECONNREFUSED) for 2 URLs, sent again with the next build"], + built.warnings.map(&:to_s) assert_equal %w[sitemap-post-00000.xml.gz], built.written end @@ -92,8 +196,26 @@ def test_urls_go_in_batches_of_ten_thousand end index_now = Mapitaine::IndexNow.new(host: "https://example.com", key: "key", transport:) - assert_equal 10_001, index_now.submit(Array.new(10_001) { |number| "https://example.com/#{number}" }) + replies = index_now.submit(Array.new(10_001) { |number| "https://example.com/#{number}" }) + assert_equal [10_000, 1], batches + assert_equal [10_000, 1], replies.map { |reply| reply.urls.size } + assert replies.all?(&:accepted?) + end + + def test_each_batch_gets_its_own_reply + statuses = ["202", Errno::ECONNRESET] + transport = lambda do |_uri, _body, _headers| + status = statuses.shift + raise status if status.is_a?(Class) + + Response.new(code: status) + end + replies = Mapitaine::IndexNow.new(host: "https://example.com", key: "key", transport:) + .submit(Array.new(10_001) { |number| "https://example.com/#{number}" }) + + assert_equal [202, "Errno::ECONNRESET"], replies.map(&:status) + assert_equal [true, false], replies.map(&:accepted?) end def test_check_submits_nothing