Repository navigation
Sitemaps for sites in many languages, rebuilt only where they changed - #1
Merged
Merged
Conversation
…hanged A source class names the records of one kind of page, their URLs and their languages. A build writes one <url> per language with the hreflang links of all of them, cuts each source into files by ranges of ids, and rewrites only the ranges whose count or latest change moved since the manifest of the last build. Changed URLs go to IndexNow. 30,000 records in 24 languages (720,000 URLs) build in 32 s, a build with nothing new takes 10 ms, and three changed records rewrite 2 of 55 files in 1 s (bench/build.rb, SQLite in memory, Ruby 3.3.6 without YJIT). Assisted-by: Claude Code
An app that already stores files with Active Storage had to give Mapitaine its bucket and credentials a second time. Storage::ActiveStorage takes the name of a service of config/storage.yml and uploads through it, without creating blobs: a service of its own puts the sitemaps in another bucket. Assisted-by: Claude Code
Storage::S3 did what an Active Storage service already does, with a second copy of the bucket and credentials and an SDK to keep up with. It is removed: the sitemaps reach S3, Google Cloud Storage or Azure through a service of config/storage.yml. Assisted-by: Claude Code
Mapitaine had a README and no site, unlike the other gems of the family. docs/ holds one in their layout, served by GitHub Pages: what the gem does on the home page, the guide, and the numbers of bench/build.rb. Its code blocks are highlighted with Rouge, and CI fails when one is not. Assisted-by: Claude Code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A site of 30,000 pages in 24 languages has 720,000 URLs to list. Regenerating all of them on every run is time spent on pages that did not change, and writing the hreflang links by hand is easy to get wrong: each language of a page must list all the others and itself.
This is the first version of Mapitaine. A source class names the records of one kind of page, their URLs and their languages; a build writes the files, the index and a manifest, and the next build rewrites only the files whose records changed.
app/sitemapswithscope,url,locales,lastmod,images; the blocks have the app's route helpers, set to the configured host.<url>per language, each listing every language of the page andx-default.COUNTandMAXquery per source, compared with the manifest of the last build, soupdate_all, imports and deletions are seen too.mapitaine:checkfor duplicate, foreign or future-dated URLs (no HTTP request),build.mapitaine/shard.mapitainenotifications, and the website indocs/(home page, guide, benchmarks) in the layout of the other gems.Tests and benchmark
47 tests, 122 assertions, green on Rails 8.1 and 8.0 with Ruby 3.3.6; RuboCop clean;
rake docs:highlight:checkpasses. The Railtie, the rake tasks and the Active Storage storage were also run in a minimal Rails 8.1 app. Not exercised: a real cloud bucket, a real IndexNow submission, and these workflows, which run here for the first time.bundle exec ruby bench/build.rb(SQLite in memory, Ruby 3.3.6 without YJIT), 30,000 records in 24 languages: