Repository navigation
Show related posts found by embeddings in the post navigation - #523
Merged
Merged
Conversation
Pull the Ollama check of backfill_suggestions into each_site_with_model, the default of the model setting into Ollama's setting/2, and the transaction around publish_stored/3 that Saving and Backfill both open into Versions.publish_committed/3, so that the embeddings can hook into them. No change in behaviour. Refs #522 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a post is published, EmbeddingGemma 2 (OLLAMA_EMBEDDING_MODEL) reads its published version in the background through Ollama's embed API: the clustering prompt with title and description, then the first 4000 characters of its plain text. The L2-normalised vector is stored as 32-bit floats with the model's name on the post version, and storing it for a post already live counts the site as changed. backfill_embeddings reads the published versions without a vector of the configured model, publishing nothing again; the release backfills read inline, since eval runs no background tasks. This is what the related posts are found by. Refs #522 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RelatedPosts ranks a site's posts by the cosine similarity of their published versions' embeddings plus a small bonus for rare shared tags, ties going to the post closer in date. The export ranks every pair once before its render jobs start, the preview only the post it shows. Both themes list up to three related posts, with their publish year, above the newer and older post, which never count among them, and every link in the post navigation now names a short post by its meta title, with the emoji in a column of its own. Closes #522 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR 0036 records why related posts come from embeddings made at publish and compared at export, CONTEXT.md defines the related post, and the README names OLLAMA_EMBEDDING_MODEL and when to run backfill_embeddings. Refs #522 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Readers at the end of a post only got the newer and the older post, and tags are too sparse to link related ones. Posts are now read by a self-hosted embedding model at publish, and the export lists the three closest posts first in the post navigation. Closes #522
OLLAMA_EMBEDDING_MODEL, defaultembeddinggemma-2:270m) reads each published version in the background; the vector is stored on the version (migration: two nullable columns onpost_versions)backfill_embeddingsreads existing posts without republishing them; ADR 0036 records the decisionHow tested
mix compile --warnings-as-errors,mix format --check-formatted,mix credo --strict,mix test: all green (2052 passed, 2 skipped)Risk
Medium: a migration and a new background call to Ollama on every publish. Without Ollama nothing is read and posts simply show no related posts. After deploying, run
bin/feather eval "Feather.Release.backfill_embeddings()"once. Each publish now also loads the embedding model, which swaps it with the suggestion model on an instance that holds one model at a time. Roll back by reverting; the two columns are nullable and unused elsewhere.🤖 Generated with Claude Code