Skip to content

Show related posts found by embeddings in the post navigation - #523

Merged
shostakovich merged 4 commits into
mainfrom
related-posts
Nov 7, 2026
Merged

shostakovich merged 4 commits into
mainfrom
related-posts

Conversation

@shostakovich

Copy link
Copy Markdown
Contributor

Why

Readers at the end of a post only got the newer and the older post, and tags are too sparse to link related ones. Posts are now read by a self-hosted embedding model at publish, and the export lists the three closest posts first in the post navigation. Closes #522

  • EmbeddingGemma 2 (OLLAMA_EMBEDDING_MODEL, default embeddinggemma-2:270m) reads each published version in the background; the vector is stored on the version (migration: two nullable columns on post_versions)
  • Export and preview score cosine similarity plus a small bonus for shared, rarely used tags; the newer and older post are left out, a quiet year follows each link
  • Short posts are named by their meta title throughout the post navigation; posts without text take no part
  • Release task backfill_embeddings reads existing posts without republishing them; ADR 0036 records the decision

How tested

  • mix compile --warnings-as-errors, mix format --check-formatted, mix credo --strict, mix test: all green (2052 passed, 2 skipped)
  • Chosen against six other variants on the published posts of two live sites (450 posts, about 0.2 s each); exported both themes locally from a copy and checked short and long posts on desktop and phone
  • Two art-director reviews over three rounds, 9/10 in both themes for short and long posts

Risk

Medium: a migration and a new background call to Ollama on every publish. Without Ollama nothing is read and posts simply show no related posts. After deploying, run bin/feather eval "Feather.Release.backfill_embeddings()" once. Each publish now also loads the embedding model, which swaps it with the suggestion model on an instance that holds one model at a time. Roll back by reverting; the two columns are nullable and unused elsewhere.

🤖 Generated with Claude Code

shostakovich and others added 4 commits November 7, 2026 15:50
Pull the Ollama check of backfill_suggestions into each_site_with_model,
the default of the model setting into Ollama's setting/2, and the
transaction around publish_stored/3 that Saving and Backfill both open
into Versions.publish_committed/3, so that the embeddings can hook into
them. No change in behaviour.

Refs #522

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a post is published, EmbeddingGemma 2 (OLLAMA_EMBEDDING_MODEL)
reads its published version in the background through Ollama's embed
API: the clustering prompt with title and description, then the first
4000 characters of its plain text. The L2-normalised vector is stored as
32-bit floats with the model's name on the post version, and storing it
for a post already live counts the site as changed. backfill_embeddings
reads the published versions without a vector of the configured model,
publishing nothing again; the release backfills read inline, since eval
runs no background tasks. This is what the related posts are found by.

Refs #522

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RelatedPosts ranks a site's posts by the cosine similarity of their
published versions' embeddings plus a small bonus for rare shared tags,
ties going to the post closer in date. The export ranks every pair once
before its render jobs start, the preview only the post it shows. Both
themes list up to three related posts, with their publish year, above
the newer and older post, which never count among them, and every link
in the post navigation now names a short post by its meta title, with
the emoji in a column of its own.

Closes #522

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR 0036 records why related posts come from embeddings made at publish
and compared at export, CONTEXT.md defines the related post, and the
README names OLLAMA_EMBEDDING_MODEL and when to run
backfill_embeddings.

Refs #522

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shostakovich
shostakovich merged commit 2a303dd into main Nov 7, 2026
6 checks passed
@shostakovich
shostakovich deleted the related-posts branch November 7, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Related posts from embeddings

1 participant