Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,12 +147,20 @@ _Avoid_: Summary, Excerpt, Teaser, Bio

**Short Post** · _de:_ Kurzbeitrag:
A post without a title. The site and the CMS's post list show it untitled; its meta title names it in the page title, in
link previews, in the feeds and in its Markdown alternate. At its first publish, when it has no slug, the self-hosted
link previews, in the feeds, in its Markdown alternate and in the post navigation of other posts. At its first publish, when it has no slug, the self-hosted
model writes a slug and, unless a member typed one, the meta title in the site's language; afterwards the slug stays.
While its text is shorter than 300 characters the form hides the title field. A review without a title is a short post
too.
_Avoid_: Note, Untitled Post, Micropost

**Related Post** · _de:_ ähnlicher Beitrag:
One of up to three other posts of the same site, closest in content, that a post lists first in its post navigation,
above its newer and older post, which never count among them. The self-hosted embedding model reads each post's
published version after it is published, unless it has no text; the static export and the preview compare what it read
and give tags a small bonus while few posts carry them. A post the model has not read, a draft included, shows none and
is shown by none. Members neither choose nor set them.
_Avoid_: Similar Post, Recommendation, See Also

**Markdown Alternate** · _de:_ Markdown-Fassung:
The home page, a post, page or project as Markdown, without the theme's markup: front matter with its title (a short
post's meta title, as in the page title), canonical URL, publish date (posts) and tags, then the title and the content;
Expand Down
8 changes: 8 additions & 0 deletions DECISIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -588,3 +588,11 @@ Accepted, 2026-10-25
- **Context:** Every deploy exports into a fresh directory ([0019](#0019-a-site-has-one-deployment-target-addressed-by-its-domain)), so every file, images and fonts included, carries the time of the export. FTP has no hashes, so `rclone sync` compares size and modification time and uploaded the whole site each time: about five minutes for rocu.de, and every file online got a new `Last-Modified`.
- **Decision:** After a successful sync the deploy stores a manifest for its target, a JSON file at `<storage_root>/deploy_manifests/<target id>.json` (outside `static_site/`, which the boot clears; no migration) holding each file's path, SHA-256 and modification time in seconds. The next deploy, after the export and precompression and before the sync, gives every file whose hash the manifest records the recorded time back (`Feather.Publishing.DeployManifest`), so rclone skips it. The manifest is written atomically, a failed sync keeps the previous one, and a missing or corrupt manifest counts as empty. Rejected: `rclone --size-only` (misses edits that keep the size) and modification times derived from the content hash (a meaningless `Last-Modified` and odd browser caching).
- **Consequences:** Unchanged files keep their `Last-Modified` and ETag, so browsers revalidate them cheaply. rclone still compares with the remote, so a manifest that does not match the server (a file changed there by hand, a restored backup of the CMS, a new FTP account) only costs uploads, and deleting the manifest makes the next deploy upload everything. Hashing the export adds a read of every file to each deploy. Deleting a deployment target or its site deletes its manifest.

## 0036. Related posts come from embeddings made at publish and compared at export

Accepted, 2026-11-07 (issue #522)

- **Context:** A reader at the end of a post only gets the newer and the older post, which rarely share its topic. Tags could link related posts, but most sites tag sporadically. The self-hosted Ollama instance ([0026](#0026-suggestions-come-from-a-self-hosted-model-made-in-the-cms-never-at-export)) also serves embedding models.
- **Decision:** After a post is published, in the background, EmbeddingGemma 2 (`OLLAMA_EMBEDDING_MODEL`, by default `embeddinggemma-2:270m`) reads its published version as the clustering prompt with title and description, then the first 4,000 characters of its plain text; a version without plain text is not read, as its link would show no more than a date. The L2-normalised vector is stored as 32-bit floats with the model's name on the version. The static export and the preview compare the published versions' vectors of the due posts in `SiteData`: cosine similarity plus 0.06 per shared tag, at most 0.12, where a tag counts only while at most three posts or a fifth of the due posts carry it; the best three that are not the post's newer or older post win, ties go to the post closer in date. Both themes list them, with their publish year, first in the post navigation above the newer and older post, and every link there names a short post by its meta title. Only vectors of the configured model are compared, and a release task reads the published versions without one, publishing nothing again. Chosen against six other variants on the posts of two live sites. Rejected: storing the lists (every publish, unpublish, tag change or due post would recompute the whole site), tags alone (most posts have none) and calling the model at export (exports would wait on it and stop being reproducible).
- **Consequences:** Without the model, or before it has read a post, that post has no related posts and is listed by none; a deploy started right after publishing can miss the new reading. Each publish asks the suggestion model and then the embedding model, which on an instance holding one model at a time means a swap, and a reading that fails waits for `backfill_embeddings`. Changing the model leaves every post unread until the backfill runs. The export compares every pair once before its render jobs start and hands them only the ids of each post's related posts: about 1.1 seconds on one core for 313 posts (measured 2026-11-07 on synthetic 768-dimension vectors), growing with the square of the post count; the preview compares only the post it shows, about 12 ms. Publishing a post changes the pages of the posts that now list it, which the deploy manifest ([0035](#0035-deploys-upload-only-changed-files-by-a-manifest-of-the-last-deploy)) uploads. Related posts appear only on the HTML page, not in feeds, Markdown alternates or `llms.txt`.
7 changes: 6 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,10 @@ In a release the same is available through `bin/feather eval`, see `Feather.Rele
(`migrate/0`, `create_user/2`). `bin/server` migrates and starts the app.
`Feather.Release.backfill_suggestions/0` fills in the model's alt texts, descriptions and short
post titles for content from before `OLLAMA_URL` was set; deploy the sites by hand afterwards.
`Feather.Release.backfill_embeddings/0` reads the published posts without an embedding of the
configured model, which their related posts need; it publishes nothing again. Run it once after
upgrading to the release that brings related posts, for posts published before `OLLAMA_URL` was
set and after changing `OLLAMA_EMBEDDING_MODEL`, then deploy.

### Environment variables

Expand All @@ -76,8 +80,9 @@ post titles for content from before `OLLAMA_URL` was set; deploy the sites by ha
| `CONFIG_ENCRYPTION_KEY` | 32 random bytes, base64; encrypts deployment credentials, archive.org keys and the private keys, mentions, Timeline posts, members' replies and outgoing deliveries of the sites' fediverse accounts (prod, required) | fixed dev/test keys |
| `UNSPLASH_ACCESS_KEY` | Unsplash API key (optional) | |
| `HARDCOVER_TOKEN` | Hardcover API key for the book search (optional; without it books are searched on OpenLibrary) | |
| `OLLAMA_URL` | Base URL of an Ollama instance (e.g. `http://ollama:11434`), which suggests alt texts, descriptions and titles of short posts (optional; without it no suggestions are made) | |
| `OLLAMA_URL` | Base URL of an Ollama instance (e.g. `http://ollama:11434`), which suggests alt texts, descriptions and titles of short posts and reads posts for their related posts (optional; without it no suggestions are made and posts show no related posts) | |
| `OLLAMA_MODEL` | The Ollama model that makes the suggestions; it must read images | `gemma4:26b-a4b-it-qat` |
| `OLLAMA_EMBEDDING_MODEL` | The Ollama model that reads published posts for their related posts; after changing it, run `Feather.Release.backfill_embeddings()` | `embeddinggemma-2:270m` |
| `FEDIVERSE_PROXY` | URL of an HTTP proxy (e.g. `http://proxy:3128`) that all requests to fediverse servers tunnel through, so they do not learn where the CMS runs (optional); the CMS still checks every address, but the proxy itself must reach only public addresses. Even with the proxy the CMS resolves the servers' names itself, so its DNS should go through a resolver that does not reveal where it runs | |
| `IMAGE_CLEANUP` | `false` disables the daily deletion of orphaned images | enabled |
| `SMTP_ADDRESS`, `SMTP_PORT`, `SMTP_USERNAME`, `SMTP_PASSWORD` | Outgoing mail (prod) | `localhost`, `587` |
Expand Down
4 changes: 4 additions & 0 deletions config/runtime.exs
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ if config_env() != :test do
config :feather, :ollama_model, ollama_model
end

if ollama_embedding_model = System.get_env("OLLAMA_EMBEDDING_MODEL") do
config :feather, :ollama_embedding_model, ollama_embedding_model
end

case System.get_env("FEDIVERSE_PROXY", "") do
"" ->
:ok
Expand Down
2 changes: 2 additions & 0 deletions config/test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ config :feather,
backfill_image_variants: false,
image_variant_sync: :inline,
alt_text_suggestions: :inline,
# Publishing reads no post for its related posts unless a test asks for it (:inline).
post_embeddings: false,
fediverse_deliveries: :inline,
fediverse_inbox: :inline,
# Deploys tell IndexNow and archive.org nothing unless a test asks for it.
Expand Down
3 changes: 2 additions & 1 deletion lib/feather/content.ex
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ defmodule Feather.Content do

alias Feather.Accounts.Scope
alias Feather.Books
alias Feather.Content.{Backfill, Blocks, Page, Pages, Post, Posts, Project, Projects}
alias Feather.Content.{Backfill, Blocks, Embedding, Page, Pages, Post, Posts, Project, Projects}
alias Feather.Content.{ProseMirror, Saving, Slug, Suggestions, Sync, Tags, Versions}
alias Feather.Repo
alias Feather.Sites.Site
Expand Down Expand Up @@ -92,6 +92,7 @@ defmodule Feather.Content do
defdelegate publish(scope, record), to: Saving
defdelegate unpublish(scope, record), to: Versions
defdelegate backfill_suggestions(scope), to: Backfill
defdelegate backfill_embeddings(scope), to: Embedding, as: :backfill
defdelegate suggest_description(scope, record), to: Suggestions
defdelegate put_suggested_description(scope, record, description), to: Suggestions
defdelegate published_slug?(scope, schema, slug, except_id \\ nil), to: Versions
Expand Down
10 changes: 4 additions & 6 deletions lib/feather/content/backfill.ex
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,10 @@ defmodule Feather.Content.Backfill do
if suggestions == %{} do
:unchanged
else
Repo.transact(fn ->
Versions.publish_stored(scope, record,
state: :published,
prepare: &Suggestions.put_suggestions(&1, suggestions)
)
end)
Versions.publish_committed(scope, record,
state: :published,
prepare: &Suggestions.put_suggestions(&1, suggestions)
)
end
end
end
152 changes: 152 additions & 0 deletions lib/feather/content/embedding.ex
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
defmodule Feather.Content.Embedding do
@moduledoc """
What the embedding model of `Feather.Ollama` reads in a published post
(ADR 0036), which its related posts are found by
(`Feather.Content.RelatedPosts`).

The model reads the clustering prompt with the title (a short post's
meta title, `none` without either) and the description, then the first
4000 characters of the plain text.

Publishing a post reads its new version in the background (`start/1`);
`backfill/1` reads the published versions read by no or another model.
The vector, which Ollama returns L2-normalised, is stored as
little-endian 32-bit floats on the version, with the model's name, so that only vectors of the configured
model are compared (`store/3`). Without the model, and for a version
without plain text, the version stays unread.
"""

import Ecto.Query, warn: false

alias Feather.Accounts.Scope
alias Feather.Content.{Blocks, Post, PostVersion}
alias Feather.{Ollama, Publishing, Repo, Sites}
alias Feather.Sites.Site
alias Feather.Validations

require Logger

@text_length 4000

@doc """
Reads the published version of a post under `Feather.TaskSupervisor`,
or right away with `config :feather, :post_embeddings, :inline`; `false`
reads nothing (tests that do not ask for it). Does nothing without
`OLLAMA_URL` or when the version is read by the configured model.
"""
@spec start(Post.t()) :: :ok
def start(%Post{published_version: %PostVersion{} = version}) do
if Ollama.configured?() and not read?(version),
do: start_in(Application.get_env(:feather, :post_embeddings, :background), version),
else: :ok
end

def start(%Post{}), do: :ok

defp start_in(:background, version) do
Feather.TaskSupervisor
|> Task.Supervisor.start_child(fn -> embed(version) end)
|> log_start_failure(version)
end

defp start_in(:inline, version) do
embed(version)
:ok
end

defp start_in(false, _version), do: :ok

defp log_start_failure({:ok, _pid}, _version), do: :ok

defp log_start_failure({:error, reason}, version) do
Logger.error("Post version #{version.id} not read: #{inspect(reason)}")
end

defp read?(%PostVersion{embedding: embedding, embedding_model: model}),
do: is_binary(embedding) and model == Ollama.embedding_model()

defp embed(%PostVersion{} = version) do
with {:ok, input} <- input(version),
{:ok, vector} <- Ollama.embedding(input) do
store(version, vector, Ollama.embedding_model())
else
{:error, _reason} -> :error
end
end

@doc """
Stores the vector, as Ollama returns it L2-normalised, as the version's
embedding by `model`. When the version is the published version
of a post the live site already shows, the site counts as changed since
its last deploy, as the related posts the export shows may change.
"""
@spec store(PostVersion.t(), [number()], String.t()) :: :ok
def store(%PostVersion{id: id} = version, vector, model) do
Repo.update_all(from(v in PostVersion, where: v.id == ^id),
set: [embedding: encode(vector), embedding_model: model]
)

touch_site_if_live(version)
end

defp touch_site_if_live(%PostVersion{id: id} = version) do
case Repo.one(from p in Post, where: p.published_version_id == ^id, select: p.site_id) do
nil -> :ok
site_id -> touch_site_if_live(Scope.for_site(%Site{id: site_id}), version)
end
end

defp touch_site_if_live(scope, version) do
target = Publishing.get_target(scope)

if target && Publishing.live?(target, version),
do: Sites.touch_site(scope),
else: :ok
end

defp input(%PostVersion{} = version) do
case Blocks.model_text(version.content, @text_length) do
"" ->
{:error, :no_text}

text ->
{:ok, "task: clustering | query: #{title(version)}. #{version.description}\n\n#{text}"}
end
end

defp title(version) do
cond do
not Validations.blank?(version.title) -> version.title
not Validations.blank?(version.meta_title) -> version.meta_title
true -> "none"
end
end

@doc """
Reads the published versions of the scope's posts that no vector of the
configured model was stored for, one after the other. Publishes nothing
again. Returns how many the model read.
"""
@spec backfill(Scope.t()) :: non_neg_integer()
def backfill(%Scope{site: %Site{id: site_id}}) do
model = Ollama.embedding_model()

from(v in PostVersion,
join: p in Post,
on: p.published_version_id == v.id,
where: p.site_id == ^site_id,
where: is_nil(v.embedding_model) or v.embedding_model != ^model,
order_by: p.inserted_at
)
|> Repo.all()
|> Enum.count(&(embed(&1) == :ok))
end

@doc "A vector as stored: little-endian 32-bit floats."
@spec encode([number()]) :: binary()
def encode(vector), do: for(x <- vector, into: <<>>, do: <<x::float-32-little>>)

@doc "A stored vector as a list of floats."
@spec decode(binary()) :: [float()]
def decode(binary), do: for(<<x::float-32-little <- binary>>, do: x)
end
5 changes: 5 additions & 0 deletions lib/feather/content/post_version.ex
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
defmodule Feather.Content.PostVersion do
@moduledoc """
A published state of a post, see `Feather.Content.publish/2`.

`embedding` and `embedding_model` hold what the embedding model read in
it (`Feather.Content.Embedding`); they are not copied from the post.
"""
use Feather.Schema

Expand Down Expand Up @@ -34,6 +37,8 @@ defmodule Feather.Content.PostVersion do
field :description, :string
field :description_generated, :boolean, default: false
field :meta_title, :string
field :embedding, :binary
field :embedding_model, :string

belongs_to :post, Feather.Content.Post
belongs_to :published_by, Feather.Accounts.User
Expand Down
Loading
Loading