diff --git a/CONTEXT.md b/CONTEXT.md index 603327b4..474b2508 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -147,12 +147,20 @@ _Avoid_: Summary, Excerpt, Teaser, Bio **Short Post** · _de:_ Kurzbeitrag: A post without a title. The site and the CMS's post list show it untitled; its meta title names it in the page title, in -link previews, in the feeds and in its Markdown alternate. At its first publish, when it has no slug, the self-hosted +link previews, in the feeds, in its Markdown alternate and in the post navigation of other posts. At its first publish, when it has no slug, the self-hosted model writes a slug and, unless a member typed one, the meta title in the site's language; afterwards the slug stays. While its text is shorter than 300 characters the form hides the title field. A review without a title is a short post too. _Avoid_: Note, Untitled Post, Micropost +**Related Post** · _de:_ ähnlicher Beitrag: +One of up to three other posts of the same site, closest in content, that a post lists first in its post navigation, +above its newer and older post, which never count among them. The self-hosted embedding model reads each post's +published version after it is published, unless it has no text; the static export and the preview compare what it read +and give tags a small bonus while few posts carry them. A post the model has not read, a draft included, shows none and +is shown by none. Members neither choose nor set them. +_Avoid_: Similar Post, Recommendation, See Also + **Markdown Alternate** · _de:_ Markdown-Fassung: The home page, a post, page or project as Markdown, without the theme's markup: front matter with its title (a short post's meta title, as in the page title), canonical URL, publish date (posts) and tags, then the title and the content; diff --git a/DECISIONS.md b/DECISIONS.md index c34f33ab..3f1f1f75 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -588,3 +588,11 @@ Accepted, 2026-10-25 - **Context:** Every deploy exports into a fresh directory ([0019](#0019-a-site-has-one-deployment-target-addressed-by-its-domain)), so every file, images and fonts included, carries the time of the export. FTP has no hashes, so `rclone sync` compares size and modification time and uploaded the whole site each time: about five minutes for rocu.de, and every file online got a new `Last-Modified`. - **Decision:** After a successful sync the deploy stores a manifest for its target, a JSON file at `/deploy_manifests/.json` (outside `static_site/`, which the boot clears; no migration) holding each file's path, SHA-256 and modification time in seconds. The next deploy, after the export and precompression and before the sync, gives every file whose hash the manifest records the recorded time back (`Feather.Publishing.DeployManifest`), so rclone skips it. The manifest is written atomically, a failed sync keeps the previous one, and a missing or corrupt manifest counts as empty. Rejected: `rclone --size-only` (misses edits that keep the size) and modification times derived from the content hash (a meaningless `Last-Modified` and odd browser caching). - **Consequences:** Unchanged files keep their `Last-Modified` and ETag, so browsers revalidate them cheaply. rclone still compares with the remote, so a manifest that does not match the server (a file changed there by hand, a restored backup of the CMS, a new FTP account) only costs uploads, and deleting the manifest makes the next deploy upload everything. Hashing the export adds a read of every file to each deploy. Deleting a deployment target or its site deletes its manifest. + +## 0036. Related posts come from embeddings made at publish and compared at export + +Accepted, 2026-11-07 (issue #522) + +- **Context:** A reader at the end of a post only gets the newer and the older post, which rarely share its topic. Tags could link related posts, but most sites tag sporadically. The self-hosted Ollama instance ([0026](#0026-suggestions-come-from-a-self-hosted-model-made-in-the-cms-never-at-export)) also serves embedding models. +- **Decision:** After a post is published, in the background, EmbeddingGemma 2 (`OLLAMA_EMBEDDING_MODEL`, by default `embeddinggemma-2:270m`) reads its published version as the clustering prompt with title and description, then the first 4,000 characters of its plain text; a version without plain text is not read, as its link would show no more than a date. The L2-normalised vector is stored as 32-bit floats with the model's name on the version. The static export and the preview compare the published versions' vectors of the due posts in `SiteData`: cosine similarity plus 0.06 per shared tag, at most 0.12, where a tag counts only while at most three posts or a fifth of the due posts carry it; the best three that are not the post's newer or older post win, ties go to the post closer in date. Both themes list them, with their publish year, first in the post navigation above the newer and older post, and every link there names a short post by its meta title. Only vectors of the configured model are compared, and a release task reads the published versions without one, publishing nothing again. Chosen against six other variants on the posts of two live sites. Rejected: storing the lists (every publish, unpublish, tag change or due post would recompute the whole site), tags alone (most posts have none) and calling the model at export (exports would wait on it and stop being reproducible). +- **Consequences:** Without the model, or before it has read a post, that post has no related posts and is listed by none; a deploy started right after publishing can miss the new reading. Each publish asks the suggestion model and then the embedding model, which on an instance holding one model at a time means a swap, and a reading that fails waits for `backfill_embeddings`. Changing the model leaves every post unread until the backfill runs. The export compares every pair once before its render jobs start and hands them only the ids of each post's related posts: about 1.1 seconds on one core for 313 posts (measured 2026-11-07 on synthetic 768-dimension vectors), growing with the square of the post count; the preview compares only the post it shows, about 12 ms. Publishing a post changes the pages of the posts that now list it, which the deploy manifest ([0035](#0035-deploys-upload-only-changed-files-by-a-manifest-of-the-last-deploy)) uploads. Related posts appear only on the HTML page, not in feeds, Markdown alternates or `llms.txt`. diff --git a/README.md b/README.md index 6d3ea61d..ea64570d 100644 --- a/README.md +++ b/README.md @@ -62,6 +62,10 @@ In a release the same is available through `bin/feather eval`, see `Feather.Rele (`migrate/0`, `create_user/2`). `bin/server` migrates and starts the app. `Feather.Release.backfill_suggestions/0` fills in the model's alt texts, descriptions and short post titles for content from before `OLLAMA_URL` was set; deploy the sites by hand afterwards. +`Feather.Release.backfill_embeddings/0` reads the published posts without an embedding of the +configured model, which their related posts need; it publishes nothing again. Run it once after +upgrading to the release that brings related posts, for posts published before `OLLAMA_URL` was +set and after changing `OLLAMA_EMBEDDING_MODEL`, then deploy. ### Environment variables @@ -76,8 +80,9 @@ post titles for content from before `OLLAMA_URL` was set; deploy the sites by ha | `CONFIG_ENCRYPTION_KEY` | 32 random bytes, base64; encrypts deployment credentials, archive.org keys and the private keys, mentions, Timeline posts, members' replies and outgoing deliveries of the sites' fediverse accounts (prod, required) | fixed dev/test keys | | `UNSPLASH_ACCESS_KEY` | Unsplash API key (optional) | | | `HARDCOVER_TOKEN` | Hardcover API key for the book search (optional; without it books are searched on OpenLibrary) | | -| `OLLAMA_URL` | Base URL of an Ollama instance (e.g. `http://ollama:11434`), which suggests alt texts, descriptions and titles of short posts (optional; without it no suggestions are made) | | +| `OLLAMA_URL` | Base URL of an Ollama instance (e.g. `http://ollama:11434`), which suggests alt texts, descriptions and titles of short posts and reads posts for their related posts (optional; without it no suggestions are made and posts show no related posts) | | | `OLLAMA_MODEL` | The Ollama model that makes the suggestions; it must read images | `gemma4:26b-a4b-it-qat` | +| `OLLAMA_EMBEDDING_MODEL` | The Ollama model that reads published posts for their related posts; after changing it, run `Feather.Release.backfill_embeddings()` | `embeddinggemma-2:270m` | | `FEDIVERSE_PROXY` | URL of an HTTP proxy (e.g. `http://proxy:3128`) that all requests to fediverse servers tunnel through, so they do not learn where the CMS runs (optional); the CMS still checks every address, but the proxy itself must reach only public addresses. Even with the proxy the CMS resolves the servers' names itself, so its DNS should go through a resolver that does not reveal where it runs | | | `IMAGE_CLEANUP` | `false` disables the daily deletion of orphaned images | enabled | | `SMTP_ADDRESS`, `SMTP_PORT`, `SMTP_USERNAME`, `SMTP_PASSWORD` | Outgoing mail (prod) | `localhost`, `587` | diff --git a/config/runtime.exs b/config/runtime.exs index fda4aa4b..a71cb3dc 100644 --- a/config/runtime.exs +++ b/config/runtime.exs @@ -25,6 +25,10 @@ if config_env() != :test do config :feather, :ollama_model, ollama_model end + if ollama_embedding_model = System.get_env("OLLAMA_EMBEDDING_MODEL") do + config :feather, :ollama_embedding_model, ollama_embedding_model + end + case System.get_env("FEDIVERSE_PROXY", "") do "" -> :ok diff --git a/config/test.exs b/config/test.exs index 7a9464d6..7b7c73a5 100644 --- a/config/test.exs +++ b/config/test.exs @@ -24,6 +24,8 @@ config :feather, backfill_image_variants: false, image_variant_sync: :inline, alt_text_suggestions: :inline, + # Publishing reads no post for its related posts unless a test asks for it (:inline). + post_embeddings: false, fediverse_deliveries: :inline, fediverse_inbox: :inline, # Deploys tell IndexNow and archive.org nothing unless a test asks for it. diff --git a/lib/feather/content.ex b/lib/feather/content.ex index ac379872..5f9c73f0 100644 --- a/lib/feather/content.ex +++ b/lib/feather/content.ex @@ -35,7 +35,7 @@ defmodule Feather.Content do alias Feather.Accounts.Scope alias Feather.Books - alias Feather.Content.{Backfill, Blocks, Page, Pages, Post, Posts, Project, Projects} + alias Feather.Content.{Backfill, Blocks, Embedding, Page, Pages, Post, Posts, Project, Projects} alias Feather.Content.{ProseMirror, Saving, Slug, Suggestions, Sync, Tags, Versions} alias Feather.Repo alias Feather.Sites.Site @@ -92,6 +92,7 @@ defmodule Feather.Content do defdelegate publish(scope, record), to: Saving defdelegate unpublish(scope, record), to: Versions defdelegate backfill_suggestions(scope), to: Backfill + defdelegate backfill_embeddings(scope), to: Embedding, as: :backfill defdelegate suggest_description(scope, record), to: Suggestions defdelegate put_suggested_description(scope, record, description), to: Suggestions defdelegate published_slug?(scope, schema, slug, except_id \\ nil), to: Versions diff --git a/lib/feather/content/backfill.ex b/lib/feather/content/backfill.ex index 2c1289cc..0e41bc10 100644 --- a/lib/feather/content/backfill.ex +++ b/lib/feather/content/backfill.ex @@ -46,12 +46,10 @@ defmodule Feather.Content.Backfill do if suggestions == %{} do :unchanged else - Repo.transact(fn -> - Versions.publish_stored(scope, record, - state: :published, - prepare: &Suggestions.put_suggestions(&1, suggestions) - ) - end) + Versions.publish_committed(scope, record, + state: :published, + prepare: &Suggestions.put_suggestions(&1, suggestions) + ) end end end diff --git a/lib/feather/content/embedding.ex b/lib/feather/content/embedding.ex new file mode 100644 index 00000000..1dc9edd5 --- /dev/null +++ b/lib/feather/content/embedding.ex @@ -0,0 +1,152 @@ +defmodule Feather.Content.Embedding do + @moduledoc """ + What the embedding model of `Feather.Ollama` reads in a published post + (ADR 0036), which its related posts are found by + (`Feather.Content.RelatedPosts`). + + The model reads the clustering prompt with the title (a short post's + meta title, `none` without either) and the description, then the first + 4000 characters of the plain text. + + Publishing a post reads its new version in the background (`start/1`); + `backfill/1` reads the published versions read by no or another model. + The vector, which Ollama returns L2-normalised, is stored as + little-endian 32-bit floats on the version, with the model's name, so that only vectors of the configured + model are compared (`store/3`). Without the model, and for a version + without plain text, the version stays unread. + """ + + import Ecto.Query, warn: false + + alias Feather.Accounts.Scope + alias Feather.Content.{Blocks, Post, PostVersion} + alias Feather.{Ollama, Publishing, Repo, Sites} + alias Feather.Sites.Site + alias Feather.Validations + + require Logger + + @text_length 4000 + + @doc """ + Reads the published version of a post under `Feather.TaskSupervisor`, + or right away with `config :feather, :post_embeddings, :inline`; `false` + reads nothing (tests that do not ask for it). Does nothing without + `OLLAMA_URL` or when the version is read by the configured model. + """ + @spec start(Post.t()) :: :ok + def start(%Post{published_version: %PostVersion{} = version}) do + if Ollama.configured?() and not read?(version), + do: start_in(Application.get_env(:feather, :post_embeddings, :background), version), + else: :ok + end + + def start(%Post{}), do: :ok + + defp start_in(:background, version) do + Feather.TaskSupervisor + |> Task.Supervisor.start_child(fn -> embed(version) end) + |> log_start_failure(version) + end + + defp start_in(:inline, version) do + embed(version) + :ok + end + + defp start_in(false, _version), do: :ok + + defp log_start_failure({:ok, _pid}, _version), do: :ok + + defp log_start_failure({:error, reason}, version) do + Logger.error("Post version #{version.id} not read: #{inspect(reason)}") + end + + defp read?(%PostVersion{embedding: embedding, embedding_model: model}), + do: is_binary(embedding) and model == Ollama.embedding_model() + + defp embed(%PostVersion{} = version) do + with {:ok, input} <- input(version), + {:ok, vector} <- Ollama.embedding(input) do + store(version, vector, Ollama.embedding_model()) + else + {:error, _reason} -> :error + end + end + + @doc """ + Stores the vector, as Ollama returns it L2-normalised, as the version's + embedding by `model`. When the version is the published version + of a post the live site already shows, the site counts as changed since + its last deploy, as the related posts the export shows may change. + """ + @spec store(PostVersion.t(), [number()], String.t()) :: :ok + def store(%PostVersion{id: id} = version, vector, model) do + Repo.update_all(from(v in PostVersion, where: v.id == ^id), + set: [embedding: encode(vector), embedding_model: model] + ) + + touch_site_if_live(version) + end + + defp touch_site_if_live(%PostVersion{id: id} = version) do + case Repo.one(from p in Post, where: p.published_version_id == ^id, select: p.site_id) do + nil -> :ok + site_id -> touch_site_if_live(Scope.for_site(%Site{id: site_id}), version) + end + end + + defp touch_site_if_live(scope, version) do + target = Publishing.get_target(scope) + + if target && Publishing.live?(target, version), + do: Sites.touch_site(scope), + else: :ok + end + + defp input(%PostVersion{} = version) do + case Blocks.model_text(version.content, @text_length) do + "" -> + {:error, :no_text} + + text -> + {:ok, "task: clustering | query: #{title(version)}. #{version.description}\n\n#{text}"} + end + end + + defp title(version) do + cond do + not Validations.blank?(version.title) -> version.title + not Validations.blank?(version.meta_title) -> version.meta_title + true -> "none" + end + end + + @doc """ + Reads the published versions of the scope's posts that no vector of the + configured model was stored for, one after the other. Publishes nothing + again. Returns how many the model read. + """ + @spec backfill(Scope.t()) :: non_neg_integer() + def backfill(%Scope{site: %Site{id: site_id}}) do + model = Ollama.embedding_model() + + from(v in PostVersion, + join: p in Post, + on: p.published_version_id == v.id, + where: p.site_id == ^site_id, + where: is_nil(v.embedding_model) or v.embedding_model != ^model, + order_by: p.inserted_at + ) + |> Repo.all() + |> Enum.count(&(embed(&1) == :ok)) + end + + @doc "A vector as stored: little-endian 32-bit floats." + @spec encode([number()]) :: binary() + def encode(vector), do: for(x <- vector, into: <<>>, do: <>) + + @doc "A stored vector as a list of floats." + @spec decode(binary()) :: [float()] + def decode(binary), do: for(<>, do: x) +end diff --git a/lib/feather/content/post_version.ex b/lib/feather/content/post_version.ex index 745b7e5c..92b43df8 100644 --- a/lib/feather/content/post_version.ex +++ b/lib/feather/content/post_version.ex @@ -1,6 +1,9 @@ defmodule Feather.Content.PostVersion do @moduledoc """ A published state of a post, see `Feather.Content.publish/2`. + + `embedding` and `embedding_model` hold what the embedding model read in + it (`Feather.Content.Embedding`); they are not copied from the post. """ use Feather.Schema @@ -34,6 +37,8 @@ defmodule Feather.Content.PostVersion do field :description, :string field :description_generated, :boolean, default: false field :meta_title, :string + field :embedding, :binary + field :embedding_model, :string belongs_to :post, Feather.Content.Post belongs_to :published_by, Feather.Accounts.User diff --git a/lib/feather/content/related_posts.ex b/lib/feather/content/related_posts.ex new file mode 100644 index 00000000..21769823 --- /dev/null +++ b/lib/feather/content/related_posts.ex @@ -0,0 +1,113 @@ +defmodule Feather.Content.RelatedPosts do + @moduledoc """ + Ranks a site's posts as related posts of each other (ADR 0036): the + other posts closest in content by the cosine similarity of their published + versions' embeddings (`Feather.Content.Embedding`), plus 0.06 per + shared tag, at most 0.12. A tag counts only while at most three posts + or at most a fifth of the due posts handed in carry it, whichever is more, + so tags on many posts add nothing. On equal scores the post closer in + publish date wins. Posts without a vector of the model take no part on + either side. + + Everything comes from the published versions (preloaded): their + vectors, tags and publish dates. The preview ranks the same published + versions, but among the posts it shows, so drafts and changed publish + dates can change its list. `all/3` compares every pair once + (the export), `of/4` one post with all others (the preview); both keep + the `limit` best. + """ + + alias Feather.Content.{Embedding, Post, PostVersion, Tags} + + @tag_bonus 0.06 + @max_tag_bonus 0.12 + @min_tag_limit 3 + + @doc "The ids of the `limit` best related posts of each post with a vector, best first, by post id." + @spec all([Post.t()], String.t(), pos_integer()) :: %{Ecto.UUID.t() => [Ecto.UUID.t()]} + def all(posts, model, limit) do + readings = readings(posts, model) + scored = readings |> Map.values() |> pair_scores(%{}) + + Map.new(readings, fn {id, reading} -> + {id, best(reading, Map.get(scored, id, []), limit)} + end) + end + + @doc "The ids of the `limit` best related posts of one of the posts, best first; none without a vector." + @spec of([Post.t()], Post.t(), String.t(), pos_integer()) :: [Ecto.UUID.t()] + def of(posts, %Post{id: id}, model, limit) do + case Map.pop(readings(posts, model), id) do + {nil, _readings} -> + [] + + {reading, others} -> + best(reading, Enum.map(Map.values(others), &{&1, score(reading, &1)}), limit) + end + end + + defp readings(posts, model) do + versions = for %Post{published_version: %PostVersion{} = v} = post <- posts, do: {post.id, v} + counting = counting_tags(versions) + + for {id, %PostVersion{embedding: embedding, embedding_model: ^model} = version} <- versions, + is_binary(embedding), + into: %{} do + {id, + %{ + id: id, + vector: Embedding.decode(embedding), + tags: version |> Tags.tag_list() |> MapSet.new() |> MapSet.intersection(counting), + publish_at: version.publish_at + }} + end + end + + defp counting_tags(versions) do + limit = max(@min_tag_limit, length(versions) / 5) + + for {tag, count} <- + versions |> Enum.flat_map(fn {_id, v} -> Tags.tag_list(v) end) |> Enum.frequencies(), + count <= limit, + into: MapSet.new(), + do: tag + end + + defp pair_scores([], scored), do: scored + + defp pair_scores([reading | rest], scored) do + scored = + Enum.reduce(rest, scored, fn other, scored -> + score = score(reading, other) + + scored + |> Map.update(reading.id, [{other, score}], &[{other, score} | &1]) + |> Map.update(other.id, [{reading, score}], &[{reading, score} | &1]) + end) + + pair_scores(rest, scored) + end + + defp best(reading, scored_others, limit) do + scored_others + |> Enum.sort_by(fn {other, score} -> {-score, date_distance(reading, other), other.id} end) + |> Enum.take(limit) + |> Enum.map(fn {other, _score} -> other.id end) + end + + defp score(reading, other), + do: dot(reading.vector, other.vector, 0.0) + tag_bonus(reading.tags, other.tags) + + defp dot([x | xs], [y | ys], sum), do: dot(xs, ys, sum + x * y) + defp dot(_rest, _other_rest, sum), do: sum + + defp tag_bonus(tags, other_tags) do + shared = tags |> MapSet.intersection(other_tags) |> MapSet.size() + min(@tag_bonus * shared, @max_tag_bonus) + end + + defp date_distance(%{publish_at: %DateTime{} = a}, %{publish_at: %DateTime{} = b}), + do: abs(DateTime.diff(a, b)) + + defp date_distance(_reading, _other), do: :undated +end diff --git a/lib/feather/content/saving.ex b/lib/feather/content/saving.ex index db4e911a..f3f5f1ee 100644 --- a/lib/feather/content/saving.ex +++ b/lib/feather/content/saving.ex @@ -40,7 +40,9 @@ defmodule Feather.Content.Saving do where `Feather.Content.Description.needed?/2` says so, and a short post's meta title and slug where `Feather.Content.ShortPost.needed?/2` does, outside the transaction, since that can take up to 90 seconds. - Without an answer it is published as it is. + Without an answer it is published as it is. After a post is published, + the embedding model reads its published version in the background + (`Feather.Content.Embedding.start/1`). """ @spec publish(Scope.t(), record) :: {:ok, record} | {:error, :stale | :slug_taken} when record: Post.t() | Page.t() | Project.t() @@ -52,11 +54,9 @@ defmodule Feather.Content.Saving do {:error, :stale} -> %{} end - Repo.transact(fn -> - Versions.publish_stored(scope, record, - prepare: &Suggestions.put_suggestions(&1, suggestions) - ) - end) + Versions.publish_committed(scope, record, + prepare: &Suggestions.put_suggestions(&1, suggestions) + ) end @doc false diff --git a/lib/feather/content/versions.ex b/lib/feather/content/versions.ex index eb7ae42a..c7e59965 100644 --- a/lib/feather/content/versions.ex +++ b/lib/feather/content/versions.ex @@ -12,7 +12,8 @@ defmodule Feather.Content.Versions do import Ecto.Query, warn: false alias Feather.Accounts.Scope - alias Feather.Content.{Page, PageVersion, Post, PostVersion, Project, ProjectVersion, Slug} + alias Feather.Content.{Embedding, Page, PageVersion, Post, PostVersion, Project, ProjectVersion} + alias Feather.Content.Slug alias Feather.Redirects alias Feather.Repo alias Feather.Sites @@ -43,6 +44,14 @@ defmodule Feather.Content.Versions do @doc false def lock_opts, do: @lock_opts + @doc false + # publish_stored/3 in its own transaction; after the commit the model reads a post's new version. + def publish_committed(scope, record, opts \\ []) do + result = Repo.transact(fn -> publish_stored(scope, record, opts) end) + with {:ok, %Post{} = post} <- result, do: Embedding.start(post) + result + end + @doc false # Publishes the stored record in the caller's transaction. `state: :unpublished` (the # default) publishes a draft or unpublished changes and leaves a published record as it diff --git a/lib/feather/ollama.ex b/lib/feather/ollama.ex index 8435381f..d5d0d95b 100644 --- a/lib/feather/ollama.ex +++ b/lib/feather/ollama.ex @@ -7,7 +7,10 @@ defmodule Feather.Ollama do The URL comes from `config :feather, :ollama_url` (`OLLAMA_URL`); without one nothing is asked and every call returns `{:error, :not_configured}`. The model comes from `config :feather, :ollama_model` (`OLLAMA_MODEL`), - by default Gemma 4 26B, which must read images. + by default Gemma 4 26B, which must read images. Embeddings + (`embedding/1`, `POST /api/embed`) come from `config :feather, + :ollama_embedding_model` (`OLLAMA_EMBEDDING_MODEL`), by default + EmbeddingGemma 2. Tests stub the requests with `Req.Test` (`config :feather, :ollama_req_options, plug: {Req.Test, Feather.Ollama}`). """ @@ -17,6 +20,7 @@ defmodule Feather.Ollama do require Logger @default_model "gemma4:26b-a4b-it-qat" + @default_embedding_model "embeddinggemma-2:270m" @timeout 90_000 @max_alt_length 250 @max_description_length 300 @@ -171,6 +175,42 @@ defmodule Feather.Ollama do end end + @doc """ + The embedding of a text by the embedding model: a vector of numbers, + L2-normalised. Ollama cuts a text longer than the model's window. + """ + @spec embedding(String.t()) :: {:ok, [number()]} | {:error, term()} + def embedding(text) when is_binary(text) do + if configured?() do + req() + |> Req.post( + url: "/api/embed", + json: %{model: embedding_model(), input: [text], truncate: true} + ) + |> decode_embedding() + else + {:error, :not_configured} + end + end + + @doc "The name of the embedding model `embedding/1` asks." + @spec embedding_model() :: String.t() + def embedding_model, do: setting(:ollama_embedding_model, @default_embedding_model) + + defp decode_embedding( + {:ok, %Req.Response{status: 200, body: %{"embeddings" => [vector]} = body}} + ) do + if vector?(vector), do: {:ok, vector}, else: log_failure({:invalid_answer, body}) + end + + defp decode_embedding({:ok, %Req.Response{status: 200} = response}), + do: log_failure({:invalid_answer, response.body}) + + defp decode_embedding({:ok, %Req.Response{status: status}}), do: log_failure({:status, status}) + defp decode_embedding({:error, exception}), do: log_failure(exception) + + defp vector?(vector), do: is_list(vector) and vector != [] and Enum.all?(vector, &is_number/1) + defp trimmed(text, max_length), do: text |> String.trim() |> String.slice(0, max_length) defp decode_answer( @@ -199,10 +239,12 @@ defmodule Feather.Ollama do defp url, do: Application.get_env(:feather, :ollama_url) - defp model do - case Application.get_env(:feather, :ollama_model) do - model when model in [nil, ""] -> @default_model - model -> model + defp model, do: setting(:ollama_model, @default_model) + + defp setting(key, default) do + case Application.get_env(:feather, key) do + value when value in [nil, ""] -> default + value -> value end end end diff --git a/lib/feather/release.ex b/lib/feather/release.ex index 6c2d2a1f..a177761d 100644 --- a/lib/feather/release.ex +++ b/lib/feather/release.ex @@ -5,6 +5,7 @@ defmodule Feather.Release do bin/feather eval "Feather.Release.migrate()" bin/feather eval 'Feather.Release.create_user("me@example.com", true)' bin/feather eval "Feather.Release.backfill_suggestions()" + bin/feather eval "Feather.Release.backfill_embeddings()" `bin/migrate` and `bin/server` (in `rel/overlays/bin`) wrap the first. """ @@ -107,21 +108,11 @@ defmodule Feather.Release do existed, site by site: the alt texts of all images (`Feather.Media.backfill_alt_texts/1`), and descriptions and short posts' meta titles and slugs of published records without unpublished - changes, by publishing them again (`Feather.Content.backfill_suggestions/1`). - Needs `OLLAMA_URL`. Deploy the sites by hand afterwards. + changes, by publishing them again (`Feather.Content.backfill_suggestions/1`); + the embedding model reads the posts published again right away. Needs + `OLLAMA_URL`. Deploy the sites by hand afterwards. """ - def backfill_suggestions do - with_app(fn -> - {:ok, _} = Application.ensure_all_started(:req) - - if Feather.Ollama.configured?() do - Enum.each(Feather.Repo.all(Site), &backfill_site/1) - else - IO.puts(:stderr, "OLLAMA_URL is not set.") - {:error, :not_configured} - end - end) - end + def backfill_suggestions, do: each_site_with_model(&backfill_site/1) defp backfill_site(site) do scope = Scope.for_site(site) @@ -133,6 +124,51 @@ defmodule Feather.Release do ) end + @doc """ + Reads the published posts of every site that no embedding of the + configured model was stored for (`Feather.Content.backfill_embeddings/1`), + after a change of the model and for posts published before embeddings + existed. Publishes nothing again. Needs `OLLAMA_URL`. Deploy the sites + by hand afterwards. + """ + def backfill_embeddings, do: each_site_with_model(&backfill_embeddings_of/1) + + defp backfill_embeddings_of(site) do + read = site |> Scope.for_site() |> Feather.Content.backfill_embeddings() + IO.puts("#{site.domain}: posts read: #{read}") + end + + defp each_site_with_model(fun) do + with_app(fn -> + {:ok, _} = Application.ensure_all_started(:req) + + if Feather.Ollama.configured?(), do: each_site_reading_inline(fun), else: not_configured() + end) + end + + defp each_site_reading_inline(fun), + do: reading_inline(fn -> Enum.each(Feather.Repo.all(Site), fun) end) + + defp not_configured do + IO.puts(:stderr, "OLLAMA_URL is not set.") + {:error, :not_configured} + end + + # eval starts no task supervisor and would end before background readings. + defp reading_inline(fun) do + previous = Application.fetch_env(:feather, :post_embeddings) + Application.put_env(:feather, :post_embeddings, :inline) + + try do + fun.() + after + case previous do + {:ok, mode} -> Application.put_env(:feather, :post_embeddings, mode) + :error -> Application.delete_env(:feather, :post_embeddings) + end + end + end + defp with_app(fun) do load_app() {:ok, _} = Application.ensure_all_started(:ecto_sql) diff --git a/lib/feather/static_site/export.ex b/lib/feather/static_site/export.ex index 162698b8..1c5128a0 100644 --- a/lib/feather/static_site/export.ex +++ b/lib/feather/static_site/export.ex @@ -51,7 +51,12 @@ defmodule Feather.StaticSite.Export do """ @spec run(Site.t(), Routes.t(), Sink.t(), keyword()) :: :ok def run(%Site{} = site, %Routes{} = routes, sink, opts \\ []) do - data = site |> SiteData.load(Keyword.take(opts, [:now])) |> SiteData.prerender() + data = + site + |> SiteData.load(Keyword.take(opts, [:now])) + |> SiteData.prerender() + |> SiteData.put_related_posts() + redirects = SiteData.redirects(data) documents = documents(data) diff --git a/lib/feather/static_site/renderer.ex b/lib/feather/static_site/renderer.ex index 1f22b4d8..656dbe50 100644 --- a/lib/feather/static_site/renderer.ex +++ b/lib/feather/static_site/renderer.ex @@ -117,6 +117,7 @@ defmodule Feather.StaticSite.Renderer do header_image: post.header_image, newer_post: newer, older_post: older, + related_posts: SiteData.related_posts(data, post), replies: Map.get(data.replies, post.id, []), reply_handle: Fediverse.handle(data.site) }) diff --git a/lib/feather/static_site/site_data.ex b/lib/feather/static_site/site_data.ex index e1c0757d..a6b5f3ad 100644 --- a/lib/feather/static_site/site_data.ex +++ b/lib/feather/static_site/site_data.ex @@ -10,12 +10,13 @@ defmodule Feather.StaticSite.SiteData do import Ecto.Query, warn: false alias Feather.Accounts.Scope - alias Feather.{Books, Content, Diagrams, Fediverse, Highlighting, Media, Redirects, Sites} + alias Feather.{Books, Content, Diagrams, Fediverse, Highlighting, Media, Ollama, Redirects} alias Feather.Books.Book - alias Feather.Content.{Blocks, Page, Post, Project} + alias Feather.Content.{Blocks, Page, Post, Project, RelatedPosts} alias Feather.Media.Image alias Feather.Redirects.Redirect alias Feather.Repo + alias Feather.Sites alias Feather.Sites.{NavigationItem, Site, SocialMediaLink} alias Feather.StaticSite.Themes @@ -26,6 +27,7 @@ defmodule Feather.StaticSite.SiteData do follower_count: 0, following_count: 0, posts: [], + related: nil, replies: %{}, member_replies: [], pages: [], @@ -46,6 +48,7 @@ defmodule Feather.StaticSite.SiteData do follower_count: non_neg_integer(), following_count: non_neg_integer(), posts: [Post.t()], + related: %{Ecto.UUID.t() => [Ecto.UUID.t()]} | nil, replies: %{Ecto.UUID.t() => [Feather.Fediverse.Mention.t()]}, member_replies: [Feather.Fediverse.MemberReply.t()], pages: [Page.t()], @@ -64,10 +67,15 @@ defmodule Feather.StaticSite.SiteData do @project_preloads [:header_image, :thumbnail_image] @book_preloads [:cover_image] + @related_count 3 + # Enough to still show three after leaving out the newer and the older post. + @ranked_count @related_count + 2 + @doc """ Loads the data of a site. `posts` are the posts whose `publish_at` has - passed, newest first; `pages` are all pages, the homepage included; a - book's `post` is its review among `posts`, nil if it is none of them; the + passed, newest first; `related` is nil until `put_related_posts/1`; + `pages` are all pages, the homepage included; a book's `post` is its + review among `posts`, nil if it is none of them; the navigation leaves out pages that are not among them; `now` is the time the site is exported at; `follower_count` counts the followers of the site's fediverse account, `following_count` the accounts that accepted @@ -242,6 +250,43 @@ defmodule Feather.StaticSite.SiteData do end end + @doc """ + Ranks all posts once (`Feather.Content.RelatedPosts.all/3`) and keeps + the ids of each post's best related posts in `related`, for an export + that renders every post. The vectors themselves stay out of the data, + which every render job gets a copy of. + """ + @spec put_related_posts(t()) :: t() + def put_related_posts(%__MODULE__{posts: posts} = data), + do: %{data | related: RelatedPosts.all(posts, Ollama.embedding_model(), @ranked_count)} + + @doc """ + Up to three related posts of a post among the posts, best first, by the + published versions (`Feather.Content.RelatedPosts`). The preview ranks + the same versions among the posts it shows, so drafts and changed + publish dates can change its list. Its newer and older post (`neighbours/2`), + which the post navigation shows anyway, are left out. Without + `put_related_posts/1` it ranks just this post. None for a draft, a post + without an embedding and posts not in the list. + """ + @spec related_posts(t(), Post.t()) :: [Post.t()] + def related_posts(%__MODULE__{posts: posts} = data, %Post{} = post) do + posts_by_id = Map.new(posts, &{&1.id, &1}) + {newer, older} = neighbours(data, post) + shown = for %Post{id: id} <- [newer, older], do: id + + data + |> ranked_ids(post) + |> Enum.reject(&(&1 in shown)) + |> Enum.take(@related_count) + |> Enum.map(&Map.fetch!(posts_by_id, &1)) + end + + defp ranked_ids(%__MODULE__{related: nil, posts: posts}, post), + do: RelatedPosts.of(posts, post, Ollama.embedding_model(), @ranked_count) + + defp ranked_ids(%__MODULE__{related: related}, %Post{id: id}), do: Map.get(related, id, []) + @doc """ The images the exported site uses, without duplicates: the images embedded in the content of its posts, pages and projects, their header diff --git a/lib/feather/static_site/theme.ex b/lib/feather/static_site/theme.ex index 6de3a0aa..39b6a94c 100644 --- a/lib/feather/static_site/theme.ex +++ b/lib/feather/static_site/theme.ex @@ -49,8 +49,11 @@ defmodule Feather.StaticSite.Theme do and each view its own: * `c:home/1` - `posts`, `current_page`, `total_pages` - * `c:post/1` - `post`, `newer_post`, `older_post`, `replies` (the - approved replies to it, oldest first, see + * `c:post/1` - `post`, `newer_post`, `older_post`, `related_posts` + (`Feather.StaticSite.SiteData.related_posts/2`, listed first in + the post navigation, left out when empty; label every link there + with `ThemeHelpers.post_nav_label/2`), + `replies` (the approved replies to it, oldest first, see `Feather.Fediverse.Mentions`, each with the members' public replies to it in `member_replies`, shown below it and marked as by the author) and `reply_handle` (the site's fediverse handle, nil without diff --git a/lib/feather/static_site/theme_helpers.ex b/lib/feather/static_site/theme_helpers.ex index d27a5a5b..f772ae69 100644 --- a/lib/feather/static_site/theme_helpers.ex +++ b/lib/feather/static_site/theme_helpers.ex @@ -483,6 +483,30 @@ defmodule Feather.StaticSite.ThemeHelpers do {:safe, [mark, escape(text)]} end + @doc """ + A post as the post navigation links it: its emoji in a column of its + own, empty without one, beside its title or a short post's meta title + (`post_meta_title/1`); with `year: true` its publish year after that. + """ + @spec post_nav_label(Content.Post.t(), keyword()) :: Phoenix.HTML.safe() + def post_nav_label(post, opts \\ []) do + year = + if opts[:year], + do: [~s( ), to_string(post.publish_at.year), ""], + else: [] + + {:safe, + [ + ~s(), + escape(post_meta_title(post)), + "", + year, + "" + ]} + end + @description_length 155 @doc """ @@ -527,9 +551,9 @@ defmodule Feather.StaticSite.ThemeHelpers do end @doc """ - What the document title, `og:title`, the Markdown alternates and - `llms.txt` name a post by: a short post's meta title, else its label - (`post_label/1`). + What the document title, `og:title`, the Markdown alternates, + `llms.txt` and the post navigation name a post by: a short post's meta + title, else its label (`post_label/1`). """ @spec post_meta_title(Content.Post.t()) :: String.t() def post_meta_title(post) do diff --git a/lib/feather/static_site/themes/feather_page/post.html.eex b/lib/feather/static_site/themes/feather_page/post.html.eex index 260ce747..767455a6 100644 --- a/lib/feather/static_site/themes/feather_page/post.html.eex +++ b/lib/feather/static_site/themes/feather_page/post.html.eex @@ -14,17 +14,29 @@ <% true -> %> <% end %> - <%= if @newer_post do %> -

- <%= gettext("Newer post") %> - <%= emoji_label(@newer_post.emoji, post_label(@newer_post)) %> -

- <% end %> - <%= if @older_post do %> -

- <%= gettext("Older post") %> - <%= emoji_label(@older_post.emoji, post_label(@older_post)) %> -

- <% end %> -

<%= gettext("All posts") %>

+
+ <%= if @related_posts != [] do %> +
+ <%= gettext("Related posts") %> + +
+ <% end %> + <%= if @newer_post do %> +

+ <%= gettext("Newer post") %> + <%= post_nav_label(@newer_post) %> +

+ <% end %> + <%= if @older_post do %> +

+ <%= gettext("Older post") %> + <%= post_nav_label(@older_post) %> +

+ <% end %> +

<%= gettext("All posts") %>

+
diff --git a/lib/feather/static_site/themes/simple_sidebar/post.html.eex b/lib/feather/static_site/themes/simple_sidebar/post.html.eex index bd8f57fc..fb633727 100644 --- a/lib/feather/static_site/themes/simple_sidebar/post.html.eex +++ b/lib/feather/static_site/themes/simple_sidebar/post.html.eex @@ -16,16 +16,26 @@ <%= if @reply_handle do %><%= replies(%{replies: @replies, handle: @reply_handle, routes: @routes, site: @site}) %><% end %>