From 82a8bbc5cd7b750960d6fbd5064f5a558446fbd9 Mon Sep 17 00:00:00 2001 From: Tobias Macey Date: Mon, 31 Aug 2026 15:19:25 -0400 Subject: [PATCH] docs: propose a pluggable search backend for Studio content search ADR 0001 chose Meilisearch for content/search as an experiment, is still in Draft, and committed to keeping the engine-specific code isolated "so it's relatively easy to swap out later if this experiment doesn't pan out." This proposes exercising that exit, not reversing it. Two motivations. First, operator cost: edx-search now ships a Typesense backend for Course Discovery and Learning-MFE courseware search, so an operator who wants both that and Studio content search has to run two search engines. Second, licensing. ADR 0001's high-availability concern has been resolved, but only in Meilisearch's Enterprise Edition, which is BUSL-1.1 and ships as separate binaries. Self-hosted Community Edition still has no replication. That is the same class of problem ADR 0001 was written to escape when Elasticsearch left Apache 2.0. Typesense's Raft clustering is GPL-3.0 and part of the ordinary self-hosted product. Proposes a narrow interface internal to content/search with implementations for both engines, Meilisearch remaining the default. Deliberately not a general search abstraction: ADR 0001's own account of django-haystack becoming "an obstacle to upgrades" is the strongest objection to this change, and the scope is drawn to avoid repeating it. Records the parity and schema findings behind the proposal, each checked against a running Typesense 30.2 rather than inferred from documentation. --- .../0002-pluggable-search-backend.rst | 240 ++++++++++++++++++ 1 file changed, 240 insertions(+) create mode 100644 openedx/core/djangoapps/content/search/docs/decisions/0002-pluggable-search-backend.rst diff --git a/openedx/core/djangoapps/content/search/docs/decisions/0002-pluggable-search-backend.rst b/openedx/core/djangoapps/content/search/docs/decisions/0002-pluggable-search-backend.rst new file mode 100644 index 000000000000..aaff782aa131 --- /dev/null +++ b/openedx/core/djangoapps/content/search/docs/decisions/0002-pluggable-search-backend.rst @@ -0,0 +1,240 @@ +Pluggable Search Backend for Studio Content Search +################################################## + +Status +****** + +Draft + +Supersedes `ADR 0001 <0001-meilisearch.rst>`_ in part: it keeps that ADR's +rejection of ``edx-search`` and of Elasticsearch/OpenSearch, and revisits only +its choice of Meilisearch as the single supported engine. + + +Context +******* + +What ADR 0001 decided, and what it left open +============================================ + +`ADR 0001 <0001-meilisearch.rst>`_ chose Meilisearch for the then-new +``content/search`` app. It is worth being precise about what that ADR committed +to, because it anticipated this one: + +* Its status is still **Draft**. It was never accepted. +* Decision 1 adopted Meilisearch "as an experiment and to evaluate it more + thoroughly." +* Decision 3 committed to keeping "the Meilisearch-specific code isolated to the + new ``content/search`` Django app, so it's relatively easy to swap out later + if this experiment doesn't pan out." + +This ADR is that swap being proposed, not a reversal of a settled decision. The +isolation ADR 0001 asked for was largely achieved. Counting engine references +across the app: + +* ``api.py`` holds 105 of them and is where essentially all engine coupling + lives. +* ``handlers.py`` has 21, but 16 are the ``meilisearch_enabled`` settings gate + and the rest are prose — a rename and one indirection. +* ``documents.py`` has 17, all either prose or the ``meili_id_...`` and + ``_meili_access_id_...`` helpers, whose behaviour is a key-slugging rule + rather than an engine call. +* ``tasks.py`` has 16, of which 14 are ``MeilisearchError`` in retry handling. + This is genuine coupling, and needs the backend to supply its own retryable + exception type. +* ``views.py``, ``urls.py``, ``models.py`` and ``plain_text_math.py`` have + effectively none. + +What changed since +================== + +The platform gained a second search engine. ``edx-search`` now ships a Typesense +backend (``search/typesense.py``, contributed as FC-0091), which serves Course +Discovery and Learning-MFE courseware search. It is a different subsystem with a +different index and no overlap with ``content/search`` — the two do not share +code and cannot share an index. + +The consequence is that an operator who wants both courseware search on +Typesense and Studio content search must deploy and operate **two** search +engines for one platform. That is the problem this ADR addresses. It is a cost +borne by operators, and it grows as more of the platform moves onto Typesense. + +High availability, and where its licence landed +=============================================== + +ADR 0001's first stated concern was that Meilisearch "doesn't (yet) support High +Availability via replication, although this is planned and under development." +That has been resolved, but not in a way that helps an Open edX operator. + +Meilisearch now splits into two editions. The Community Edition is MIT-licensed +and free; the Enterprise Edition is licensed under BUSL-1.1 and shipped as +separate binaries and Docker images (``getmeili/meilisearch-enterprise``). +Replicated sharding — the feature that answers ADR 0001's concern — requires +Enterprise Edition v1.37 or later. A self-hosted Community Edition deployment +still has no replication, and therefore still has no high availability. + +This matters here more than it would elsewhere, because it is the same problem +ADR 0001 was written to escape. That ADR's opening argument against +Elasticsearch is that "in 2021, the license of Elasticsearch changed from Apache +2.0 to a more restrictive license," and that this "is problematic for many Open +edX operators." Meilisearch's high-availability story has since arrived behind a +BUSL-1.1 licence. An operator who cannot or will not take a commercial licence +is left where they started: running the platform's Studio search on a +single-node engine with no replication path. + +Typesense's clustering is not gated this way. It is GPL-3.0, and its +Raft-backed replication is part of the ordinary self-hosted product — the same +``typesense/typesense`` image, configured with a shared nodes file. + +One further point from ADR 0001 should be set aside rather than reused: its +second concern, the absence of boolean operators in keyword search, is **not** a +Typesense advantage. Typesense supports boolean operators in filters, as +Meilisearch already does, and no difference in the keyword query itself has been +established. + +Meilisearch remains a good engine, and for a single-node deployment that never +needs replication it remains a reasonable choice. The argument here is about how +many engines the platform obliges an operator to run, and about whether the +answer to the platform's high-availability question is allowed to be a +commercial licence. + +The abstraction risk, taken seriously +===================================== + +ADR 0001's own context is the strongest objection to this proposal. It records +that ``edx-search`` formerly used django-haystack as a cross-engine abstraction, +and that this "was ripped out after the package was abandoned upstream and it +became an obstacle to upgrades and efficiently utilizing Elasticsearch (the +abstraction layer imposed significant limits)." + +That history is a warning about a *general* search abstraction attempting to +span engines with different models. The proposal below is deliberately not that. +The interface is narrow, internal, and shaped by the operations +``content/search`` actually performs — it is not a general-purpose search API, +carries no ambition to serve other apps, and can be changed freely because +nothing outside this app depends on it. + +Feasibility +=========== + +A capability-by-capability review found no parity blocker, and each mapping was +executed against a running Typesense 30.2 rather than inferred from +documentation: + +* ``distinctAttribute`` maps to ``group_by``; the ordered + ``searchableAttributes`` list to ``query_by`` plus explicit + ``query_by_weights``; ``sortableAttributes`` to per-field ``sort``; and the + ``rankingRules`` entry that puts ``"sort"`` ahead of relevance needs no + counterpart, because an explicit ``sort_by`` already outranks text matching. +* The ``create``/``swap_indexes``/``delete`` rebuild becomes a collection alias + repoint, which is simpler and reclaims disk immediately. +* ``delete_documents(filter=...)`` maps to a ``filter_by`` delete. +* The tenant-token property ADR 0001 specifically credited Meilisearch for — + minting a restricted, permission-scoped key locally so the browser can query + the engine directly without routing through Django — holds for Typesense. + Scoped search keys are an HMAC of a parent key over a rule carrying + ``filter_by`` and ``expires_at``, and require no API call to create. The + browser-direct architecture is preserved. +* ``_wait_for_meili_task`` and the task-polling layer around every write + disappear, because Typesense writes are synchronous. + +An explicit Typesense schema covering library blocks, collections and containers +was written and validated against a live server. Three points found there are +recorded because they will otherwise be rediscovered painfully: + +* A ``content\..*`` string wildcard field, the pattern ``edx-search`` uses, + rejects the array sub-fields that container documents carry. Explicit + ``string[]`` overrides are required ahead of the wildcard. +* ``max_facet_values`` defaults to 10 and also caps the reported total, so a + truncated facet list is not detectable from the response. Every faceted + request must set it explicitly. Meilisearch's equivalent defaults to 100. +* ``documents.py`` serialises datetimes with ``.timestamp()``, which yields a + float, so those fields must be declared ``float`` rather than the ``int64`` + ``edx-search`` uses for its own datetimes. + + +Decision +******** + +1. We will introduce a **narrow backend interface** within ``content/search``, + covering only the operations the app performs today: index lifecycle + (create, alias/swap, delete), document upsert and delete-by-filter, search + with filters, facets and sorting, faceted-value lookup for the tag tree, and + minting a permission-scoped client credential. +2. We will implement that interface for **both Meilisearch and Typesense**, and + select between them by setting. The Meilisearch implementation is the + existing code behind the new interface, and remains supported. +3. Meilisearch stays the **default**, so no existing deployment changes + behaviour by upgrading. Choosing Typesense is opt-in. +4. The interface is **internal to** ``content/search``. It is not a general + search abstraction, is not offered to other apps, and explicitly does not + attempt to replace or extend ``edx-search``. +5. We will keep the Authoring MFE's ``search-manager`` on one engine-agnostic + client seam rather than branching per engine throughout the component tree. +6. We will not attempt to unify the ``content/search`` index with the + ``edx-search`` courseware index. They serve different subsystems, carry + different fields, and index different content (published versus draft). + + +Consequences +************ + +1. An operator who already runs Typesense for courseware search or Course + Discovery can run Studio content search on it too, and drop Meilisearch. An + operator who prefers Meilisearch keeps it, unchanged and by default. +2. ``content/search`` acquires an internal indirection it does not have today. + This is a real cost. It is bounded by keeping the interface small and + private to the app, and by the fact that both implementations live in-tree, + so a change to the interface can be made across both at once — the failure + mode that made django-haystack painful, an abandoned external abstraction, + does not apply. +3. The task-polling layer becomes conditional rather than unconditional, since + Meilisearch still needs it. It is confined to the Meilisearch + implementation. +4. The Authoring MFE's ``search-manager`` is written against the raw + ``meilisearch`` JavaScript client rather than an InstantSearch adapter, so a + meaningful share of it must be reworked to sit behind a client seam. This is + the largest single piece of work, and larger than the backend change. +5. Result grouping changes shape: Typesense returns ``grouped_hits`` where + Meilisearch's ``distinctAttribute`` returns flat ``hits``. The seam has to + normalise this. +6. Highlighting and cropping do not map exactly. Typesense decides per field + whether to snippet and how much context to keep, where Meilisearch crops to a + token count per attribute. Result cards need a visual pass, not a rename. + + +Alternatives Considered +*********************** + +Leave ``content/search`` on Meilisearch only +============================================ + +The status quo. It is a defensible choice on its merits — Meilisearch works, and +its earlier HA gap has closed. It is rejected only because it obliges operators +who have already adopted Typesense elsewhere in the platform to run a second +engine indefinitely, with no path off it. + +Replace Meilisearch with Typesense outright +=========================================== + +Simpler than a pluggable backend: no interface, no indirection, one code path. +Rejected because it forces the change on every operator currently running +Meilisearch for Studio search, to solve a problem only some of them have. The +indirection in Decision 1 is the price of not doing that. + +Route Studio search through the existing ``edx-search`` Typesense backend +========================================================================= + +Investigated and rejected on the evidence. The ``courseware_content`` documents +that backend produces lack the fields Studio search needs — breadcrumbs, +block type, the tag hierarchy, publish status, sortable display name, usage and +context keys — and index published content only, while Studio authors drafts. +``edx-search``'s own documentation states it supports Course Discovery and +Learning-MFE courseware search, and that other use cases are not supported. + +Extend ``edx-search`` to cover Studio search +============================================ + +Rejected for the same reasons ADR 0001 rejected it, which have not changed: the +``edx-search`` API is a mix of abstractions and direct engine usage, and +``content/search`` was deliberately built outside it.