docs: ADR 0001, one search library across Typesense and Meilisearch

[blarghmatey]

ADR 0001 for this repo: one search library for the platform's four search consumers, with Typesense and Meilisearch both first-class, Elasticsearch retired rather than merely excluded.

Reworked from openedx/openedx-platform#39049 per the conclusion there (@bradenmacdonald's preference was a new openedx_search repo with the revised proposal as its ADR 0001), then revised again from his review here. #39049 is closed pointing at this.

I filled in the existing 0001-purpose-of-this-repo.rst stub rather than adding an 0002, on the reading that "why this repo exists" and "what it decided" are the same document. Happy to split it into a short charter plus an 0002 if you would rather 0001 stay brief.

What it decides

1. Use cases register, rather than each surface carrying its own engine integration. There are four separate integrations today: content/search (Meilisearch only), edx-search (four engines per its README), openedx/forum (es/meilisearch/typesense), and edx-notes-api (Elasticsearch 7 only).
2. All four are in scope as adopters. Whichever consumer stays on Elasticsearch keeps Elasticsearch alive for everyone who runs that service, so leaving one out forfeits the goal.
3. Typesense and Meilisearch are implemented, neither shaped around the other. Algolia is not implemented but must stay adoptable.
4. Elasticsearch and OpenSearch are retired, which needs a DEPR proposal for edx-search as a prerequisite.
5. Filters are abstracted; 6. index lifecycle is managed uniformly; 7. Django stays out of the query path; 8. the frontend abstracts at the InstantSearch client interface using the vendors' adapters; 9. no request-handling thread ever waits on an index write; 10. engine backends are verified against real engines in CI, not mocks.

What changed from the review

Four of these reversed something I had written, so they are worth calling out rather than burying in a diff.

Algolia is out of the implementation and only stays adoptable. Building it now means carrying a backend with no user, and being SaaS-only it cannot run in CI, which is precisely how forum's Typesense backend shipped non-functional. What survives is the useful half: the interface and adapter contract are specified so adding it is additive. On the frontend that is nearly free, because the adapter contract is Algolia's client interface.

The frontend uses the existing InstantSearch adapters (typesense-instantsearch-adapter, instant-meilisearch) with React Query and Paragon, instead of the "engine-agnostic client seam" I had proposed building. This corrected a conflation in my own text: InstantSearch's widgets cannot do multi-select hierarchical facets, which is why search-manager was hand-written, but its client interface has nothing to do with that, and that is all the adapters implement. I verified the part that decides it — both implement searchForFacetValues, and instant-meilisearch passes facetQuery/facetName straight through to the same Meilisearch method search-manager calls today. So the tag tree keeps its bespoke UI and loses only its bespoke transport.

Decision 9 was too weak. Saying the API must not assume a task queue does not answer what happens on the engine that has one, and picking the synchronous engine is not a design. Submitting a write is now separate from confirming it landed, and only a worker may wait. How the browser learns the index caught up is deliberately left to #38993.

Typesense holds its whole index in RAM (roughly 2-3x the searchable-field size), which is a real argument for Meilisearch as the default and one I had not been weighing. Recorded with @bradenmacdonald's condition attached. It does not undercut the HA argument: replication under an open licence and indexing cost are separable, and it is fine for the default to favour the cheaper-scaling engine while the other stays fully supported.

Also from review: edx-search stayed Elasticsearch-oriented even after haystack was removed, which is the missing explanation for why adding a second engine there is awkward; Elasticsearch's memory footprint was a motivation independent of licensing (tutor#1141); Studio courseware and library content are separate use cases, and splitting studio_content becomes ordinary registration rather than a special case; content/search adopts first, because a library that handles the hardest case handles the rest.

One correction to a review request: #36868 is an issue rather than a PR, and closed as completed. What shipped for it is #38384, now cited as prior art, along with the design point it settled (Django migrations run once, so index schema changes cannot be migration files).

Still open in the ADR

Default engine; whether Studio courseware and learner courseware share an index; who picks up each consumer's migration.

Evidence

Claims are verified against live sources rather than memory, and where a claim is someone's account rather than something I checked, it says so. The facet feasibility result (29 checks against a real Typesense 30.2) and the same-host indexing measurements are carried over from #39049. doc8 --max-line-length 120 is clean, including its docutils validity check.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ULXYpjr5Hq3s9K4imMvsWf

[openedx-webhooks]

Thanks for the pull request, @blarghmatey!

This repository is currently maintained by @ormsbee.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[bradenmacdonald]

Nice! Some feedback:

[blarghmatey]

Thanks, that was a useful review. Revised in fc8979d and replied on each thread. Everything is taken; four items reversed something I had written, so here is the short version of what moved.

Algolia is out of the implementation. You are right that building a backend nobody has asked for is not worth the maintenance, and it also resolves the CI objection I had raised against my own proposal. What I kept is narrower: the backend interface and the frontend adapter contract are specified so adding Algolia is additive rather than a redesign. There is a Rejected Alternatives entry recording why it is not being built, so the next person to propose it finds the reasoning.

The frontend uses your adapter suggestion, and that is the change I am most glad about, because it replaces something I was proposing to build and maintain. It also corrected a mistake of mine: I had written that the frontend must support multi-select hierarchical facets "which InstantSearch does not," conflating the widgets with the client interface. Only the widget limitation is real. I checked the part that decides whether the adapter route works, since the tag tree is what would fall through a shared abstraction: both adapters implement searchForFacetValues, and instant-meilisearch passes facetQuery and facetName straight through to the same Meilisearch method search-manager calls today. So the tag tree keeps its bespoke UI and loses only its bespoke transport, which is a much smaller job than the one I had described as the largest in the proposal.

Your async point landed. Saying the API must not assume a task queue does not answer what happens on the engine that has one, and choosing Typesense is not a design. Submitting a write is now separate from confirming it landed, and the wait is callable only from a worker. I deliberately left the "how does the browser know" half to #38993 rather than presume it, but if you think the ADR should commit to a mechanism instead of reserving space for one, that is much cheaper to change now than after there are implementations.

Typesense's memory profile is a better argument for Meilisearch as the default than anything I had against it, and I had not been weighing it. Verified: entire index in RAM, roughly 2-3x the searchable-field size, where Meilisearch can serve an index larger than memory. Recorded as an Open Question with your condition attached. It does not undercut the HA case, since replication under an open licence and indexing cost are separable, and the default favouring the cheaper-scaling engine is compatible with the other being fully supported.

Two smaller things worth flagging rather than leaving in the diff:

• #36868 is an issue rather than a PR, and closed as completed. I linked it, but cited #38384 as the prior art to build on instead of a plan to execute, plus the design point it settled: Django migrations run once, so index schema changes cannot be expressed as migration files.
• The studio_content split you proposed in #38993 is now a Consequence saying the two efforts do not conflict. The frontend cost you noted there, a multi-index request to keep "search all libraries and courses" working, is a multiSearch in the adapter interface rather than new machinery, which is a small argument for doing the split before the migration rather than after.

Still open and genuinely undecided: the default engine, whether Studio courseware and learner courseware should share an index (your question, now explicit rather than answered by implication), and who picks up each consumer's migration.

@ormsbee this is ready for your read whenever you have time. The one structural question is whether ADR 0001 should stay a short charter with this as 0002, since I filled in the stub rather than adding a file. Happy either way.

[bradenmacdonald]

Thanks @blarghmatey, sounds great. I'm not sure why, but GitHub isn't letting me mark all my comments as resolved now that you've addressed them.

[blarghmatey]

Resolved them all from my side, since as PR author I can. Every one had a reply first, so the reasoning is still there if anyone wants to reopen a thread rather than take my word that it was addressed.

The two I would most want a second look at, since I made judgement calls rather than just applying a suggestion: whether Decision 9 should commit to a mechanism for how the browser learns the index caught up rather than reserving space for whatever #38993 concludes, and whether leaving the default engine open is right when your RAM argument is fairly decisive on its own.

[ormsbee]

@ormsbee this is ready for your read whenever you have time. The one structural question is whether ADR 0001 should stay a short charter with this as 0002, since I filled in the stub rather than adding a file. Happy either way.

We generally want to keep ADRs short and more limited in scope, largely because (a) it keeps discussions from spiraling on forever; and (b) it makes it easier to find and change our minds on individual decisions in the future. This document lists ten decisions, which I would normally expect to come in at least 3-4 ADRs (e.g. backend engine support, frontend approach, platform adoption plan, etc.)

That being said, much of what's written here is a summary and natural evolution of many, many discussions we've already had on this topic, and I think it's a very useful synthesis of where we are, how we got here, and the core architectural approach we want to adopt in order to address the shortcomings of all our existing systems. So unless we get really bogged down on one of the sub-decisions here, I think it's fine to leave as a big ADR 0001.

The one thing I'll ask is that when we inevitably need to flesh things out later in the development cycle, we do it as new ADRs and update 0001 to make references to those new ADRs, instead of trying to update them in place here. For instance, at some point we may want an ADR to explicitly lay out how we expect clients of this library to test their registered use cases against real engines, and best practices for this.

[ormsbee]

LGTM. I'm very interested in hearing @Ali-Salman29's thoughts on this.

[bradenmacdonald]

I just found out this isn't accurate: edx-notes-api does already support Meilisearch: https://github.com/openedx/edx-notes-api/blob/master/notesapi/v1/views/meilisearch.py