Skip to content

fix(firestore-vector-search): restore batched backfill #3097

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
CorieW wants to merge 6 commits into
base: kits
Choose a base branch
from
+1,681 −86
Open

fix(firestore-vector-search): restore batched backfill #3097

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6294c27
fix(firestore-vector-search): restore the extension's batched backfill
CorieW Sep 2, 2026
a6c430d
fix(firestore-vector-search): batch OpenAI embeddings and record chun…
CorieW Sep 2, 2026
940bab9
fix(firestore-vector-search): run one backfill pass per deploy
CorieW Sep 2, 2026
ac55b20
refactor(firestore-vector-search): name the backfill document data type
CorieW Sep 2, 2026
4b2f759
fix(firestore-vector-search): harden the task id parse and the doc tr…
CorieW Sep 2, 2026
395d746
test(firestore-vector-search): follow the metadata gate and the one-p…
cabljac Sep 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions kits/firestore-vector-search/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,6 @@
- OpenAI embeddings are back on the extension's model and size: `EMBEDDING_PROVIDER: openai` requests `text-embedding-ada-002` at its native 1536 dimensions with a batch size of 16, replacing `text-embedding-3-small` pinned at 512 with a batch size of 1. Vectors written by an earlier version of the kit are not comparable with the ones it writes now, so re-embed the collection after upgrading. The vector index the kit creates for OpenAI is still declared with 512 dimensions, exactly as the extension declared it, so it does not cover the 1536-dimension vectors and `findNearest` fails against it; create the 1536-dimension index yourself if you query an OpenAI-embedded collection.
- Initial release of kit, see README for differences between the legacy extension and this kit
- No Eventarc events are published, matching the extension. The extension declares `onStart`, `onSuccess`, `onError` and `onCompletion` under `firebase.extensions.firestore-vector-search.v1.*` in its `extension.yaml` but never publishes any of them. Earlier `0.0.2-rc` builds of this kit published all four from `embedOnWrite` when `EVENTARC_CHANNEL` was set; they no longer do, and `EVENTARC_CHANNEL` is no longer read. If you subscribed to those events on an rc build, the subscription now receives nothing.
- Restored the extension's batched backfill: the backfill and update triggers enumerate the collection by document reference instead of loading it into memory, chunk it into 50 document ids per Cloud Task, run one task at a time, and embed each chunk in provider-sized batches with a single API call per batch. Documents are marked `BACKFILLED` or `FAILED_BACKFILL` in one batched write, and a failed batch no longer fails the task.
- Restored the index metadata gate at `_<instance id>/index`: a backfill or update pass runs only when the embedding provider, the vector dimension or the input/output field names differ from the last recorded pass, so a redeploy that changes none of them no longer re-embeds the whole collection. Unlike the extension, the progress counters are merged into that document rather than replacing it, so the gate survives its own first pass.
- With both `DO_BACKFILL` and `UPDATE_ON_CONFIGURE` set, only the backfill pass is enqueued. The two passes share one task thread on `_<instance id>/index`, and the backfill pass covers every document the update pass would.
91 changes: 53 additions & 38 deletions kits/firestore-vector-search/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,9 +150,9 @@ This kit is version 0.1.3 of the extension repackaged as an npm package, and it
the least literal of the ports. The seven functions, the Firestore vector index,
the query document collection and the callable all survive with their names and
settings intact, so a `.env` copied from your installed instance needs no value
changes. Multimodal embedding, the backfill, and the shape of the status field
written onto your documents all changed, so read this before you point the kit at
a collection an installed instance has already embedded.
changes. Multimodal embedding and the shape of the status field written onto
your documents both changed, so read this before you point the kit at a
collection an installed instance has already embedded.

### `EMBEDDING_PROVIDER: multimodal` is not implemented

Expand All @@ -178,8 +178,9 @@ the extension's behaviour, because matching it means writing nothing at all.
### You set `INSTANCE_ID` yourself, and it names the query collection

The extension derived its instance id at install and used it for the query
collection (`_<instance id>/index/queries`), the index metadata document
(`_<instance id>/index`) and its task queues. Here `INSTANCE_ID` is a setting you
collection (`_<instance id>/index/queries`), the index metadata and backfill
progress document (`_<instance id>/index`, with its `enqueues` subcollection)
and its task queues. Here `INSTANCE_ID` is a setting you
provide, and it must match this instance's key in the `instances` map in
`firebase.json`. To keep serving the query documents your clients already write
to, set it to your installed instance's id. The four task queue names can also be
Expand All @@ -198,35 +199,40 @@ whatever `EMBEDDING_PROVIDER` is set to. If either does not exist, `firebase
deploy` prompts you for a value and fails outright when running
non-interactively (CI). Create the one you do not need with a placeholder value.

### `UPDATE_ON_CONFIGURE` now re-embeds on every deploy

This setting was declared by the extension but never read. Reconfiguring an
installed instance re-embedded documents only when the provider, the vector
dimension or the input/output field names had actually changed, which the
extension tracked in its index metadata document.

The kit keeps no such metadata and does no comparison. `UPDATE_ON_CONFIGURE: true`
enqueues a full re-embed of every document that already has an embedding after
*every* `firebase deploy`, whether anything relevant changed or not, and
`DO_BACKFILL: true` embeds the whole collection after the first deploy. On a large
collection that is a large Vertex AI or OpenAI bill per deploy. Set
`UPDATE_ON_CONFIGURE: false` and re-embed deliberately when you change providers.

### Backfill is one task per document, and reads the collection in one go

The extension chunked the collection into batches sized to the provider (16
documents per OpenAI call), embedded each batch in a single API call, and tracked
progress in its metadata document. The kit reads the entire collection with one
`get()` and enqueues one Cloud Task per document, each of which embeds one
document with one API call.

Two consequences. A collection large enough that a single `get()` does not fit in
the trigger's 512 MiB will fail the backfill outright, and there is no
resume-from-progress. Backfilling *n* documents now costs *n* task invocations and
*n* embedding calls rather than *n*/batch size.

There is also no install-time progress reporting, since there is no extension
install UI to report into. Watch the function logs instead.
### `UPDATE_ON_CONFIGURE` is read, and the backfill gate is stricter than the extension's

This setting was declared by the extension but never read: its update pass was
gated on `DO_BACKFILL` instead. The kit reads `UPDATE_ON_CONFIGURE`, so the two
passes are controlled independently — `DO_BACKFILL` after the first deploy,
`UPDATE_ON_CONFIGURE` after every redeploy.
Comment on lines +205 to +207

@cabljac cabljac Sep 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this sentence doesn't match what the code does. initVectorSearch runs on both afterFirstDeploy and afterRedeploy (src/index.ts:118-123), and handleInit returns after enqueuing the backfill whenever doBackfill is on (src/handlers.ts:207-212), so with DO_BACKFILL=true it is the backfill trigger that gets enqueued on every redeploy too, and UPDATE_ON_CONFIGURE never gets a look in. Lines 301-302 already say "after your first deploy and after every redeploy", so the two paragraphs contradict each other.

That also makes the one-pass rule a deviation worth naming. The extension maps install to backfillTrigger and configure to updateTrigger (extension.yaml:400-406), and the update pass only touches documents that already have an embedding, so a reconfigure that changed INPUT_FIELD_NAME left never-embedded documents alone. Here it backfills them as well. Probably the right call given the shared thread, but it changes which documents get written on a redeploy, so it belongs in the Differences list rather than only in the handleInit comment.

One more thing this section should say, because "redeploying without changing anything enqueues nothing" is only true from the second kit deploy. On a migrated instance the first deploy re-embeds the whole collection: the extension's _<instance id>/index document holds only the progress counters after any pass that enqueued tasks (its set() replaced the config fields), so the gate opens, and the pass then treats every extension-written document as unprocessed because it reads status.state (src/backfill.ts:470-472) where the extension wrote status.<instance id>.state. That is a bigger bill than the extension's reconfigure, which skipped documents the write trigger had marked COMPLETED and re-embedded only the ones its backfill had marked BACKFILLED. With DO_BACKFILL=true it lands on the first kit deploy, so people should expect it. Worth checking my reading of the status path there, since that is the #3011 shape and I may be behind on where that landed.

Could you reword this paragraph to what actually happens (both settings act on every deploy, backfill wins when both are on), and add the install/configure mapping and the first-deploy re-embed to the Differences list?


Both passes are then gated on the index metadata document at
`_<instance id>/index`, as the extension's were: a pass runs only when the
embedding provider, the vector dimension or the input/output field names differ
from what the last pass recorded there. Redeploying without changing any of them
enqueues nothing and costs nothing.

The two passes share that document as their task thread, so only one of them
runs per deploy: with both settings on, the backfill pass runs, which covers
every document the update pass would have (the update pass is the same
eligibility rule plus "and already has an embedding").

The extension's gate did not survive its own first pass, because the progress
counters it wrote to the same document replaced the recorded configuration. The
kit merges instead, so the comparison fields persist and the gate holds on every
later deploy. To force a full re-embed without changing any setting, delete the
`_<instance id>/index` document; its `queries` subcollection is untouched, so
Comment on lines +219 to +224

@cabljac cabljac Sep 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section names two deviations (the merge, and the one-pass rule), but reading backfill.ts against the helper package I count more, and the rule is that every one is documented here so a reader can decide whether it matters to them. The ones I can see:

  • Every enqueue document is committed before the first task is dispatched, and the remainder is always committed (backfill.ts:185-216). The extension dispatched task-1 before its own enqueue document was in the batch (trigger.ts:70-83) and never committed the chunks past the last multiple of 50 (trigger.ts:89), so 51 to 99 chunks lost the tail.
  • Update-pass failures stay on their own document (backfill.ts:431-438, 392-398). The extension dropped rejected results and shifted the rest, so one document could receive another's embedding.
  • A batch result shorter than its input marks the trailing documents FAILED_BACKFILL. The extension marked them BACKFILLED with no new vector. Side effect of the fix above, but observable.
  • A document id with no document (a subcollection parent) is skipped (backfill.ts:462). The extension's shouldBackfill threw a TypeError there and the task retried until the queue gave up.
  • The update pass records all four comparison fields on the metadata document (handlers.ts:324-333). The extension recorded only embeddingProvider for that pass, so even without the set() wipe the other three comparisons would have read undefined and its gate would have opened on every reconfigure.
  • getValidDocs collects inside the transaction callback (backfill.ts:453-484). The extension accumulated outside it, so a transaction retry double-counted.

All of these look justified to me under "matching would reproduce a bug", and the PR body already argues most of them. Could you add each to this list with one line on what the extension did? The handleInit comment and the PR body are not where a user will look.

the query documents your clients write to survive.

### There is no install-time progress reporting

The extension reported backfill progress and failures through the extension
install UI (`setProcessingState`). There is no such surface for a kit, so
progress is visible in the function logs and in the progress fields on
`_<instance id>/index` (`backfillJobsTotal`, `backfillJobsProcessed`,
`backfillJobsSkipped`, `backfillJobsFailed`, `backfillStatus`) instead. One
document per chunk is written under `_<instance id>/index/enqueues`, as the
extension did, each carrying its chunk of document ids and its own status.

### The `status` field on your documents is a different shape

Expand All @@ -243,10 +249,12 @@ status: { state: "COMPLETED" }
status: { state: "ERROR", message: "<error message>" }
```

The states themselves are narrower too: `PROCESSING` and `BACKFILLED` are no
longer written, only `COMPLETED` and `ERROR`. Anything reading
`status.<instance id>.state`, or a security rule or index keyed to it, needs
updating. The field name is still `STATUS_FIELD_NAME`, defaulting to `status`.
The states themselves are narrower too: `PROCESSING` is no longer written. The
write triggers write `COMPLETED` or `ERROR`, and the backfill and update passes
write `BACKFILLED` or `FAILED_BACKFILL` alongside a `completeTime`, as the
extension did. Anything reading `status.<instance id>.state`, or a security rule
or index keyed to it, needs updating. The field name is still
`STATUS_FIELD_NAME`, defaulting to `status`.

Query documents carry the flat shape too, with a `request` record alongside the
state:
Expand Down Expand Up @@ -316,6 +324,13 @@ for; the Firebase CLI grants these for you.
`onSuccess`, `onError` and `onCompletion` under
`firebase.extensions.firestore-vector-search.v1.*` but never published any of
them, and the kit publishes none either. `EVENTARC_CHANNEL` is not read.
- The backfill and update passes still enumerate the collection by document
reference, chunk it into 50 document ids per Cloud Task, run one task at a
time, and embed each chunk in provider-sized batches with a single API call per
batch (16 documents per OpenAI call). A document whose input is not a string is
skipped, as is one whose status is already set to anything other than
`BACKFILLED`. A failed batch marks its documents `FAILED_BACKFILL` and the task
still succeeds.
- The indexed collection is still `COLLECTION_NAME` (default `products`), the
input, output and status fields still default to `input`, `embedding` and
`status`, and embeddings are still written as native Firestore vectors.
Expand Down
Loading
Loading