Repository navigation
A sync or a new document waits behind the whole extraction backlog: jobs are claimed strictly by run_at #1300
Description
What happens, and what you expected
claim_one orders by run_at only (crates/utopia-store/src/jobs.rs, ORDER BY run_at). Every job kind shares that one queue. So with a few hundred documents waiting for extraction, an hourly sync_source job and the process_document jobs for newly synced items wait until every older extract_document job has run.
On a local model that was days. No source synced for about 10 hours, and 423 newly synced items stayed pending, so they weren't searchable. A full-text rebuild re-queues its documents at the back in the same way.
Expected: cheap intake work (sync_source, process_document, embed_ontology) runs ahead of, or beside, long LLM work, so new material is searchable while the graph catches up.
How to see it
- Queue a few hundred documents for extraction against a slow chat model.
- Leave a GitHub-issues source on a 60-minute interval.
SELECT kind, min(run_at) FROM jobs WHERE status='queued' GROUP BY 1;shows the sync job behind the extraction backlog. It doesn't run until the backlog drains.
Version
0.1.0-rc9 (ghcr image). The code is unchanged on dev.
How it runs
docker compose on macOS
Models
qwen3-8b through Ollama's OpenAI-compatible endpoint; bge-m3 embeddings
Ideas
Any one of these would do:
- a
prioritycolumn ordered beforerun_at; - a separate worker lane for jobs that make no chat call;
- enqueueing intake jobs with an earlier
run_at.
Locally we use a BEFORE INSERT trigger that backdates sync_source, process_document and embed_ontology by a year. It works, but the fix belongs in the queue itself.
Activity
WaylandYang commentedon Oct 18, 2026
Thank you for the detailed report and the reproduction query. Confirmed on
dev:claim_one(crates/utopia-store/src/jobs.rs:374) orders only byrun_at, and the only claim index isjobs_claim_idx (run_at)(migrations/0001_core.sql:100), so async_sourceorprocess_documentqueued now waits behind every olderextract_document.One more thing a fix has to cover: an extraction job keeps its worker slot while it waits for the model's concurrency permit (
run_workercaps atworker_concurrency, 64 by default). Ordering alone only decides who gets the next free slot, and on a slow model that slot can be a long time coming, so a small reserved lane for jobs that make no chat call may be needed alongside the ordering.A PR is welcome. The shape we'd lean towards is a
priorityderived fromkind(so no call site changes), an index on(priority, run_at) WHERE status = 'queued', and a test showing a latersync_sourceis claimed before an earlierextract_document. Please say here whether you also plan the reserved lane before you start. Your #1304 fixes one source of the backlog and can go in on its own.QuintinBotes commentedon Oct 18, 2026
Thanks, that matches what I see. Here
worker_concurrencyis 8, and every worker is usually anextract_documentwaiting on the model permit.As a stopgap, a local
BEFORE INSERTtrigger backdatessync_sourceandprocess_document, so they are claimed first. With it in place, the hourly GitHub syncs today still ran only when a worker freed up: at 09:54, 10:59, 11:59, 13:02, 14:07, 15:11 and then 17:03, so one hour was skipped. So I agree that ordering alone isn't enough on a slow model and a small reserved lane is needed too. The shape you describe (priorityderived fromkind, an index on(priority, run_at) WHERE status = 'queued', and the claim test) looks right to me.I haven't started a PR for this; if I pick it up I'll say so here first, and say whether it includes the lane.
QuintinBotes commentedon Oct 18, 2026
I'll pick this up, as two PRs so each stays one reviewable change:
prioritycolumn onjobs, set fromkind(so no call site changes), an index on(priority, run_at) WHERE status = 'queued',claim_oneordering bypriority, run_at, and a test that a latersync_sourceis claimed before an earlierextract_document. The migration takes the next free number ondev(0114 today).For "no chat call" I'd start with
sync_source,process_document,hydrate_rss_entry,embed_ontology,build_vector_index,materialize_inferencesandnoop. Is that the list you'd use?One question before I write the migration: does a
prioritycolumn onjobscount as a data model change that needs an ADR, or is the queue internal enough that the PR description is enough?WaylandYang commentedon Oct 19, 2026
Thanks, the two-PR split is right.
Kinds with no chat call. Your list matches what
dispatchdoes ondev; please addmemory_ingest(it embeds and re-indexes arememberdocument and queues its extraction). Two notes:embed_ontologyandmemory_ingestwait on the embedding gate, not the chat gate.process_documentcan call the OCR or transcription reader for scans and recordings. That isn't the chat gate either, and MinerU waits withDeferredinstead of holding the slot.Everything else (
extract_document,align_*,resolve_*,adjudicate_entities,govern,propose_*,explore_mappings, errata and implication reads) makes chat calls.ADR. No ADR is needed: the queue is internal state, not the knowledge data model or the API, so the PR description is enough. Take the next free migration number when you rebase; 0114 has since landed on
dev.#1316. @linhongyu510 has meanwhile opened #1316, which orders by a
CASEonkindinclaim_one. Since you picked this up here first and the agreed shape includes the index and a claim test, let's continue with your PR. @linhongyu510, thank you for jumping on it; a review of QuintinBotes's PR would be very welcome.QuintinBotes commentedon Oct 19, 2026
Thanks, I've added
memory_ingest, and the claim order is up as #1324. There's no ADR; it uses migration 0116 because #1318 holds 0115.One question on the lane before I write it. I'd make it two slots on top of
worker_concurrencythat only claimjob_priority(kind) = 0: when every regular slot is taken,run_workermay still claim a no-chat job into a free lane slot. The alternative is to carve the lane out ofworker_concurrency. That leaves model workworker_concurrency - 2slots, which is nothing at 1 or 2 and would need a special case. The cost of the extra slots is thatworker_concurrencyno longer bounds the total: it becomes the bound on everything else, plus two. The test would sit in its own binary, sincerun_workerclaims any queued job. Withworker_concurrencyat 1 and a chat job holding the slot, async_sourcequeued after it still runs.Does adding the lane on top of
worker_concurrencywork for you, or would you rather it come out of it?WaylandYang commentedon Oct 20, 2026
Thanks for #1324; it's in.
On the reserved lane: on top of
worker_concurrency, please. Carving the lane out breaks at 1 or 2, as you say, and the lane exists because the regular slots are parked on the model permit, which is a separate limit. Two extra slots for short intake jobs also fit comfortably in the default connection pool (32).A few details:
job_priority(kind) = 0; regular slots keep claiming anything, since the claim order already puts intake first.worker_concurrencyis described (the system settings text and the docs), say it bounds model work, plus two slots reserved for intake jobs.process_document(a big transcription, for example) can still hold a lane slot. That's acceptable for now.The test in its own binary, with
worker_concurrencyat 1, sounds right.WaylandYang commentedon Oct 22, 2026
Fixed on dev by #1324 (intake kinds are claimed ahead of model work, migration 0116) and #1349 (two worker slots stay free for jobs that make no chat call, so a sync runs even while extractions hold every regular slot).
materialize_typedwas left in the model-work bucket; that follow-up is #1351 / #1358.