changes: wait for a branch to move instead of polling

Problem

A client following a branch has to poll. On an OmniGraph 0.13.0 server reached over HTTPS from a laptop, a caught-up GET /graphs/{id}/changes?cursor= costs 0.45 s, the same as a limit 0 head probe. Polling every 30 s per followed branch, per window, costs that much server time and still leaves the view up to 30 s behind.

Proposed change

GET /graphs/{id}/changes?cursor=…&wait=<seconds>, or a branch-head read with wait. It answers as soon as the branch moves past the cursor, or when the wait elapses with caught_up: true and the same cursor. Constraints:

The wait is bounded (for example at most 60 s).
Waiters per actor are bounded (429 beyond).
The answer is exactly what a poll would return.
The server sees its own publications at once. It sees other processes' publications through the version probe it already makes, run once per graph for all waiters rather than once per client.

Alternatives considered

Shorter polling. It costs more requests.
Server-sent events or a WebSocket. A new connection model.
An ETag on branch reads keyed by the head, with 304. This saves the body but not the request.

Related: #1048.

Summary

A client that follows a branch can only poll GET /graphs/{id}/changes. At 4ba904d3afe7824f6981878bd35c33817b7fd831 the route has no wait parameter and no push channel. A caught-up poll returns blocks: [], the same cursor, and caught_up: true.

The 0.45 s figure was not reproduced. On a local file server the same call takes about 1 ms. The report used OmniGraph 0.13.0 over HTTPS from a laptop. This run used a loopback HTTP server on local files. The missing wait parameter is present at tag v0.13.0 and at this main.

The useful design is a bounded long-poll on GET /changes. When the wait ends, the server runs the existing poll once and returns that body. A shared probe per branch covers other processes. A new in-process signal covers this process's own publications. Lance 13.0.0 does not provide that wait.

No other open issue has this scope. This issue stays open.

Reproduction

Main SHA: 4ba904d3afe7824f6981878bd35c33817b7fd831 (4ba904d3, "feat(engine): expose ann diagnostics and budget native writer buffers (#1086)"). git fetch origin main returned the same SHA.

Environment: Linux, rustc 1.97.1. The debug omnigraph and omnigraph-server binaries in this checkout were built on 2026-10-20 14:48 UTC, after that commit, and no server or engine source file is newer than the binaries. omnigraph version prints omnigraph 0.14.0, internal-schema 16, http-contract 0.13. The server is omnigraph-server --cluster file:///tmp/og1056/data --bind 127.0.0.1:8080 with a static bearer for actor act-alice and a policy that allows read and change. The graph id is demo. One Person node is on main. Transport is HTTP on localhost. Storage is the local filesystem. This is not the reported laptop-to-HTTPS path, and it is not object storage.

I did not change the tree.

Claim: `/changes` and the branch-head read have no wait and no push channel

Reproduced, in code, in OpenAPI, and on the running server.

CHANGE_FEED_PARAMS allows branch, cursor, start, page_token, limit, kind, type, op, and set. There is no wait.

omnigraph/crates/omnigraph-server/src/handlers.rs

Lines 4827 to 4837 in 4ba904d

    
           pub(crate) const CHANGE_FEED_PARAMS: &[&str] = &[ 
        
               "branch", 
        
               "cursor", 
        
               "start", 
        
               "page_token", 
        
               "limit", 
        
               "kind", 
        
               "type", 
        
               "op", 
        
               "set", 
        
           ];

The same list is at tag v0.13.0 (handlers.rs around the CHANGE_FEED_PARAMS constant). An unknown query parameter is HTTP 400. The handler comment says an unknown parameter is never ignored.

GET /graphs/demo/changes?cursor=<cursor>&wait=1 returned:

400 {"error":"unknown query parameter 'wait'","code":"bad_request"}

GET /openapi.json on this server lists the same parameter names. The body has no text/event-stream and no websocket route. GET /snapshot is the branch metadata read. Its query is the branch name. It has no wait and no ETag. ETag and If-None-Match exist for Blob bytes, not for a branch head.

omnigraph changes poll --help at this SHA, after #1090, has --target, --graph, --branch, --cursor, --start, --limit, --kind, --type, --op, --set, and --json. It has no --wait. The remote client in poll_changes_page sends that same query set.

Claim: a caught-up `GET /changes?cursor=` returns `caught_up: true` and the same cursor

Reproduced.

GET /graphs/demo/changes?start=now returned HTTP 200 in 0.009 s:

{"blocks":[],"cursor":"<opaque>","caught_up":true}

Fifteen further calls with that cursor, after one warmup call:

	seconds
min	0.000878
median	0.001062
mean	0.001190
max	0.002278

Every call was HTTP 200. Every body had blocks: [], caught_up: true, and the same cursor. next_page_token was absent. That matches ChangeFeedOutput: caught_up is present with cursor on a terminal page, and it is true when the page reached the captured head.

omnigraph/crates/omnigraph-api-types/src/lib.rs

Lines 1032 to 1048 in 4ba904d

    
           /// One bounded feed poll result. 
        
           #[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] 
        
           pub struct ChangeFeedOutput { 
        
               pub blocks: Vec<ChangeBlockOutput>, 
        
               /// Continue this poll's captured cut. Absent on a terminal page. 
        
               #[serde(default, skip_serializing_if = "Option::is_none")] 
        
               pub next_page_token: Option<String>, 
        
               /// Durable caller-owned cursor, advanced only over complete commits and 
        
               /// returned only on a terminal page — an interrupted poll never advances 
        
               /// it. 
        
               #[serde(default, skip_serializing_if = "Option::is_none")] 
        
               pub cursor: Option<String>, 
        
               /// Present with `cursor` on a terminal page: true when the page reached 
        
               /// its captured head, false when more complete commits already wait. 
        
               #[serde(default, skip_serializing_if = "Option::is_none")] 
        
               pub caught_up: Option<bool>, 
        
           }

The empty-chain path in changes/feed.rs mints that cursor and sets caught_up from cut_commit_id == cut.head.

Claim: a caught-up poll costs 0.45 s, the same as a `limit 0` head probe

Not reproduced. The report's 0.45 s was not measured here. The setup is not comparable.

GET /changes?limit=0 is not a head probe. It returned:

400 {"error":"change page limit must be greater than zero","code":"bad_request"}

I treated the head probe as POST /graphs/demo/query with this source:

query probe() {
  match { $p: Person }
  return { $p.name }
  limit 0
}

The first call took 0.031 s and returned row_count: 0, rows: [], and the branch head graph_commit_id. The next 15 calls:

	seconds
min	0.001917
median	0.002163
mean	0.002232
max	0.002765

GET /graphs/demo/snapshot over 15 calls: min 0.000965 s, median 0.001050 s, mean 0.001066 s, max 0.001360 s. The body is branch name, manifest version, schema stamp, and dataset rows. It is not a change page.

On this host the caught-up poll and the limit 0 query are both about 1–2 ms. They are the same order of magnitude. They are not 0.45 s. I did not run the reported HTTPS laptop path, and I did not use OmniGraph 0.13.0 as the server binary. #752 reports a trivial served read of about 0.35 s on a remote server, with about 0.10–0.15 s of that in the round trip. That is the same class of cost as the 0.45 s figure. It is not a measurement from this run.

Claim: the server sees its own publication on the next request, and another process cannot publish while this server holds the cluster lock

Partially reproduced.

Same process: one mutate through this server (insert Person name grace) returned the new commit. The next GET /changes?cursor=<previous cursor> returned that insert, caught_up: true, and a new cursor, in 0.027 s. The publication was visible to the next request on the same server. Nothing was waiting on the connection. This does not show a wake of an in-flight request.

Other process: omnigraph mutate against file:///tmp/og1056/data/graphs/demo.omni while the server was up exited 1 before a write:

[state_lock_held] __cluster/lock.json: cluster state lock already exists
(lock id 01M5CYPRKSB9TWSH6TX41YB2SY)

command_outcome.effects was none. I did not observe a second process move the branch.

What the code does today

A caught-up poll still probes the manifest. poll_change_feed takes the warm coordinator when the request names the bound branch. It calls probe_latest_incarnation. If that incarnation matches the held one, it reuses the cached cut. A miss refreshes the manifest and drops the cut cache.

omnigraph/crates/omnigraph/src/db/omnigraph.rs

Lines 2690 to 2756 in 4ba904d

    
               pub async fn poll_change_feed( 
        
                   &self, 
        
                   request: crate::changes::ChangeFeedRequest, 
        
               ) -> Result<crate::changes::ChangeFeedPage> { 
        
                   let branch = request 
        
                       .branch 
        
                       .as_deref() 
        
                       .filter(|branch| *branch != "main") 
        
                       .map(str::to_string); 
        
                   // Warm same-branch capture, mirroring `resolve_target_inner`: a poll of 
        
                   // the branch this handle is already bound to reuses the warm coordinator 
        
                   // (no cold manifest open, no lineage re-fold), so a caught-up poll's cost 
        
                   // does not grow with commit history. A different branch, or a stale 
        
                   // probe, falls back to the cold open / write-lock refresh. 
        
                   let requested = branch.as_deref(); 
        
                   let cut = 
        
                       { 
        
                           let coord = self.coordinator.read().await; 
        
                           if requested == coord.current_branch() { 
        
                               let held = coord.manifest_incarnation(); 
        
                               if coord.probe_latest_incarnation().await?.matches(&held) { 
        
                                   // The cut is a pure projection of the manifest, so while 
        
                                   // the incarnation (including the named BranchIdentifier) 
        
                                   // is unchanged the cached cut is exactly valid — a 
        
                                   // caught-up poll then pays O(1) CPU instead of re-cloning 
        
                                   // the whole lineage projection and re-walking to genesis 
        
                                   // on every poll. 
        
                                   let cached = self.feed_cut_cache.read().await.as_ref().and_then( 
        
                                       |(incarnation, cut)| { 
        
                                           (incarnation == &held && cut.branch.as_deref() == requested) 
        
                                               .then(|| Arc::clone(cut)) 
        
                                       }, 
        
                                   ); 
        
                                   match cached { 
        
                                       Some(cut) => cut, 
        
                                       None => { 
        
                                           let cut = Arc::new(coord.build_change_feed_cut().await?); 
        
                                           *self.feed_cut_cache.write().await = Some((held, Arc::clone(&cut))); 
        
                                           cut 
        
                                       } 
        
                                   } 
        
                               } else { 
        
                                   drop(coord); 
        
                                   let mut coord = self.coordinator.write().await; 
        
                                   if requested == coord.current_branch() { 
        
                                       let held = coord.manifest_incarnation(); 
        
                                       let mut refreshed = false; 
        
                                       if !coord.probe_latest_incarnation().await?.matches(&held) { 
        
                                           coord.refresh().await?; 
        
                                           refreshed = true; 
        
                                       } 
        
                                       let refreshed_incarnation = coord.manifest_incarnation(); 
        
                                       let cut = Arc::new(coord.build_change_feed_cut().await?); 
        
                                       drop(coord); 
        
                                       if refreshed { 
        
                                           self.invalidate_read_caches().await; 
        
                                       } 
        
                                       *self.feed_cut_cache.write().await = 
        
                                           Some((refreshed_incarnation, Arc::clone(&cut))); 
        
                                       cut 
        
                                   } else { 
        
                                       Arc::new(coord.capture_change_cut(requested).await?) 
        
                                   } 
        
                               } 
        
                           } else { 
        
                               Arc::new(coord.capture_change_cut(requested).await?) 
        
                           }

The probe for main checks that manifest versions v and v+1 exist, and v+2 when v+1 exists. A gap, a missing v, or an error falls back to Dataset::latest_version_id, which lists versions. A named branch also reads the native branch identifier, so a delete and recreate does not match on the version number alone.

omnigraph/crates/omnigraph-catalog/src/lib.rs

Lines 932 to 1013 in 4ba904d

    
           /// The latest attached version of `dataset`'s location (main or the branch 
        
           /// tree it is checked out on), asked for by name from the version it holds. 
        
           /// 
        
           /// `latest_version_id` lists `_versions/`. On a store whose listing is 
        
           /// ordered (S3, GCS, Azure) that is one request returning up to a thousand 
        
           /// keys however few of them are new, and a warm read pays it on every query. 
        
           /// Attached versions are contiguous: Lance writes version `v + 1` only with a 
        
           /// put-if-absent over a read of `v`, and nothing deletes a version of a live 
        
           /// branch's `__manifest`, so while the held version `v` exists the first 
        
           /// absent name above it ends the history. (`cleanup` collects a retired 
        
           /// branch's whole tree; the named-branch probe's identifier read catches 
        
           /// that.) A reclamation of old `__manifest` versions must keep every version 
        
           /// above one a live handle holds. 
        
           /// 
        
           /// The probe HEADs `v` and `v + 1` together, then `v + 2` when `v + 1` 
        
           /// exists. `v` must answer: a removed or unreadable `__manifest` answers "not 
        
           /// found" for every name, and only the listing reports it as the failure it 
        
           /// is. A gap of more than two versions, a detached handle, an absent `v` or 
        
           /// any error falls back to the listing, which stays the authority. Like the 
        
           /// listing, the probe relies on the store showing a completed write to a 
        
           /// later request. 
        
           async fn latest_attached_version(dataset: &Dataset) -> lance::Result<u64> { 
        
               let held = dataset.version().version; 
        
               if !lance_table::format::is_detached_version(held) { 
        
                   let scheme = dataset.manifest_location().naming_scheme; 
        
                   let base = dataset.branch_location().path; 
        
                   let store = dataset.object_store(None).await?; 
        
                   let [at, next, after] = [held, held + 1, held + 2].map(|v| scheme.manifest_path(&base, v)); 
        
                   let (held_present, next_present) = futures::join!(store.exists(&at), store.exists(&next)); 
        
                   match (held_present, next_present) { 
        
                       (Ok(true), Ok(false)) => return Ok(held), 
        
                       (Ok(true), Ok(true)) => { 
        
                           if let Ok(false) = store.exists(&after).await { 
        
                               return Ok(held + 1); 
        
                           } 
        
                       } 
        
                       _ => {} 
        
                   } 
        
               } 
        
               dataset.latest_version_id().await 
        
           } 
        
           async fn probe_dataset_latest_incarnation( 
        
               dataset: &Dataset, 
        
               active_branch: Option<&str>, 
        
           ) -> Result<ManifestIncarnation> { 
        
               if active_branch.is_none() { 
        
                   return Ok(ManifestIncarnation { 
        
                       version: latest_attached_version(dataset) 
        
                           .await 
        
                           .map_err(OmniError::storage)?, 
        
                       e_tag: dataset.manifest_location().e_tag.clone(), 
        
                       timestamp_nanos: Some(dataset.manifest().timestamp_nanos), 
        
                       branch_identifier: lance::dataset::refs::BranchIdentifier::main(), 
        
                   }); 
        
               } 
        
               let branch = active_branch.expect("named-branch arm checked above"); 
        
               // A named branch's native identifier is its lifetime witness. Pair it with 
        
               // the branch-local latest version instead of loading the manifest body: 
        
               // version changes catch ordinary commits, while the identifier catches a 
        
               // same-source delete/recreate even when version, e-tag, and timestamp all 
        
               // repeat. Read the version first so a recreation between the two probes 
        
               // yields the replacement identifier rather than a false match to the held 
        
               // lifetime. 
        
               let held = async { 
        
                   let version = latest_attached_version(dataset) 
        
                       .await 
        
                       .map_err(|error| branch_ref_error(error, branch))?; 
        
                   let native = dataset.manifest().branch.as_deref().ok_or_else(|| { 
        
                       OmniError::manifest_internal("named coordinator has no native manifest ref") 
        
                   })?; 
        
                   let branch_identifier = 
        
                       crate::branch_control::get_live_manifest_branch_contents(dataset, native) 
        
                           .await? 
        
                           .identifier; 
        
                   Ok::<_, OmniError>(ManifestIncarnation { 
        
                       version, 
        
                       e_tag: None, 
        
                       timestamp_nanos: None, 
        
                       branch_identifier, 
        
                   }) 
        
               }

Each request runs that probe. There is no shared probe task and no list of waiters.

A publication from this handle updates the coordinator in place before the write returns (commit_updates_on_branch_with_expected holds db.coordinator and publishes through it). The next request's probe then matches the new held version. There is no Notify (or other wake) at that publication for a parked reader to await. "At once" means the next request on this handle sees the new head. It does not mean an in-flight request is told.

omnigraph/crates/omnigraph/src/db/omnigraph/table_ops.rs

Lines 1373 to 1400 in 4ba904d

    
           // Choose and publish through one coordinator lock. A same-name cached 
        
           // coordinator can still hold another incarnation or incomplete lineage, 
        
           // so only an exact captured view is reused without a fresh open. 
        
           let mut active = db.coordinator.write().await; 
        
           let published = if branch == active.current_branch() { 
        
               let captured_view_matches = active 
        
                   .branch_identifier() 
        
                   .await 
        
                   .map_err(OmniError::unpublished)? 
        
                   == txn.authority.branch_identifier 
        
                   && active.exact_graph_head() == txn.authority.graph_head 
        
                   && active.version() == txn.base.graph_manifest_version(); 
        
               if !captured_view_matches { 
        
                   // Keep the handle's binding while replacing only stale state. 
        
                   // The publisher still reads fresh authority and applies its CAS. 
        
                   *active = db 
        
                       .open_coordinator_for_branch(branch) 
        
                       .await 
        
                       .map_err(OmniError::unpublished)?; 
        
               } 
        
               publish_prepared( 
        
                   &mut active, 
        
                   &changes, 
        
                   expected_table_versions, 
        
                   lineage_intent, 
        
                   expectation, 
        
                   publication, 
        
               )

The caught-up cost test says the warm path opens no data tables. The manifest term is one version probe, not a fold of history (change_feed_caught_up_poll_manifest_reads_are_flat_in_history). On local files that probe is the ~1 ms above. On an object store the same two existence checks are round trips. That is the cost a poll interval repeats. I did not measure an object store.

Limits a held request would meet

GET /changes is a read. resolve_graph_handle calls ingress::admit, which takes one read observer for the whole handler and response. The process cap is DEFAULT_READ_OBSERVERS = 128. Past that cap the status is 429, server read observer cap 128 exceeded.

spawn_read keeps the handler future alive after the HTTP receiver is dropped. The comment says a lost result waiter cannot drop the engine future. A sleep inside that future would hold one of the 128 slots until the sleep ends, including after the client goes away.

omnigraph/crates/omnigraph-server/src/operations.rs

Lines 409 to 421 in 4ba904d

    
               /// Transfer an already admitted read to a process task. Losing the result 
        
               /// waiter cannot drop its engine future or release its input reservation. 
        
               /// Descendants of this observer may finish after admission closes; no new 
        
               /// observer or queue entry is acquired here. Native tails outside the 
        
               /// engine future remain outside this logical ownership boundary. 
        
               pub(crate) fn spawn_read<T, R, F>(self, reservation: R, operation: F) -> OwnedReadResponse<T> 
        
               where 
        
                   T: Send + 'static, 
        
                   R: Send + 'static, 
        
                   F: Future<Output = Result<T, ApiError>> + Send + 'static, 
        
               { 
        
                   let (sender, receiver) = oneshot::channel(); 
        
                   let response_observer = self.clone();

GET /changes does not call try_admit. These caps do not apply to it today:

per actor in flight: 16 (OMNIGRAPH_PER_ACTOR_INFLIGHT_MAX)
per actor bytes: 4 GiB
global in flight: 64
global bytes: 256 MiB
active actors: 1024

The body timeout is 30 s (OMNIGRAPH_BODY_TIMEOUT_SECONDS). A GET has no body, so that timer does not bound the handler. axum::serve has no request deadline. Every 429 uses Retry-After: 60.

Shutdown grace is 25 s by default (DEFAULT_SHUTDOWN_GRACE). After the signal, in-flight work may finish until that grace. Then the process exits. A 60 s wait is longer than that grace.

Authorization for the feed runs once, before poll_change_feed. A signed data token's lifetime is 60 s to 24 h. A static bearer does not expire. Policy is the graph policy captured for the request.

Design

Use GET /graphs/{id}/changes?cursor=<cursor>&wait=<seconds>.

The server checks the head. If the cursor is already behind the head, it polls immediately. If the branch moves during the wait, it polls once at that wake. If the wait ends with the same head, it polls once and returns that caught-up page: empty blocks, the same cursor, caught_up: true. The response is the body poll_change_feed already returns, including 400, 404, 409, and 410. A second response shape would drift from the poll.

wait=0 is a poll. wait with page_token is 400. A page token continues one captured cut. A wait would mix two cuts. start=now&wait= is the same as a cursor taken at now, then a wait. A cursor that is already behind does not sit for the rest of the wait.

Share one probe per graph branch, not one probe per client. Waiters on main share the main manifest probe. Waiters on another branch need that branch's probe. A probe of main does not see the other branch move. Run the probe only while that branch has waiters. Stop it when the last waiter leaves. The existing check is two existence calls, plus v+2 when v+1 exists, and the branch identifier on a named branch. One result wakes every waiter whose cursor is at or behind that head.

Add an in-process signal at a successful head move on this handle, beside the coordinator update in commit_updates_on_branch_with_expected. That signal does not exist today. It wakes waiters in this process without a probe interval. It is not a lock that other processes honor. The cluster lock remains the writer fence. The probe remains the way a different process's publication is seen. RFC 0036 §10 describes a read-only replica that learns from a manifest probe. This run had one server. A second writer was refused by __cluster/lock.json.

Bounds

Reject wait above a server cap with 400. Do not use 60 s as that cap while shutdown grace is 25 s. A lost connection is safe for the cursor: the cursor advances only when the client receives a terminal page, and a retry of the same cursor is the feed's at-least-once rule. The cost is the slot held until shutdown kills the process. A cap of 20 s sits under the default grace. An operator who raises the shutdown grace and the proxy idle timeout together can raise the cap. The response has no bytes until the end, so a proxy idle timer counts the whole wait. A cap of 60 s sits on the common 60 s proxy idle time, and the proxy can drop the answer.

Count waiters apart from the 128 read observers. Take a read observer only for the poll that builds the answer. A waiter that holds a read observer for the whole wait makes ordinary reads return 429. The per-actor in-flight cap of 16 does not apply to this GET today, so it is not the waiter cap unless the route starts using it on purpose.

When the waiter cap is exceeded, return 429 and Retry-After. That is the existing capacity status (server read observer cap and the workload rejects). 503 means the graph is unavailable or operation admission is closed. A full waiter list is not that condition. Today's Retry-After is the constant 60 for every 429. A waiter refusal should name a delay on the order of the cap, or the client waits 60 s and then polls again.

Disconnect, auth, branch lifetime, schema

Cancel the wait when the connection closes, and release the waiter slot in that cancel. spawn_read does not do this. A wait placed inside that future holds the slot after the client is gone.

Check the bearer and the policy again at wake, before the poll. Do the same check before the timeout page. caught_up: true is a read of the head. A token that expires during the wait, or a policy that no longer allows read, gets 401 or 403 and no blocks.

If the branch is deleted, return the 404 the poll returns. Do not return caught_up: true with the old cursor. The named-branch identifier already treats delete and recreate as a new lifetime. The cursor is bound to that lifetime. There is no branch rename on omnigraph branch at this SHA.

If the new head crosses a schema boundary, return the 409 the poll returns (schema_boundary). #1048 may later narrow that rule. The wait calls the poll. It does not keep its own schema rule. An empty commit still moves the cursor. Wake on any head move, then return the poll page.

Other options

A branch-head wait on GET /snapshot, or on a smaller head read, tells the client that the head moved. The client then calls /changes to get the page. That second call remains. The issue asks for the poll body. Use the head wait only for a caller that wants the head id.

Shorter polling multiplies the probe. It does not remove the gap. The gap is the poll interval.

Server-sent events or a websocket replace the poll with a stream. The stream is a second connection model. It holds a slot with no natural end unless it has the same cap, the same cancel, and the same auth check. With those bounds it is a long-poll under another media type. #806 asks for a Neo4j connector and export. It does not specify this route.

An ETag on the head, with 304, saves the body. It does not save the probe. Blob GET already has ETag. /changes and /snapshot do not. The reported cost is the request, so 304 does not close the issue.

HTTP contract and CLI

#1089 is an open RFC for one HTTP style, applied in contract batches. wait is a new query parameter. A 0.13 server returns 400 for it, as this server did. Old clients that omit wait keep working. New clients that send wait to an old server do not. Put wait in the next contract batch so the Omnigraph-Http-Api value says the server accepts it. Do not add it under 0.13 with no signal.

The CLI flag is --wait <seconds> on changes poll, forwarded by the remote client. Embedded use of a local directory does not pay the remote probe. This local caught-up poll was about 1 ms. The flag still belongs on the command so a server target has one client path. #1090 is merged. The command already takes --target and --graph. It does not take a positional store path.

MCP changes_poll should grow the same parameter when the route does. #1089 asks for MCP names to follow the HTTP operations.

Liability

The long-poll adds held connections, a waiter budget, a disconnect cancel, a second auth check, and one probe task per watched branch. It removes one probe per client per poll interval, and it removes the poll interval as the freshness bound. The in-process signal must stay a wake hint. Invariant 11 and the deny-list reject a process-local lock that is described as a distributed fence. The manifest publication remains the only publication door.

A 60 s wait inside today's read-observer future would add a way for clients to occupy the 128 read slots. That is the failure mode to avoid. The separate waiter budget and the cancel on disconnect are what keep the feature inside the existing bound.

Lance

Lance does not fix this. The pin is unmodified Lance 13.0.0 (Cargo.toml). No open pull request in this repository upgrades Lance. Lance 14 is at v14.0.0-beta.21 (2026-10-19) and is not the pin.

Lance 13.0.0 Dataset has latest_version_id, is_stale, and has_successor_version. The docs on latest_version_id call it a fast check for a changed dataset. has_successor_version is a hidden contiguous-history probe and can return false if an intermediate manifest was removed. None of the three waits or notifies a caller. OmniGraph's probe is its own existence checks plus the listing fallback. It does not call has_successor_version. A search of open Lance pull requests for a dataset watch or version subscription returned nothing. Internal Notify values in Lance 13 are for scans, blobs, and MemWAL, not for a branch head.

https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.21

Related work

changes: continue change feeds across additive schema changes and outside the request's type filter #1048 — a schema boundary stops the feed. The wait returns that 409. It does not change the boundary rule.
changes: read the version history of one entity #1049 — history of one entity. Different read.
performance: a read pinned to a commit costs 0.5–0.9 s more than the same read at its branch #1052 — a commit-pinned read costs more than a branch read. Different cost. Not measured here.
changes: a footprint answer from graph diff, and its filters applied before table opens #1053 — a footprint answer from graph diff. Different read.
api: read answers report the table versions they read and the schema they were answered under #1054 — read answers report table versions and schema. Different fields.
server: several named queries in one request, answered at one snapshot #1055 — several named queries in one request. Different route.
performance: every change page restarts its comparison (re-plan, re-resolve endpoints, re-sort keys) #970 — each change page restarts its comparison. Orthogonal. A caught-up poll does not walk that page.
Neo4J connector, real-time exporter, etc. #806 — Neo4j connector and export. The cross-reference is not a wait design.
performance: capture a write from the process's own last publication instead of re-reading __manifest #837 — a write re-reads state this process wrote. Related to "own publication", and it does not add a reader wake.
Attribute the served read floor (~0.2 s server-side on a trivial read) and expose it as usage #752 — the remote trivial-read floor. Same cost class as the 0.45 s report. Not a wait.
rfc: standardize HTTP API conventions #1089 — HTTP conventions. Open. The contract batch is where wait belongs.
feat(cli)!: replace legacy routing with explicit targets #1090 — merged. CLI targets are --target and --graph. No --wait was added.
rfc: graph access and cross-process writes #1017 — open draft for graph access and cross-process writes. It does not add a feed wait, and it does not lift the single-writer boundary.
rfc: storage outcomes per operation and restartable ownership #1093 — open draft for storage outcomes and restart. It does not add a publication signal for readers.
RFC 0036 §10 — a read-only replica can wake a rebuild from a manifest probe. That wake is for the replica generation, not for a change-feed client.

No merged or open pull request adds wait, a long-poll, or a push channel for branch movement.

Supersession

Nothing supersedes #1056. The issues above share the feed or the read path. None of them is a wait for the branch head with a poll body. The issue stays open.

Test plan

Owner: omnigraph-server HTTP tests for the route, and the engine change-feed owner only if the wake signal is on the handle. Do not add a second fixture for the poll body. Assert that the wait body equals one poll.

Caught-up cursor, no publication, wait at the cap. The status is 200. The body equals a poll: empty blocks, same cursor, caught_up: true. The elapsed time is at least the wait.
Same-server mutate during the wait. The response arrives before the cap. The body equals a poll taken after that commit.
Several waiters on one branch, one external manifest advance. The probe count is one per interval for that branch, not one per waiter.
Client disconnect. The waiter count returns before the cap. A following read is not held by the abandoned wait.
Waiter count above the actor cap. Status 429, Retry-After set, no extra probe.
wait above the cap, and wait with page_token. Status 400.
Branch delete during the wait. Status 404. The body is not caught_up: true.
Schema boundary during the wait. Status 409, the same refusal as a poll (changes: continue change feeds across additive schema changes and outside the request's type filter #1048).
Token expired at wake, and policy no longer allows read. Status 401 or 403. No blocks.
OpenAPI lists wait. The contract header is the batch that introduces it. omnigraph changes poll --wait sends it only to a server target on that contract.

Limitations

I did not reproduce 0.45 s. I did not use HTTPS, a laptop path, or object storage. I did not run the 0.13.0 binary. The parameter list at tag v0.13.0 matches this main. Local times are from one debug server, 15 calls each, after a warmup. I did not leave a failing test in the tree. The 400 on wait=1 is the proof that the parameter is absent.

Navigation Menu

Repository navigation

changes: wait for a branch to move instead of polling #1056

Problem

Proposed change

Alternatives considered

Summary

Reproduction

Claim: `/changes` and the branch-head read have no wait and no push channel

Claim: a caught-up `GET /changes?cursor=` returns `caught_up: true` and the same cursor

Claim: a caught-up poll costs 0.45 s, the same as a `limit 0` head probe

Claim: the server sees its own publication on the next request, and another process cannot publish while this server holds the cluster lock

What the code does today

Limits a held request would meet

Design

Bounds

Disconnect, auth, branch lifetime, schema

Other options

HTTP contract and CLI

Liability

Lance

Related work

Supersession

Test plan

Limitations

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	pub(crate) const CHANGE_FEED_PARAMS: &[&str] = &[
	"branch",
	"cursor",
	"start",
	"page_token",
	"limit",
	"kind",
	"type",
	"op",
	"set",
	];

	/// One bounded feed poll result.
	#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
	pub struct ChangeFeedOutput {
	pub blocks: Vec<ChangeBlockOutput>,
	/// Continue this poll's captured cut. Absent on a terminal page.
	#[serde(default, skip_serializing_if = "Option::is_none")]
	pub next_page_token: Option<String>,
	/// Durable caller-owned cursor, advanced only over complete commits and
	/// returned only on a terminal page — an interrupted poll never advances
	/// it.
	#[serde(default, skip_serializing_if = "Option::is_none")]
	pub cursor: Option<String>,
	/// Present with `cursor` on a terminal page: true when the page reached
	/// its captured head, false when more complete commits already wait.
	#[serde(default, skip_serializing_if = "Option::is_none")]
	pub caught_up: Option<bool>,
	}

	pub async fn poll_change_feed(
	&self,
	request: crate::changes::ChangeFeedRequest,
	) -> Result<crate::changes::ChangeFeedPage> {
	let branch = request
	.branch
	.as_deref()
	.filter(\|branch\| *branch != "main")
	.map(str::to_string);
	// Warm same-branch capture, mirroring `resolve_target_inner`: a poll of
	// the branch this handle is already bound to reuses the warm coordinator
	// (no cold manifest open, no lineage re-fold), so a caught-up poll's cost
	// does not grow with commit history. A different branch, or a stale
	// probe, falls back to the cold open / write-lock refresh.
	let requested = branch.as_deref();
	let cut =
	{
	let coord = self.coordinator.read().await;
	if requested == coord.current_branch() {
	let held = coord.manifest_incarnation();
	if coord.probe_latest_incarnation().await?.matches(&held) {
	// The cut is a pure projection of the manifest, so while
	// the incarnation (including the named BranchIdentifier)
	// is unchanged the cached cut is exactly valid — a
	// caught-up poll then pays O(1) CPU instead of re-cloning
	// the whole lineage projection and re-walking to genesis
	// on every poll.
	let cached = self.feed_cut_cache.read().await.as_ref().and_then(
	\|(incarnation, cut)\| {
	(incarnation == &held && cut.branch.as_deref() == requested)
	.then(\|\| Arc::clone(cut))
	},
	);
	match cached {
	Some(cut) => cut,
	None => {
	let cut = Arc::new(coord.build_change_feed_cut().await?);
	*self.feed_cut_cache.write().await = Some((held, Arc::clone(&cut)));
	cut
	}
	}
	} else {
	drop(coord);
	let mut coord = self.coordinator.write().await;
	if requested == coord.current_branch() {
	let held = coord.manifest_incarnation();
	let mut refreshed = false;
	if !coord.probe_latest_incarnation().await?.matches(&held) {
	coord.refresh().await?;
	refreshed = true;
	}
	let refreshed_incarnation = coord.manifest_incarnation();
	let cut = Arc::new(coord.build_change_feed_cut().await?);
	drop(coord);
	if refreshed {
	self.invalidate_read_caches().await;
	}
	*self.feed_cut_cache.write().await =
	Some((refreshed_incarnation, Arc::clone(&cut)));
	cut
	} else {
	Arc::new(coord.capture_change_cut(requested).await?)
	}
	}
	} else {
	Arc::new(coord.capture_change_cut(requested).await?)
	}

	/// The latest attached version of `dataset`'s location (main or the branch
	/// tree it is checked out on), asked for by name from the version it holds.
	///
	/// `latest_version_id` lists `_versions/`. On a store whose listing is
	/// ordered (S3, GCS, Azure) that is one request returning up to a thousand
	/// keys however few of them are new, and a warm read pays it on every query.
	/// Attached versions are contiguous: Lance writes version `v + 1` only with a
	/// put-if-absent over a read of `v`, and nothing deletes a version of a live
	/// branch's `__manifest`, so while the held version `v` exists the first
	/// absent name above it ends the history. (`cleanup` collects a retired
	/// branch's whole tree; the named-branch probe's identifier read catches
	/// that.) A reclamation of old `__manifest` versions must keep every version
	/// above one a live handle holds.
	///
	/// The probe HEADs `v` and `v + 1` together, then `v + 2` when `v + 1`
	/// exists. `v` must answer: a removed or unreadable `__manifest` answers "not
	/// found" for every name, and only the listing reports it as the failure it
	/// is. A gap of more than two versions, a detached handle, an absent `v` or
	/// any error falls back to the listing, which stays the authority. Like the
	/// listing, the probe relies on the store showing a completed write to a
	/// later request.
	async fn latest_attached_version(dataset: &Dataset) -> lance::Result<u64> {
	let held = dataset.version().version;
	if !lance_table::format::is_detached_version(held) {
	let scheme = dataset.manifest_location().naming_scheme;
	let base = dataset.branch_location().path;
	let store = dataset.object_store(None).await?;
	let [at, next, after] = [held, held + 1, held + 2].map(\|v\| scheme.manifest_path(&base, v));
	let (held_present, next_present) = futures::join!(store.exists(&at), store.exists(&next));
	match (held_present, next_present) {
	(Ok(true), Ok(false)) => return Ok(held),
	(Ok(true), Ok(true)) => {
	if let Ok(false) = store.exists(&after).await {
	return Ok(held + 1);
	}
	}
	_ => {}
	}
	}
	dataset.latest_version_id().await
	}

	async fn probe_dataset_latest_incarnation(
	dataset: &Dataset,
	active_branch: Option<&str>,
	) -> Result<ManifestIncarnation> {
	if active_branch.is_none() {
	return Ok(ManifestIncarnation {
	version: latest_attached_version(dataset)
	.await
	.map_err(OmniError::storage)?,
	e_tag: dataset.manifest_location().e_tag.clone(),
	timestamp_nanos: Some(dataset.manifest().timestamp_nanos),
	branch_identifier: lance::dataset::refs::BranchIdentifier::main(),
	});
	}
	let branch = active_branch.expect("named-branch arm checked above");
	// A named branch's native identifier is its lifetime witness. Pair it with
	// the branch-local latest version instead of loading the manifest body:
	// version changes catch ordinary commits, while the identifier catches a
	// same-source delete/recreate even when version, e-tag, and timestamp all
	// repeat. Read the version first so a recreation between the two probes
	// yields the replacement identifier rather than a false match to the held
	// lifetime.
	let held = async {
	let version = latest_attached_version(dataset)
	.await
	.map_err(\|error\| branch_ref_error(error, branch))?;
	let native = dataset.manifest().branch.as_deref().ok_or_else(\|\| {
	OmniError::manifest_internal("named coordinator has no native manifest ref")
	})?;
	let branch_identifier =
	crate::branch_control::get_live_manifest_branch_contents(dataset, native)
	.await?
	.identifier;
	Ok::<_, OmniError>(ManifestIncarnation {
	version,
	e_tag: None,
	timestamp_nanos: None,
	branch_identifier,
	})
	}

	// Choose and publish through one coordinator lock. A same-name cached
	// coordinator can still hold another incarnation or incomplete lineage,
	// so only an exact captured view is reused without a fresh open.
	let mut active = db.coordinator.write().await;
	let published = if branch == active.current_branch() {
	let captured_view_matches = active
	.branch_identifier()
	.await
	.map_err(OmniError::unpublished)?
	== txn.authority.branch_identifier
	&& active.exact_graph_head() == txn.authority.graph_head
	&& active.version() == txn.base.graph_manifest_version();
	if !captured_view_matches {
	// Keep the handle's binding while replacing only stale state.
	// The publisher still reads fresh authority and applies its CAS.
	*active = db
	.open_coordinator_for_branch(branch)
	.await
	.map_err(OmniError::unpublished)?;
	}
	publish_prepared(
	&mut active,
	&changes,
	expected_table_versions,
	lineage_intent,
	expectation,
	publication,
	)

	/// Transfer an already admitted read to a process task. Losing the result
	/// waiter cannot drop its engine future or release its input reservation.
	/// Descendants of this observer may finish after admission closes; no new
	/// observer or queue entry is acquired here. Native tails outside the
	/// engine future remain outside this logical ownership boundary.
	pub(crate) fn spawn_read<T, R, F>(self, reservation: R, operation: F) -> OwnedReadResponse<T>
	where
	T: Send + 'static,
	R: Send + 'static,
	F: Future<Output = Result<T, ApiError>> + Send + 'static,
	{
	let (sender, receiver) = oneshot::channel();
	let response_observer = self.clone();

changes: wait for a branch to move instead of polling #1056

Description

Problem

Proposed change

Alternatives considered

Activity

ragnorc commented on Oct 20, 2026

Summary

Reproduction

Claim: /changes and the branch-head read have no wait and no push channel

Claim: a caught-up GET /changes?cursor= returns caught_up: true and the same cursor

Claim: a caught-up poll costs 0.45 s, the same as a limit 0 head probe

Claim: the server sees its own publication on the next request, and another process cannot publish while this server holds the cluster lock

What the code does today

Limits a held request would meet

Design

Bounds

Disconnect, auth, branch lifetime, schema

Other options

HTTP contract and CLI

Liability

Lance

Related work

Supersession

Test plan

Limitations

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Claim: `/changes` and the branch-head read have no wait and no push channel

Claim: a caught-up `GET /changes?cursor=` returns `caught_up: true` and the same cursor

Claim: a caught-up poll costs 0.45 s, the same as a `limit 0` head probe