Skip to content

docs(rfc): optimize graphs through the running server - #1186

Open
aaltshuler wants to merge 3 commits into
ModernRelay:mainfrom
aaltshuler:codex/rfc-served-optimization
Open

aaltshuler wants to merge 3 commits into
ModernRelay:mainfrom
aaltshuler:codex/rfc-served-optimization

Conversation

@aaltshuler

Copy link
Copy Markdown
Contributor

Running optimize currently requires direct storage access, which cannot safely run beside the server that owns the graph. This RFC proposes executing it through that server while keeping orchestration, operational history and log collection in the control plane.

The proposal uses one synchronous request, exact completed responses and explicit unknown outcomes after response loss. Durable maintenance journals, original-ID lookup and offline recovery are deferred; native authorization, resource admission and storage settlement remain required. The opening plain-language section explains the problem, responsibilities and tradeoff.

This docs-only PR separates design review from implementation #1177. That draft still implements the earlier durable-invocation protocol and needs alignment; this RFC does not approve or ship it. The only changes are the RFC and its registry entry.

Validation: documentation checker (407 Markdown files), spelling and git diff --check pass. No runtime code changed.

@aaltshuler
aaltshuler marked this pull request as ready for review October 27, 2026 11:06
@aaltshuler aaltshuler changed the title docs(rfc): separate served optimization design from implementation docs(rfc): optimize graphs through the running server Oct 27, 2026
ragnorc added a commit to ragnorc/omnigraph that referenced this pull request Oct 29, 2026
Adapt rather than take ModernRelay#1186's ceilings and list every difference; share
the record's bounds with the deployment receipts; cap the resolution walk
per attempt; name the exit from a full unresolved list; justify the
callback over a two-phase engine API; narrow the containment claim; skip
the artifact write under a pending shutdown and size its limit against the
grace at qualification.
@ragnorc

ragnorc commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Thanks for this draft. I read it next to the remote-optimize RFC (reverted from main in #1203 because it merged before review closed; it is being reopened as a new PR) and against your closing review of #1168, since both drafts answer that review. The two agree on most of the engine contract: the serving owner runs the work, the same gates and one publication, the input commit captured under the gates, a no-work run publishes nothing, disconnect stops waiting not execution, an explicit grant that admin never implies, one operation per cluster refused not queued, source ceilings from descriptors before any rewrite, full-text rebuild separate, a derived-cache failure never erasing a commit, and the route shipping disabled until measured. The other RFC takes four of this draft's rules by name.

Measured against the requirements in your #1168 review, this draft leaves four open:

  1. Durable acceptance and original-id lookup. Your review rejected rfc: served graph maintenance #1168 because "the control-plane design requires durable native operation identity/results and original-ID lookup after an uncertain response; this RFC makes run acceptance ephemeral". This draft keeps acceptance ephemeral (no record, no 202, no lookup) and resolves the conflict by asking LCY-04.2 and DEC-08-76 to be amended to permit unknown, unrecoverable results. That moves the obligation into every client ("must disable automatic POST retries after ambiguous dispatch"), and the operator of a graph written by agents can never learn what one finished run did. The Alternatives section argues against a record that blocks serving until an administrator reconciles, which is feat(maintenance)!: add served optimization and interrupted-operation recovery #1177's shape and is rightly rejected. The other RFC's record does not do that: it stores the intent nonce before the first manifest request, resolves an in-doubt publication from main's own history with commit_id_answers, never gates serving on an unresolved record, and shares the deployment receipts' bounds. The deployment route already works this way (caller-allocated id, 202 after the ledger CAS, lookup by id).
  2. Cancellation and progress. The lifecycle here is refusal, completed report, lost response, error. There is no way to stop a run short of shutdown and no way to observe one. Your review named cancellation among the facts that need one contract.
  3. Bounded writer impact. "Ordinary writes ... may wait for optimization" with no bound, and the ceilings are whole-request refusals. A catalog of a few hundred tables is then either unoptimizable through the server or holds main's writer gates for the whole run (a no-op planning pass over 657 tables measured at 577 s). The other RFC bounds the gate hold, defers a table over its ceiling with its measure, and resumes from a stored position on the next run, so the prefix is reported rather than silent.
  4. The post-commit tail. Your review named the mutable adjacency-artifact write as a shutdown hazard. This draft leaves the tail as it is on main today: the configuration clear fails a run after its commit is durable, the artifact persist runs under the main branch guard, and its stat hard-codes committed: false. The other RFC runs the artifact scan outside every writer gate, bounds the object before and during the build, and never drops an issued write.

Two things here are better than the other RFC's text and are being folded into it: the plain-language summary with the "what the caller receives / what it establishes / what to do next" table, and the sentences that a resource limit must not select another correctness protocol and that an index or storage error is an error, not evidence of an empty column. The per-table unsupported layout refusal becomes a per-table outcome there too, and the note about experimental roots left by #1177 is being carried over, naming __cluster/maintenance.json.

Given that, I would rather have one RFC than two, as both drafts say. If the four points above are ones you want to carry in this draft, it would grow into the same shape; if not, I suggest closing this one once the reopened PR is up, with the folded-in parts credited there. Happy to be argued out of either point.

One mechanical note: the PR Title check wants rfc: optimize graphs through the running server (no docs(rfc): prefix).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants