Repository navigation
rfc: remote optimize through the serving owner - #1190
Conversation
A bounded, manually requested optimize of a served graph's main branch, executed by the process that serves and publishes it: today's gate shape and one publication, a gate-hold bound with an exact outcome per table, a durable operation record under a caller-chosen id before any effect, lookup by that id after a lost response, and restart resolution from the graph's history. Replaces the closed served-maintenance proposal in the form its review asked for.
|
Recommendation: revise. WhyA served graph on object storage cannot run WhatThis pull request is a design document. It adds the RFC and three pending amendments. It changes no executable code. The engine keeps today's schema gate, main branch gate, and table gates, and it stops starting tables after a time bound. The server adds HowAcceptance is a conditional put of the operation record. The run then holds the writer gates, works one table at a time, and publishes the finished tables once. A callback records the intended commit before that publication. On restart, a ImpactThe serving process is the right owner, and the lost-response shape matches deployment. The recovery key, the writer-wait bound, and the per-table result do not yet match the code this RFC cites. Fix those in the document before an implementation follows it. Do not accept pull request #1186 beside this one. Alignment with #1186#1186 (head That smaller contract avoids a second durable record. The server-runtime RFC currently forbids that record. It also leaves the operator unable to learn what a finished run did. #1186 cites control-plane text that requires a native operation id and lookup (LCY-04.2 and DEC-08-76), then asks that text to allow a permanent unknown. Deployment on this server already uses a caller-chosen id, stores acceptance before work, and answers a lost response by that id. This RFC is the contract that fits that pattern, after recovery uses the commit id the publisher writes. Two accepted optimize RFCs would leave two contracts. Bring these rules across from #1186:
Reuse the Keeping today's gates is the right first cut. A schema apply or branch create that queues during the run stalls later writers, because the schema gate is write-preferring ( Blocking1. The live boundary is the cluster lock
Say this boundary in the summary. Keep direct optimize for a stopped server, on local and object storage. 2. The callback site does not yet know the published commit id
Put the hook on the publish attempt, after
3. An in-doubt publication has to stay non-terminalA terminal record is never rewritten. The accepted storage-outcomes RFC already changed this case. An ambiguous manifest request answers A live process therefore has to resolve a DataFusion with no completion evidence still contains the process ( 4. The writer-wait bound omits work that still holds a gateThe schema permit and the table guards drop at
The stated wait is
5. Stable order plus the time bound can skip the dirty tailTables run in request order, then by type key. The next default request starts at the same key. Each clean table is planned again before a later table can start. The 576.8 s figure is a full no-op plan of 657 tables from a remote runner. It is not in this repository. It shows that a 60 s default does not cover that catalog in one request. Today's plan is also up to 8 tables at once ( The deferred list is the continuation. The next full-catalog request, and the CLI, start at the first deferred table and use a new operation id. The gates drop between ids, so writers run between slices. A request that passes 6. The result list drops two current outcomesA drifted table is Full-text coverage stays pending. Stock Lance 7. Lookup, cancel, and a repeated POST need one auth and status rule
POST of the same id and the same request returns 200 with the existing record. Deployment returns 202 until the work is terminal, and 200 when it is done (
8. The four record puts are not definite on this base
An ambiguous acceptance put is the lost answer this record is here to close. Name #1176 as a dependency. Do not start table work, and do not publish, while one of the four puts is in doubt. Read the object back. Equal bytes mean this write landed. Optional
Tradeoff and liabilityThe record is the right place for facts the manifest cannot store: a refusal, a no-op, a cancellation, a deferral, and the principal. It is the wrong place for a second copy of the commit. History already holds that commit. This design removes the need to stop the server, and it removes the blind retry from the closed proposal. It adds one control object, a caller-chosen id, four conditional puts, a 32-record cap, and one active slot. Those are worth keeping. It also adds three costs. The callback can record an id the publisher has not assigned. A terminal state can freeze an in-doubt put. The time bound ignores the branch gate and the next slice of the catalog. Those costs add long-term liability. Five similar operations (cleanup, repair, full-text rebuild, index build, named-branch optimize) should add a One owner for the commit idThree places talk about the same identity: the lineage nonce, the The operation record should be the one server object for maintenance kinds. ValidationReviewed head Merge base with
The code checks are reads of
LimitationsEngine tests were not run. They do not exercise this document. The 576.8 s and 286.8 s runs were not measured here. Lance HTML guides were not re-fetched. The compaction and row-id claims come from the pinned Lance 13.0.0 crate and from Inline comments (same review, side RIGHT, commit 2a6c6c4)File: Line 24. Blocking. A direct target is not local-only.
State that boundary. Keep the direct path for a stopped server, on local storage and on object storage. Line 173. Blocking. This order plus The next default request starts at the same type key. Each clean table is planned again ( The deferred list has to be the continuation. The next full-catalog request, and the CLI, start at the first deferred table and use a new operation id. Release the gates between ids. Line 181. Blocking. This bound is short of the wait the current function holds. The schema permit and the table guards drop at Drop the branch gate before the artifact scan. Report the config clear after the graph commit, beside the artifact. Say what SIGTERM does while an issued artifact write is still running. Line 206. Blocking. This call site has a nonce, not the id history will store.
Put the hook after Line 244. Blocking. This write is capped at 4 GiB ( "Inside the existing shutdown grace" is not shown. Await the write only after it is issued. Do not hold writer gates for the scan. A missing or stale artifact stays safe: load rejects it and rebuilds in memory. Line 287. Blocking. On this merge base these puts are not definite.
Name that dependency. Do not start table work, and do not publish, while a put is in doubt. Equal bytes on read-back mean this write landed. Line 313. Blocking. Match the nonce with History stores The storage-outcomes RFC keeps the server up on Line 92. Blocking. This list drops two current results.
Line 333. Blocking. A repeated POST of a non-terminal record must not look finished. Deployment returns 202 until the work is terminal and 200 when it is done. Return 202 while this record is non-terminal, and 200 when it is terminal. GET and cancel return the principal, the options, and the outcomes, and cancel stops the run. Both need State the CLI exit for |
Record the intent nonce and resolve by commit_id_answers; keep an in-doubt publication non-terminal; release the writer gates before the artifact scan; add a resume position, resource ceilings and per-table failure isolation; require maintain on every route; depend on ModernRelay#1175 and ModernRelay#1176.
|
Revised in Blocking findings
Optional items. #1175 is in Main's |
|
Recommendation: revise. WhyA served graph cannot run WhatThis pull request is a design document. It adds the RFC and three pending amendments. It changes no executable code. The engine keeps today's gates and its one publication. It stops starting tables at a time bound and at resource ceilings. The server adds HowAcceptance is a conditional put of the operation record. The run holds the writer gates, works one table at a time, and publishes the finished tables once. The record stores the intent nonce before any manifest request. On restart, and in the live process, a ImpactThe serving process is the right owner. The revision fixes the lock boundary, the nonce, the non-terminal What the revision resolvedChecked against the PR tree, which has the same Rust as merge base
Blocking1. The resume key skips the deferred table
If that table is stored in Example: keys A, B, C. The run defers at C. Store the last table that was not deferred, and start after that key. When the first table of the catalog is deferred, omit 2. The first-table rule overrides the ceilings§1.3 says the first table always starts. §1.4 says a table over a per-table ceiling is deferred before its first write. Both sentences apply to the first table. The per-table ceiling wins, including for the first table. "Always starts" applies to the time bound and the byte budget only. The sentence that the per-table byte ceiling is at most the run budget covers the byte budget. It does not cover the fragment, row, Blob, or index ceilings. A first table over one of those is deferred, and the run continues with the next table. That is the progress. A start that ignores the ceiling rewrites an oversized table in the serving process. 3. Deletion files have no recorded size on
|
|
Revised in 1. Resume key. Fixed with an inclusive key rather than the last non-deferred table. 2. First table and ceilings. Adopted. Only the time bound and the run budget never stop the first table. A per-table ceiling defers it like any other table, and the run continues with the next one. 3. Deletion-file size. Confirmed: 4. Ghost 5. Cancel of 6. History match. The walk is adopted: main's The settling test differs from the suggestion. It is main's graph head, not the existence of 7. Artifact limit. Adopted, with the bound placed before and during the build rather than estimated only. The manifest's row counts give the fixed-width part of the body exactly: each id's 4-byte length prefix, and the offsets and targets in both directions. The id bytes are not in the manifest, so an estimate from cardinalities alone would not bound them. The build therefore does not start when the fixed-width part already passes 8. Optional callback. Adopted. Optional points:
The evidence table gains rows for:
The decision log has an entry for this revision. Also, #1175 now has main merged in ( |
|
Recommendation: revise. WhyA served graph cannot run WhatThis pull request is a design document. Head HowAcceptance is a conditional put of the operation record. The run holds the writer gates, works one table at a time, and publishes the finished tables once. The record stores the intent nonce before any manifest request. On restart, and in the live process, a ImpactThe serving process is the right owner. The eight blocking findings from the review on What the revision resolvedChecked against head
The artifact rule now aims to bound the build before the write. That placement is right. The formula in §1.8 is finding 2 below. Blocking1.
|
|
Revised in Blocking1.
The walk now has a stop that does not depend on reaching the input head. Every commit on main is written on main, including a fast-forward merge ( §2.2 now lists the published commit's history id. It is written by the terminal write, from the publisher's answer, or by resolution. The sentence on the storage-outcomes RFC now says that this rule is later than that RFC's version test, and that a version that only changes configuration ( 2. The artifact bound matches
The build reads each edge table's scan one batch at a time instead of calling
The header's count and length fields are counted at their widest. The count only grows. Past the limit, the build drops what it holds before it keeps another batch and before
A traversal read with no artifact builds the same structure today with no limit. Where this differs from the review. The pre-check is a lower bound, not an upper bound, because a pre-check that refuses needs a lower bound. An upper bound over the limit does not show that the object is over it. The pre-check reads each pinned edge table's live row count, Optional
Evidence rows
|
|
Recommendation: revise. WhyA served graph cannot run WhatThis pull request is a design document. Head HowAcceptance is a conditional put of the operation record. The run holds the writer gates, works one table at a time, and publishes the finished tables once. The record stores the intent nonce before any manifest request. On restart, and in the live process, a ImpactThe serving process is the right owner. The two blocking findings from the review on What the revision resolvedChecked against head
Blocking1. The resolution read misses the commit it must judge§2.4 reads the parent chain through §2.4 treats The catalog already has one lookup for this. The walk must start at the branch head row. It follows 2. The offset count omits arrays
|
|
Revised in Blocking1. The walk looks up each commit through
In §2.4, the walk now starts at the head row of a fresh read of main's The three endings now name the full lookup:
The cost sentence now says that a buffered commit costs no request and a released one costs one 2. Both offset sides are counted. A new id of type T now adds 4 bytes in every CSR whose source type is T and in every CSC whose destination type is T. The text says why, from the code: Optional
The decision log has a fourth entry. |
|
Recommendation: ready. WhyA served graph cannot run WhatThis pull request is a design document. Head HowAcceptance is a conditional put of the operation record. The run holds the writer gates, works one table at a time, and publishes the finished tables once. The record stores the intent nonce before any manifest request. On restart, and in the live process, a ImpactThe serving process is the right owner. Both blocking findings from the review on What the revision resolvedChecked against head
Optional1. A released commit is not one read of
|
…tor's path resolution after fifth review
|
Thanks. I took the optional finding and the path-resolution note in
The decision log has a fifth entry. |
A graph served from object storage cannot be optimized on 0.14.
optimizeruns only in a direct process, a direct target is local-only, and a direct process beside a live server refuses on the cluster lock. Fragment debt therefore grows without bound on every served S3 graph. The earlier proposal (#1168) was closed because it mixed engine, server and control-plane work without an ownership model, made acceptance ephemeral with blind retry, and added concurrency changes that interacted badly. Its closing review asked for focused RFCs that start with a bounded, manually requested remoteoptimizethrough the existing owner. This is that first RFC, covering the engine and the server together.The engine keeps today's gate shape and its one publication. What changes is that a request bounds how long the run keeps starting tables while it holds the writer gates. Tables are worked one at a time and reported one by one (optimized, nothing to do, deferred, failed), and a
before_publishcallback receives the intended graph commit id before the publication. The server records the request in a new__cluster/maintenance.json, written only by the lock holder, under an operation id the caller chose. The record is durable before any effect. The server executes the request as an owned operation, answers lookups by that id after a lost response, and on restart resolves an unfinished record by looking for the intended commit in main's history; it never executes the run again. Once its single write is issued, the adjacency artifact refresh runs to completion and is reported separately, so it never leaves an uncertain storage write behind a certain publication. Three graph-scoped routes (submit, look up, cancel) reuse existing error codes, andomnigraph optimizedispatches to them for a server target.I checked the design against current main (
dac660fb): the optimize gates and the point where it creates its lineage intent (optimize.rs); the deployment ledger and its retention (omnigraph-cluster), whosedeny_unknown_fieldsstate is why the record is a separate object; the owned-operation and shutdown contract; the HTTP conventions; and the error-code registry.check-docs.py,check-agents-md.shandtypospass. There is no code change.For review: the decision to challenge is keeping today's gate hold and bounding it, rather than changing how maintenance coexists with writers. Writes to main wait up to the bound plus one table's work plus the publication, and a queued schema apply or branch create stalls other writers for that long. The second is where the record lives and what it holds: a separate control object, four conditional writes per run, a 32-record, 4 MiB bound, and
unknown(never "expired") for an id it no longer holds. The third is caller-chosen ULIDs as operation ids. The RFC files pending amendments to RFC 1017's exception table, RFC 0011's Decision 11 and the server-runtime RFC's "no second durable operation record". It also states which parts of #1175 and the failure-model branch it reuses, and that cleanup, automatic triggers, workers and the coexistence redesign are out of scope.