Skip to content

Merge certified origin floors into a replication source's seq row, apart from its applied cursors - #3109

Merged
kriszyp merged 5 commits into
mainfrom
feat/origin-floor-certificates-core
Oct 9, 2026
Merged

kriszyp merged 5 commits into
mainfrom
feat/origin-floor-certificates-core

Conversation

@kriszyp

@kriszyp kriszyp commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

⊙ Problem

A replication source can report, per origin, the highest log key it has durably applied (originCursors, #3080), but no field on the [seq, peer] row can hold a proof that resume may start somewhere: an applied position can sit above a transaction still open at the origin, since a log is appended in commit order, not key order. harper-pro#922 item 2 carries the origin-closed timestamp floor that #3085 certifies (no later append below it) between peers, and needs core to persist it in the same row, under the same commit ordering and blob rule as the cursors.

❓ Your call: Is a second field warranted rather than letting the certificate advance originLogKey? Yes: the two quantities answer different questions (applied position vs. proven closure), and folding a floor into the position would let a later data frame bury it; harper-pro's resume reads both and takes the max, and its item-4 admission will read the floor alone.

💡 Solution

An end_txn may now carry originFloors: [nodeId, closedFloor, relayable][], read by updateRecordedSequenceId only from a tagged stream that passes onFailure — a stored floor is what resume trusts, so core enforces for floors what it documents for cursors (admitsOriginFloors). It merges each entry by max into nodes[].closedFloor, skipping a floor that is not a positive finite number, records nodes[].relayable (whether the source received that origin directly with full table coverage, which is what lets a relay forward the floor; an equal floor can turn it on), and treats a floor-only end_txn like a cursor-only one: the row is written when a floor rose or a relayable flag turned on, and not otherwise. The outer dispatch gate admits an end_txn that carries only floors by the same admission check, so an idle connection's floor-only updates reach the row. resources/DESIGN.md records the field, the separation from originLogKey, and the held-stream and onFailure rules.

⚠️ Look hardest: the admission check is the one place a floor could enter the row without the apply loop being able to hold it back on a failed segment.

⚖️ Alternatives

❓ Your call: The pre-push adjudicator's do-less alternative — fold the floor into originCursors as max(position, key below floor) with no new field — was rejected: harper-pro uses a stored floor only while the peer still certifies (a downgraded peer may write below a floor it saved), and relay forwarding needs the relayable bit; neither survives the fold.

❓ Your call: A higher floor replaces relayable with its own flag rather than keeping the highest relayable floor as a separate field: a relay stops forwarding an origin whose newest floor arrived non-relayable, although the older relayable floor is still true. Changing this later means adding a row field.

❓ Your call: Floors are merged only from a tagged stream that passes onFailure (round-1 finding): core enforces for floors what it only documents for cursors, because a stored floor is what resume trusts. A source without the tag gets its floors ignored rather than a failure.

❓ Your call: Two apply loops writing the same peer's seq row (the old and new worker during a handover) are a read-modify-write race that can overwrite a newer floor with an older certified one; seqId and originLogKey already share that shape, one apply loop serves a database per thread, and the next 5 s tick repairs it, so this PR keeps the inherited mechanism rather than making the three-field merge transactional.

✅ Verification

  • unitTests/resources/replicationSeqCursor.test.js: a new case drives floor-only rises at an unchanged scalar (each writes), an equal floor whose relayable flag turns on (writes once), a lower floor, a repeated one, a stream without onFailure and a NaN floor (no write, checked after a later frame that does write), and a second origin, and asserts the data cursor is untouched — npx mocha unitTests/resources/replicationSeqCursor.test.js: 6 passing.
  • npm run typecheck (harper-pro, which compiles this core): 0 errors; npm run check:design-docs: index complete, budgets met; oxlint clean on the touched files.
  • End-to-end route: harper-pro's integrationTests/cluster/idleOriginFloorResume.test.mjs on the companion PR exercises this merge through a live A–B–C cluster (floors stored per origin on the [seq, peer] rows, read back through a fixture), 3/3 passing twice.

Refs harper-pro#922

🤖 Generated by Claude Fable 5.1; posted via @kriszyp.

Related PRs: #3080 overlaps (the originCursors merge in updateRecordedSequenceId this extends), 1 other independent
Complexity: medium

Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex; adjudicated=domain; declined=cursor-grok,cursor-kimi,cursor-muse; rounds=4; full=1 @ 7077108

Review-Attention: study ~13m (critical: Table.ts; decisions: enforce-floors-document-cursors, relayable-follows-latest-floor, floors-in-seq-row, do-less-alternative) @ 7077108

kriszyp and others added 3 commits October 7, 2026 22:19
…art from its applied cursors

An end_txn can now carry originFloors: [nodeId, closedFloor, relayable][], an
origin's closed timestamp floor as certified by its producer (the floor of
harper#3085) and whether the source received that origin directly with full
table coverage. The apply loop merges them by max into nodes[].closedFloor and
nodes[].relayable on the connection's [seq, peer] row, kept apart from
originLogKey: an applied position can sit above a transaction still open at the
origin, a floor cannot, so only the floor is proof that resume may start there.
A floor-only end_txn writes the row like a cursor-only one.

harper-pro#922 item 2 is the consumer: its replication layer carries the floor
between peers and resumes from it.

Dispatch-Task: hp-922-phase1-origin-floors
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hhrw3yq9XSUgnNo3ymfe4B
…nite one, and make the floor test deterministic

A stored floor is what resume trusts, so core now enforces for floors what it
only documented for cursors: an end_txn's originFloors are merged only when the
stream is tagged and passes onFailure, since an untagged or callback-less source
reports a failed segment nowhere this loop can hold the floor back. An entry
whose floor is not a positive finite number is skipped. The test's no-op frames
are followed by a frame that writes, so the write count is checked after they
have run rather than after a sleep.

Dispatch-Task: hp-922-phase1-origin-floors
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hhrw3yq9XSUgnNo3ymfe4B
… it, and settle the DESIGN wording

Dispatch-Task: hp-922-phase1-origin-floors
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hhrw3yq9XSUgnNo3ymfe4B

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for merging certified origin floors apart from applied cursors during transaction replication, updating the design documentation, replication apply loop logic, and adding corresponding unit tests. The review feedback recommends enhancing robustness by explicitly verifying that originFloors is an array in the admitsOriginFloors helper, and adding defensive checks during the iteration of originFloors to prevent potential runtime TypeErrors from malformed entries.

Comment thread resources/Table.ts
Comment thread resources/Table.ts
@kriszyp
kriszyp marked this pull request as ready for review October 9, 2026 12:05
@kriszyp
kriszyp merged commit 910a0db into main Oct 9, 2026
53 checks passed
@kriszyp
kriszyp deleted the feat/origin-floor-certificates-core branch October 9, 2026 12:05
kriszyp added a commit to HarperFast/harper-pro that referenced this pull request Oct 9, 2026
…anges; restore certified origin floors by pinning core at merged harper#3109 (#1017)

* Refuse a core sync that drops the committed pointer's changes, and pin core at harper#3109 so certified floors reach the seq row again

harper-pro#1011 merged with `core` pinned at its still-open companion
harper#3109; the Sync Core it triggered (#1016) moved `core` to a harper main
that does not contain that commit, removing the Table.ts merge of
`event.originFloors` into `nodes[].closedFloor`. The receiver kept sending
floors, core ignored them, and idleOriginFloorResume.test.mjs timed out on
every Cluster 6/6 leg waiting for the first certificate.

`sync-core.sh` now fetches the tracked tip, freezes its SHA and runs
`core-sync-guard.sh` before moving anything: the committed pointer must be the
tip, an ancestor of it, or merge into it without changing its tree (a squash-
or rebase-merged companion), else the sync is refused with the diffstat.
`core` moves to harper#3109 merged with current harper main, and a unit test
drives a real core table with the receiver's end_txn shape so a core that
ignores floors fails in seconds rather than 90 s per case on six legs.

Refs #922
Depends-on: HarperFast/harper#3109

Dispatch-Task: ci-regression-harper-pro-1016-20261009
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LrZVGMv3XzudCdEkxX5yjX

* Compare tree ids in the core sync guard, track `.` and the configured remote like submodule update does, and restore the worker flag after the seam test

Review round 1 (Codex graded, Gemini, Cursor composer): `git diff --quiet` is
a configurable comparison, so the no-op merge test now compares the merged tree
id with the candidate's tree id; `submodule.core.branch=.` means the
superproject's own branch and the fetch remote follows the checked-out core
branch's remote, as `git submodule update --remote` did, with a fixture case;
merge-tree's stderr is kept apart from the tree id; nested submodules are still
updated after the checkout; the seam test resets `setMainIsWorker`; the fixture
teardown is guarded; narrating comments trimmed.

Dispatch-Task: ci-regression-harper-pro-1016-20261009
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LrZVGMv3XzudCdEkxX5yjX

* Accept a core pointer whose own pull request merged with revisions, proven by the merge commit the Sync Core workflow looks up

A coordinated PR merges with core pinned at its companion's branch commit; if
the companion then merges with edits to the lines it added, the pointer's exact
content is on no tip and the guard's merge test conflicts, stalling every sync
after a merge that was legitimate. The workflow now resolves the committed
pointer's merged pull request (`commits/{sha}/pulls`) and passes its merge
commit as CORE_SYNC_SUPERSEDED_BY, which the guard accepts only when the
candidate contains it. workflow_dispatch gains a drop_content input for the
explicit override. The seam test's end_txn events match the receiver's shape
(no timestamp field).

Dispatch-Task: ci-regression-harper-pro-1016-20261009
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LrZVGMv3XzudCdEkxX5yjX

* Take the superseding merge commit from a workflow input instead of looking it up, and unshallow from the selected remote

Review round 2: the automatic lookup accepted any merged pull request that
contained the pointer, so a stacked PR that dropped the lines would have passed
with no human step, replaying #1016. The proof is now a workflow_dispatch input
(superseded_by) a person supplies; the guard still requires the candidate to
contain it. Also: the guard unshallows from the remote sync-core.sh selected,
the merge-tree error file is removed on every exit, the revised-companion
fixture compares tree ids, and the fixture file's header is one line.

Dispatch-Task: ci-regression-harper-pro-1016-20261009
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LrZVGMv3XzudCdEkxX5yjX

* Initialize an empty core before syncing, refuse a superseding commit that is behind the pointer, and stop on a detached superproject tracking `.`

Review round 3: a plain clone leaves core/ empty, so the fetch ran against
the superproject and the guard reported an unrelated comparison (the old
command failed later, at the lock-file copy) — the submodule is initialized
first, with a fixture case; a pasted CORE_SYNC_SUPERSEDED_BY behind the pointer
can never be its merged pull request and is now refused; branch `.` on a
detached superproject stops as `git submodule update --remote` does instead of
fetching the remote's default branch; two narrating comments removed.

Dispatch-Task: ci-regression-harper-pro-1016-20261009
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LrZVGMv3XzudCdEkxX5yjX

* Never re-initialize a deinitialized core from the sync

Round 4: `submodule update --init` after a deinit regenerates the module
config without core.worktree and corrupts the git dir (AGENTS.md), so the sync
initializes core only when it was never initialized and stops otherwise.

Dispatch-Task: ci-regression-harper-pro-1016-20261009
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LrZVGMv3XzudCdEkxX5yjX

* Reach the deinit refusal in the plain-clone fixture

The first sync left the fixture's package.json modified, so the second run
stopped at the dirty-manifest check before the deinit refusal it asserts.

Dispatch-Task: ci-regression-harper-pro-1016-20261009
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LrZVGMv3XzudCdEkxX5yjX

* Pin core to the merged origin-floor companion on main

The squash-merged harper#3109 tree matches the previous companion merge exactly. The core sync guard accepts the divergent commit IDs by content.

Dispatch-Task: hp-1017-repin-3109-merge
Co-Authored-By: GPT-5 Codex <noreply@openai.com>

* Make the core sync guard advisory: warn in the Sync Core PR, never refuse

A companion that merges with different content than the pinned commit is
indistinguishable from a dropped companion, and a refusal leaves Sync Core
stuck until someone overrides it. The guard now prints a Markdown warning
(dropped commits and diffstat) and always exits 0; sync-core.sh applies the
validated SHA regardless and writes the warning to CORE_SYNC_WARNING_FILE,
which the workflow puts at the top of the PR body and in the job summary.
The superseded_by / drop_content overrides are removed. Detection is the
seam test and the cluster shards.

Dispatch-Task: hp-1017-advisory-guard
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nCqStpkWerMFwdBvgTEA3

* Keep the core sync going when the warning file cannot be written

An unwritable CORE_SYNC_WARNING_FILE no longer exits under set -e before the
checkout; a failed mktemp reports the comparison as undecidable; git output is
uncolored so it cannot corrupt the Markdown warning; the job summary ends the
warning with a newline.

Dispatch-Task: hp-1017-advisory-guard
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013nCqStpkWerMFwdBvgTEA3

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: GPT-5 Codex <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant