Skip to content

feat(gateway): honour Idempotency-Key on POST /v1/harnesses, so a retried create replays instead of duplicating - #613

Merged
richard-epsilla merged 2 commits into
HarnessRouter:mainfrom
Jahanzeb-git:fix/harness-create-idempotency
Oct 29, 2026
Merged

richard-epsilla merged 2 commits into
HarnessRouter:mainfrom
Jahanzeb-git:fix/harness-create-idempotency

Conversation

@Jahanzeb-git

Copy link
Copy Markdown
Contributor

Closes #582

POST /v1/harnesses honours an Idempotency-Key. The reservation uses the same control store
/responses already uses, and is taken before _skills_prepare / _plugins_prepare run. Those
write blobs and packages, so reserving after them would duplicate the side effects even when the
Harness vertex is not duplicated.

  • replay returns the harness the first request created
  • same key, different body → 409 idempotency_key_reused
  • a create that fails before the vertex lands releases the key, so the retry is not poisoned
  • no key → unchanged behaviour
  • the hash is taken on the body as sent, before the prepare steps rewrite it

Deploy to Cloud sends deploy:<hid>:<fingerprint> per logical create, so a deploy retried later
(a lost answer, a closed laptop) replays Cloud's copy instead of making a second one. A different
harness, or a changed form, is a different key.

Protocol text in harnesses.md §5.1, capability advertised as harness_idempotency, conformance
check F-10.

Tests

  • gateway/tests/test_harness_idempotency.py — 6 tests: replay, 409, no-key unchanged, key
    released after a failed create, a >48KB skill still hashes the same after the prepare rewrite,
    capability advertised
  • full gateway suite: 1503 passed, 19 skipped
  • conformance suite tests: 155 passed, 3 skipped

One choice worth a second opinion: if the key's harness has since been deleted, the replay answers
404 harness_not_found, mirroring _idem_replay. Releasing the key and creating again is the
alternative — I picked the mirror for consistency, but it's a few lines either way.

All three parts from the Work section are in this PR rather than split, the protocol text and check are small enough that splitting them would have cost more review time than it saved.

…create replays instead of duplicating (HarnessRouter#582)

The reservation goes through the control store /responses already uses, and is taken before
_skills_prepare/_plugins_prepare — those write blobs, so reserving after them would duplicate the
side effects even when the vertex is not duplicated. A create that fails before the vertex lands
releases the key.

Deploy to Cloud sends deploy:<hid>:<fingerprint> per logical create, so a deploy retried later
replays the copy instead of making a second one. Protocol text in §5.1, conformance check F-10.
@vercel

vercel Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
unified-harness-protocol Ready Ready Preview Oct 29, 2026 4:40pm UTC

Request Review

…tays in the caller's workspace, and makes the harness again once its harness is deleted (HarnessRouter#582)

- A repeat while the first create is still writing waits for it (Tasks §6), instead of 404.
- The key is scoped to the workspace the harness is made in and kept apart from /v1/responses'
  keys, so a workspace's key never answers with another workspace's harness.
- A key whose harness was deleted names none, so a deploy whose Cloud copy was deleted makes it
  again instead of failing until the key expires. A claim left by a create that died is taken over
  after two minutes.
- Kept 24 hours (Tasks §6), the capability reported only with a store, the key sent on the deploy's
  POST only.
- Protocol: §5.1 states the window, the wait and the delete rule; errors.md lists
  idempotency_key_reused; the schema names the capability; F-10 cleans up after itself and checks
  the delete rule (suite 2026.10.19.post4); CHANGELOG.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@richard-epsilla richard-epsilla left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @Jahanzeb-git. Claiming the key before the prepares is the right call, and hashing the body before they rewrite it caught a real trap. I pushed one commit on top (3bbcc43) for four things the tests didn't reach, plus the protocol pieces around them.

1. A repeat while the first create is still running got a 404. The repeat finds the claim, reads the harness, finds nothing yet, and answers harness_not_found. That's the timeout case the key exists for: a create with a large skill is still writing its blob when the client gives up and sends it again. Tasks §6 already settles it for tasks: "If the first request is still running, the server MUST wait for it and return its result rather than returning a partial or a conflict." The repeat now waits for the harness to land.

2. A key could answer with another workspace's harness. The claim was keyed by organization only. A workspace's key that reused a key string got the other workspace's harness back from the replay, which is the cross-workspace read _scope_keeps exists to refuse. The claim is now scoped to the workspace the harness is made in. It's also kept apart from /v1/responses keys, which share the store.

3. On the deleted-harness question you raised: I went with creating again rather than 404. Deploy to Cloud is what decided it. The person deletes Cloud's copy and deploys again with the form unchanged. The PUT answers 404, the deploy creates, and it sends the same deploy:<hid>:<fingerprint> key. With the mirror, that deploy failed with "not found" until the key expired. A response replay reads a result, so 404 fits there. A create's purpose is a harness that exists, so once its harness is deleted the key names none, and a repeat makes one. The invariant still holds: at most one live harness per key.

4. A claim left by a create that died (a process killed between the claim and the write) would hold the key for its whole window. After two minutes with no harness, a repeat takes it over. Release-and-reclaim goes through the atomic create, so one racer wins and the others wait on it.

Smaller ones:

  • The key is kept 24 hours rather than 1, as Tasks §6 asks of a task's key. A deploy retried the next morning should still find its copy.
  • harness_idempotency is reported only when there is a store to keep keys in. Without one, a keyed create is refused 503.
  • The deploy sends the key on the POST only, not the PUT.
  • Protocol:
    • §5.1 now states the 24-hour window, the caller's scope, the wait, and the delete rule as MUSTs.
    • errors.md lists idempotency_key_reused, and the 2026-10-19 schema names the capability.
    • F-10 deletes what it makes. As it was, it ran after F-07's cleanup, so its harness stayed on the server. It also checks the delete rule.
    • The suite is now 2026.10.19.post4, with a CHANGELOG entry.

Tests: five new gateway tests (in progress, workspace, deleted, stale claim, responses key), each of which fails on 8921b23, and a deploy test where Cloud keeps keys and the first answer is lost. Full gateway suite 1512 passed, conformance tests 155 passed. I'll verify it on a release candidate before it ships as 0.37.11.

@richard-epsilla richard-epsilla left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on 0.37.11-rc.1 (sha256:9eb7543d…) on the test instance over real HTTP: two simultaneous creates with one key make one harness, a changed request gets 409 idempotency_key_reused, the key makes a new harness after a delete, one key string in two workspaces makes two harnesses that can't read each other, and no key behaves as before. Conformance F-10 passes live and leaves nothing behind. A deck turn completes on the candidate. Merging for 0.37.11.

@richard-epsilla
richard-epsilla merged commit 8d5c031 into HarnessRouter:main Oct 29, 2026
12 checks passed
@Jahanzeb-git

Jahanzeb-git commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor Author

All four are mine, and they are the same mistake: I tested the path through the
code instead of the reason it exists.

  1. Every test I added starts after the first create has finished. Reserve,
    create, replay. The window is the whole point of the key and I never put a test
    inside it, which is why _harness_idem_claim is a function I did not write.

  2. I scoped the claim to the org because the org was in hand. It should be
    scoped to the workspace the harness is made in, since that is what _scope_keeps
    is protecting. Same mistake in the harness\n prefix: two features share one
    store and I let them share one namespace.

  3. I read "the harness is gone" as "the answer is gone". A response replay
    reads a result; a create's key stands for a thing that should exist. Once the
    thing is deleted the key names nothing. deploy:<hid>:<fingerprint> is the exact
    key that would have 404-looped, and I wrote that key, so I had the case in front
    of me and reasoned about it in the abstract anyway.

  4. except Exception: idem_release(...) covers exceptions. A process killed
    between the reserve and _vg_upsert throws nothing, so nothing releases and the
    key is held for the day. My code only treats a failure as real if something
    raises. The 120 seconds and the takeover do not depend on the dying process
    being polite.

They are one question in four places: what the key means before the thing it
names exists. I answered that for the happy path and stopped.

What I would have caught with more care: the first by asking what the key is
for rather than what it does, the other three by writing one test per clause of
5.1 instead of one per path through _harness_create_as.

Now the part that is not about the diff.

You fixed these rather than sending them back at me. That is the review load I
am offering to take a share of.

25 to 35 hours a week, as a contractor: the gateway and runner review queue,
the conformance edges, whatever you have already decided the shape of and have
not had time to land. I am in Karachi, UTC+5, so my evening is your morning.

Pick one issue and I will read it and come back with a scope and a number for
it. If it is not a fit, say so and I will keep sending patches either way.

@richard-epsilla

richard-epsilla commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Thank you, @Jahanzeb-git. That is the right reading of all four, and each one now has a test of its own in gateway/tests/test_harness_idempotency.py: the repeat inside the window, the workspace scope, the key after a delete, and the claim a killed create leaves behind. The key shipped in 0.37.11 and is live on Cloud too. A deploy whose answer was lost now gets the same Cloud copy back instead of a second one.

On the offer: yes to the first step. The issue is #476, input requests in the end-the-turn shape. The shape is settled in the assessment there, and nobody is building it yet. It runs through every place you named:

  • the spec: Lifecycle §3 widened to incomplete with incomplete_details.reason: "input_required", and incomplete_details defined
  • the gateway: the input_request output item, the input_response input item, and the input_requests capability
  • the runner: ask_user for bases without a question tool of their own
  • conformance: one check per MUST, the habit from your list

Please post your scope as a comment on #476: how you would split it into PRs, what you would test for each clause, and what you would leave out. Once you have commented there, the issue can be assigned to you.

@phanisarman, it is your proposal: if you were already planning the PR, say so on #476 and it stays yours.

This branch was successfully deployed

1 active deployment
Preview — 3bbcc431 Deployed Oct 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UHP: an Idempotency-Key on POST /v1/harnesses, so a retried create never makes a second harness

3 participants