Skip to content

a response timeout for every request, a liveness timeout for held share acknowledgements, and both as the protocol's sign of life - #246

Open
plebhash wants to merge 6 commits into
stratum-mining:mainfrom
plebhash:2026-10-21-unbounded-waits
Open

plebhash wants to merge 6 commits into
stratum-mining:mainfrom
plebhash:2026-10-21-unbounded-waits

Conversation

@plebhash

@plebhash plebhash commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Closes #245

The spec has a dozen request/response pairs and says how long the reply may take for none of them; 5.3.19 says "by timeout" and 6.2 "in a reasonable time". It lets a server aggregate SubmitShares.Success with no limit on the hold and promises jobs at no rate. A client has no sentence to cite for when a server is dead, so each client picks its own limit and servers end up satisfying the strictest one. SV1 got its unwritten 90-second rule that way; mujina#65 and the Bitaxe firmware both shipped a 2-minute job-gap timeout that fires on healthy servers whenever the mempool is quiet.

Following the Working Group call of 2026-10-29, the request rule is stated once in chapter 3 and 5.3.13 carries the one exception. Four commits, one concern each:

  • 3, response timeout, 30 s: a request is any message with a .Success/.Error counterpart or whose section requires a particular reply; the receiver MUST reply within 30 seconds of receiving it, unless the section says otherwise (an accepted UpdateChannel gets no response, 5.3.8, only a SetTarget when the new max_target is below the channel's current target, 5.3.7 and 5.3.21). The sender SHOULD NOT treat the peer as unresponsive until twice the response timeout has passed since it sent the request, unless the request's section gives it a different wait (extension 0x0001 does for RequestExtensions); the second half covers transit, coarse timers and retransmission (rkuester's two-sided rule, as TCP pairs delayed ACK and RTO). 6.4.5 says a ProvideMissingTransactions sent in time counts as the reply to DeclareMiningJob, with Success/Error then due within the timeout of ProvideMissingTransactions.Success. 5.3.19 and 6.2 name the requester's wait, twice the response timeout from sending, in place of their wording. Both timeouts bind the node that produces the reply; a proxy that passes a request further relays without delay (GitGab19).
  • 5.3.13: a server MUST NOT hold an acknowledgement longer than 60 seconds from receipt; count batchers flush when it runs out. The client SHOULD NOT treat the server as unresponsive before the liveness timeout plus the response timeout have passed after sending a submission that has neither a Success covering it nor an Error; a client with no shares is owed nothing. A note gives the one-timer implementation (follow one submission, on its acknowledgement move to the most recent outstanding one) that meets the floor without a send time per share. The delayed-validation sentences of 5.3.14 are dropped: a client learns a prevhash only from SetNewPrevHash and a Custom Job is validated at SetCustomMiningJob, so the case never arises, and an error falls under the response timeout like any reply.
  • 5.1: the hashrate-driven rate already in 5.1 and 5.1.1 is the only one at which jobs are promised. A client that holds a Job MUST NOT read job silence as a dead server; the response timeout and the acknowledgement bound are the signs of life instead. A channel's first Job and its SetNewPrevHash are due within the response timeout of receiving the Open*MiningChannel, like the .Success, so a server that answers and then goes quiet is caught (GitGab19).
  • 5.3.21: the SetTarget owed after an accepted UpdateChannel is due within the response timeout of receiving the UpdateChannel, and the client's grace period is twice that. The max_target row of 5.3.7 no longer says the delay depends on job readiness, since a target applies to every job received after it; it points at the bound.
  • 0x0001: RequestExtensions gets its own two-sided deadline, 5 s for a supporting server and 10 s for the client, in place of the "2x the initial connection time" strategy that only the client can measure (GitGab19).

The call aligned on 60 seconds for share acknowledgements and left the generic deadline open for research. The research puts the two in different bands, which is why the acknowledgement bound is not the response timeout. Request/response deadlines settle around 30 seconds: SIP gives a transaction 64×T1 = 32 s (RFC 3261), ICE consent expires after 30 s without an answer (RFC 7675), SCTP heartbeats every 30 s (RFC 4960). Session liveness settles at 60 to 90: SCTP caps its RTO at 60 s (RFC 4960), BGP suggests a 90 s hold with keepalives at a third of it (RFC 4271), PCEP a 30 s keepalive with a dead timer a few multiples of it (RFC 5440). 30 seconds is also what the call converged on from the client side: nothing under 10 so flaky links are not cut off, giving up at 30. The 30-second value is the part put forward for review.

@gimballock

Copy link
Copy Markdown

Support

So we ran into the same problem mujina did and their proposed fix is a better than ours.

Our firmware has dropped the connection after 2 minutes without a job since December 2024, on SV1 and SV2, the same rule mujina#65 shipped. On an SRI testnet4 pool, 12% of gaps between new templates exceeded 2 minutes over an hour (max 257 s). The 5.1 MUST NOT is right.

Why the TCP keepalive plus the Acknowledgement bound pair beats the job gap:

  • Each covers the other's blind spot. TCP keepalive handles transport failures and the acknowledgement bound handles a wedged application. Together they cover both failure modes the job gap was meant for.
  • No false positives from quiet mempools. Neither rule infers death from the server choosing not to send work, which SV2 explicitly allows. The job gap is the only rule of the three that misfires on healthy servers.
  • No protocol change for keepalive. It's an option on the client's socket, so it works against every existing server today. That's why it fits as a note rather than a requirement.
  • The acknowledgement bound measures something servers can promise. A server can't promise a job by a deadline, but it can always flush pending acknowledgements on a timer.

Two suggestions:

  1. Say what the client measures: the age of the oldest unacknowledged submission, not the time since the last acknowledgement. A client that has found no shares is owed nothing.
  2. Add a informational note recommending TCP keepalive for half-open connections, mujina#65's original reason. With no shares in flight, the bound can't detect them.

Comment thread 05-Mining-Protocol.md Outdated

Response to `SubmitSharesStandard` or `SubmitSharesExtended`, accepting results from the miner.
A server MAY aggregate acknowledgements for multiple successful share-submission messages from the same channel into a single `SubmitShares.Success` response.
A server MUST NOT hold the acknowledgement of a successful submission for longer than 60 seconds after receiving it. A server that aggregates by count sends whatever it has batched when that time runs out. The only exception is a submission whose validation is delayed as described in §5.3.14; for it, the time is counted from when the result is known.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can validation be delayed? A share includes a job_id, not a prev_hash, and only the server assigns a job_id. The server either knows the job and validates the share at once, or rejects it. If there's no real case, I'd remove this exception and the sentence in 5.3.14.

If there is one, the delay should still count against the 60 seconds, since the client can't see when validation finishes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, there is no case. A client learns a prevhash only from SetNewPrevHash (5.3.17), so it cannot be ahead of the server on a server-issued job, and a Custom Job is validated when the server accepts SetCustomMiningJob, before any share for it exists (5.3.18, 5.3.19). The 5.3.17 "details" the sentence pointed at never existed. Dropped the exception from 5.3.13 and the two sentences from 5.3.14; an error is immediate and the chapter 3 response timeout bounds it like any reply.

Comment thread 05-Mining-Protocol.md Outdated
Response to `SubmitSharesStandard` or `SubmitSharesExtended`, accepting results from the miner.
A server MAY aggregate acknowledgements for multiple successful share-submission messages from the same channel into a single `SubmitShares.Success` response.
A server MUST NOT hold the acknowledgement of a successful submission for longer than 60 seconds after receiving it. A server that aggregates by count sends whatever it has batched when that time runs out. The only exception is a submission whose validation is delayed as described in §5.3.14; for it, the time is counted from when the result is known.
This bound is the sign of life the protocol gives a client. No message asks the server for a reply on demand, and new Jobs arrive only when the server has one (§5.1), so a client that is submitting shares and hears nothing has only this bound to tell a slow server from a dead one.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd state timing rules like this one from both sides. The responder MUST reply within a stated time, and the requester SHOULD NOT give up before a longer one. TCP pairs its times this way, with a delayed ACK under 0.5 s (RFC 1122 4.2.3.2) and a retransmission timeout of at least 1 s (RFC 6298 2). For share acknowledgements, I'd add:

A client SHOULD NOT treat the server as unresponsive until 90 seconds after sending a submission that has received neither a SubmitShares.Success covering it nor a SubmitShares.Error.

The extra 30 seconds cover network delay, coarse timers, TCP resending a lost acknowledgement (1, 2, 4, and 8 s for four losses), etc. The 90 seconds also nicely matches SV1, where it has worked for years.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and taken further than 5.3.13: every rule on the branch had the same number on both sides. Chapter 3 now says the requester SHOULD NOT treat the peer as unresponsive until 30 seconds after the reply was due, counted from when it sent the request, with the margin named once there. 5.3.13 carries your sentence, 90 seconds after sending a submission with neither a Success covering it nor an Error, and 5.3.21's grace period becomes the response timeout plus the margin. I left SV1 out of the text, since the 90 seconds there is a job interval rather than an acknowledgement bound.

Comment thread 05-Mining-Protocol.md Resolved
Comment thread 05-Mining-Protocol.md Outdated
Comment on lines +573 to +575
When the server then accepts the `UpdateChannel`, §5.3.7 requires it to send a new `SetTarget` that respects the new `max_target`, and the server MUST send it within the time §5.3.13 allows for a share acknowledgement. A server that does not accept the `UpdateChannel` responds with `UpdateChannel.Error` instead, and the bound stays at the previous `max_target`.

An accepted `UpdateChannel` has no response (§5.3.8), so a client cannot tell whether a `SetTarget` was sent before or after the server accepted it. A client that lowers `max_target` SHOULD allow the server a reasonable grace period to send the new `SetTarget` or the `UpdateChannel.Error` before treating a target above the new value as a violation of the bound (for instance: by falling back to a different server). The grace period ends when the `UpdateChannel.Error` arrives, or when the time the client allowed runs out. A `SetTarget` within the new `max_target` does not end it, since it may have been sent before the server accepted the `UpdateChannel`. The target of a `SetTarget` received during the grace period still applies until the next one, like any other.
An accepted `UpdateChannel` has no response (§5.3.8), so a client cannot tell whether a `SetTarget` was sent before or after the server accepted it. A client that lowers `max_target` SHOULD allow the server at least that time as a grace period to send the new `SetTarget` or the `UpdateChannel.Error` before treating a target above the new value as a violation of the bound (for instance: by falling back to a different server). The grace period ends when the `UpdateChannel.Error` arrives, or when the time the client allowed runs out. A `SetTarget` within the new `max_target` does not end it, since it may have been sent before the server accepted the `UpdateChannel`. The target of a `SetTarget` received during the grace period still applies until the next one, like any other.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The client's grace period should be longer than the server's bound, not equal to it, for the same reasons I gave on 5.3.13. I'd make it the same 90 seconds.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and it follows from the two-sided rule: the grace period is now the server's deadline plus the chapter 3 margin. The number differs from your 90 because the branch you reviewed is stale; the SetTarget owed after an UpdateChannel is a reply to a request and falls under the 30 s response timeout, not the 60 s acknowledgement bound, so the grace period is 60 s.

Comment thread 05-Mining-Protocol.md Resolved
@plebhash plebhash changed the title bound share acknowledgement delay, and use it as the protocol's sign of life a response timeout for every request, a liveness timeout for held share acknowledgements, and both as the protocol's sign of life Nov 10, 2026
@plebhash
plebhash force-pushed the 2026-10-21-unbounded-waits branch 3 times, most recently from f49c90e to 445c397 Compare November 10, 2026 19:22
@plebhash

Copy link
Copy Markdown
Member Author

I had some local changes that I had forgotten to push before @rkuester and @GitGab19 left their reviews.

apologies for that!

I've pushed them and would appreciate a new round of reviews.

the changes I had forgotten to push are:

  • chapter 3 now defines the two numbers: a 30 s response timeout for every request (5.3.19 and 6.2 point at it instead of "by timeout" / "reasonable time"; 6.4.5 says ProvideMissingTransactions counts as the reply to DeclareMiningJob), and a 60 s liveness timeout for held share acks
  • no transport keepalive requirement, per Ryan's comment on waits with no time: the spec lets one side wait on the other without saying how long #245
  • 5.3.2 uses the liveness timeout as the floor before a channel-less connection may be closed, and 5.3.21's SetTarget deadline is the response timeout rather than the share-ack bound

@plebhash
plebhash force-pushed the 2026-10-21-unbounded-waits branch 3 times, most recently from 380f561 to 582b2c6 Compare November 11, 2026 01:48
Comment thread 03-Protocol-Overview.md Resolved
Comment thread 03-Protocol-Overview.md Resolved
Comment thread 05-Mining-Protocol.md Resolved
Comment thread 06-Job-Declaration-Protocol.md Resolved
@plebhash
plebhash force-pushed the 2026-10-21-unbounded-waits branch from 582b2c6 to cd6d8c1 Compare November 11, 2026 19:25

@bit-aloo bit-aloo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we have these time bounds, especially for share submission and acknowledgement, do we still need the current share count based batching for acknowledgements?

Right now, batching is based on the number of shares, but with these changes, it also becomes time dependent. For example, an acknowledgement should be sent within 30 seconds of the oldest unacknowledged share being submitted. Could we drop the share count dependency entirely and make batching purely time based?

I have a few more questions about the request-response time bounds for other messages as well. Need to think a bit more about the edge cases there.

@plebhash

plebhash commented Nov 12, 2026 •

Copy link
Copy Markdown
Member Author

Once we have these time bounds, especially for share submission and acknowledgement, do we still need the current share count based batching for acknowledgements?

batching criteria is up to implementations, it's not defined on spec.

the spec does not impose that it needs to be based on count, so this is not true:

Right now, batching is based on the number of shares

Comment thread 03-Protocol-Overview.md Resolved
Comment thread 05-Mining-Protocol.md Resolved
@gimballock

gimballock commented Nov 12, 2026 •

Copy link
Copy Markdown

do we still need the current share count based batching for acknowledgements?
Could we drop the share count dependency entirely and make batching purely time based?

At low target share rate (<10 spm) the share count is too low to trigger the counter arm of this so the timer bound will always trigger.
At target share rates >10 the count-based buffer will fill up before the timer bound so the counter rule will dominate.

The real question is, what is the value to the client of the freshness of acknowledgments? If we only want acks to be sufficiently/minimally fresh then the timeout is enough, if we prefer fresher information for some reason then including the count-based rule does that for us.

Personally, I can't think of a good reason why the firmware would care about freshness (beyond the liveness check) but the client isn't always a miner, it could be a translator or jdc.

@GitGab19 GitGab19 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK

@plebhash
plebhash requested a review from rkuester November 13, 2026 12:49

@gimballock gimballock left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK.

One suggestion, here or as a follow-up: the timeouts say when to give up but not how to come back, so every client of a server that stops answering reaches the same deadline together and reconnects in lockstep. For §3:

A node that gives up on a peer under these timeouts and reconnects SHOULD delay each reconnect attempt after the first by a randomized backoff (for example, exponential with jitter), so that nodes that lost the same server do not reconnect in lockstep.

10,000 clients retrying together arrive within about a round trip; spread over 30 s, that is about 330 a second, for about 15 s of added delay on average.

…replies

The spec has a dozen request/response pairs (SetupConnection, the
Open*MiningChannel messages, SubmitShares*, SetCustomMiningJob,
AllocateMiningJobToken, DeclareMiningJob, ProvideMissingTransactions,
RequestTransactionData, CoinbaseOutputConstraints) and says how long
the reply may take for none of them. 5.3.19 says "by timeout" and 6.2
says "in a reasonable time" without a number. Every implementation
turns that into a constant of its own, and the side being waited on
has to satisfy the strictest one; SV1 ended up with an unwritten
90-second rule that way.

State the rule once, in chapter 3, for every request: the receiver
MUST reply within 30 seconds. DeclareMiningJob may be answered first
by ProvideMissingTransactions; 6.4.5 says that counts as the reply,
and that Success or Error is then due within the response timeout of
ProvideMissingTransactions.Success.
A reply that a section lets be held, as 5.3.13 does for
acknowledgement batching, is bounded by a second number, the liveness
timeout of 60 seconds, the value the Working Group call of 2026-10-29
aligned on for share acknowledgements. UpdateChannel has an
.Error counterpart but an accepted one gets no response (5.3.8), so
the rule exempts a message whose section says so; 5.3.21 bounds the
SetTarget that 5.3.7 requires instead. 5.3.19 and 6.2 name the
requester's wait, twice the response timeout from sending, in place
of their wording, since both describe when the miner or the JDC
falls back.

Both timeouts bind the node that produces the reply. A proxy that
passes an OpenStandardMiningChannel further (5.3.2) cannot answer
before its own upstream node does, and a compliant pool may take the
whole 30 seconds, so a per-hop deadline would be unmeetable; per-hop
budgets would instead stack past the patience below. A relaying node
forwards and relays without delay, which adds only transit.

A shared deadline makes every client of a server that stops answering
give up together and reconnect together, within about a round trip of
each other. Ask for a randomized backoff between reconnect attempts so
that they spread out instead.

State the rule from both sides. The responder's deadline and the
requester's patience are measured from different moments on different
clocks, so the requester SHOULD NOT give up until twice the response
timeout has passed since it sent the request, unless the request's
section gives it a different wait, as extension 0x0001 does for
RequestExtensions, which a server without the extension ignores, and
as 5.3.13 and 5.3.21 do. TCP pairs its times
the same way: a delayed ACK under 0.5 seconds (RFC 1122, 4.2.3.2)
against a retransmission timeout of at least 1 second that doubles
per loss (RFC 6298). The second half covers transit, coarse timers
and a few retransmissions.

30 seconds is where request/response deadlines settle in protocols
that set one: SIP gives a transaction 64 times its 500 ms T1, 32
seconds (RFC 3261), ICE consent expires after 30 seconds without an
answer (RFC 7675), SCTP heartbeats every 30 seconds (RFC 4960). It is
also what the Working Group call of 2026-10-29 converged on from the
client side: nothing under 10 seconds, so that clients on flaky links
are not cut off, and giving up at 30. A server has every reason to
answer at once, so the number mostly tells a client how long it must
wait before it may give up.

stratum-mining#245
5.3.13 lets a server aggregate SubmitShares.Success across submissions
and says nothing about how long it may hold them. The illustration
batches every 10 successful submissions, which at an ordinary vardiff
share rate is minutes of silence from a healthy server. A client that
is submitting shares and hears nothing has no sentence to cite for when
the server is wedged, and the protocol offers no other application-level
sign of life: no message asks for a reply on demand, and jobs are not
promised at any rate.

Bound the hold by the chapter 3 liveness timeout, 60 seconds from
receipt of each submission, the value the Working Group call of
2026-10-29 aligned on. A server that batches by count flushes what it
has when the time runs out; at the common target of 6 shares per
minute that still leaves about 6 acknowledgements per batch.

State the client's side too: it SHOULD NOT give up before the
liveness timeout plus the response timeout have passed after sending a
submission that has received neither a Success covering it nor an
Error. A client
that has found no share is owed nothing, and a dead host is left to
the transport. A note gives the one-timer way to meet the floor
without a send time per outstanding submission: follow one
submission, and on its acknowledgement move to the most recent
outstanding one, so the server can only stop the timer by clearing
the backlog.

5.3.14 said an error may be delayed when the server cannot validate
the submission at once, when the miner learns a new prevhash before
the server does. That cannot happen: a client learns a prevhash only
from SetNewPrevHash (5.3.17), and a Custom Job is validated when the
server accepts SetCustomMiningJob, before any share for it exists
(5.3.18, 5.3.19). A delay the client cannot see would also be no
bound for it. Drop the two sentences; an error is immediate, and the
chapter 3 response timeout bounds it like any reply.

stratum-mining#245
Nothing in the spec requires a server to send jobs at any rate beyond
what 5.1 and 5.1.1 already say: fast enough that the client does not
roll nTime past MAX_FUTURE_BLOCK_TIME. A server whose template does not
change can leave a channel without a new job for a whole block
interval. Clients coming from SV1 expect otherwise, because SV1 pools
send a job every minute or so to stay under the liveliness limit one
client set a decade ago, and a client that carries that expectation
over drops healthy connections whenever the mempool is quiet:
mujina#65 shipped a 2-minute job timeout before review caught it, and
the Bitaxe firmware has had one since December 2024, which on an SRI
testnet4 pool fired on 12% of template gaps in an hour.

Say in 5.1 that the hashrate-driven rate is the only one promised,
that a client holding a job MUST NOT read job silence as a dead
server, and that the response and liveness timeouts of chapter 3 are
the signs of life to use instead.

A client that holds no job yet has neither sign: the first job is not
a reply to a request, and there are no shares to acknowledge. 5.3.15
and 5.3.16 make the first job the first message after the channel
opens and 5.3.17 sends a prevhash when a channel opens, with no
deadline on either, so a server that answers Open*MiningChannel and
then goes quiet is undetectable. Bound both by the response timeout
from receipt of the Open*MiningChannel, alongside the .Success, so
the generic wait of chapter 3 covers them without a second clock.

stratum-mining#245
5.3.21 asks a client that lowers max_target to allow the server "a
reasonable grace period" before treating a target above the new value
as a violation. Reasonable has no number, and the server cannot learn
what the client picked, so a strict client falls back from a server
that did nothing wrong.

An accepted UpdateChannel has no response of its own, but 5.3.7
requires a SetTarget when the new max_target is below the channel's
current target, which makes that SetTarget the reply chapter 3 bounds. Say so: the server MUST send it
within the response timeout, and the client's grace period is twice
that.

The max_target row of 5.3.7 said the delay depends on "new job
readiness on the server". A target applies to every job received
after it (5.3.21), so the server need not wait for a job; the row now
points at the bound instead.

stratum-mining#245
5.3.2 lets a server close a connection that opens no channel "within a
reasonable period". The client cannot learn the period, so a client
that has a reason to connect before it has a channel to open (a proxy
whose first downstream device has not arrived yet) cannot know whether
it will still be connected when it does.

Give the period the chapter 3 liveness number: a server SHOULD NOT
close a channel-less connection before 60 seconds have passed since
its SetupConnection.Success, and MAY after. A client that needs the
connection longer opens a channel or expects to reconnect.

stratum-mining#245
The extension lets a client wait "2x the initial connection time" for
a RequestExtensions reply, retry, and consider a fallback pool after
about 1 second. The connection time is a number only the client has,
so no server can be held to it, and the base protocol now gives a
server 30 seconds for any request (chapter 3), which would stall every
connection to a server without the extension, since such a server
ignores the message and the client may not proceed before the reply.

State the deadline from both sides, in seconds. A supporting server
MUST answer within 5 seconds; answering is a lookup in its own list of
extensions. A client that hears nothing for 10 seconds treats the
server as not supporting the extension, reconnects, and proceeds
without RequestExtensions; the reconnect is what the ordering rule
requires, since the response must arrive before any other message.
The retry adds nothing on an ordered transport and the fallback-pool
advice is client policy, so both go. Chapter 3 names this wait as the
example of a section giving the requester a different one.

stratum-mining#245
@plebhash
plebhash force-pushed the 2026-10-21-unbounded-waits branch from edd1e35 to 2f80096 Compare November 13, 2026 21:23
@plebhash

Copy link
Copy Markdown
Member Author

@gimballock thanks, done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

waits with no time: the spec lets one side wait on the other without saying how long

5 participants