Repository navigation
a response timeout for every request, a liveness timeout for held share acknowledgements, and both as the protocol's sign of life - #246
Conversation
|
Support So we ran into the same problem mujina did and their proposed fix is a better than ours. Our firmware has dropped the connection after 2 minutes without a job since December 2024, on SV1 and SV2, the same rule mujina#65 shipped. On an SRI testnet4 pool, 12% of gaps between new templates exceeded 2 minutes over an hour (max 257 s). The 5.1 MUST NOT is right. Why the TCP keepalive plus the Acknowledgement bound pair beats the job gap:
Two suggestions:
|
|
|
||
| Response to `SubmitSharesStandard` or `SubmitSharesExtended`, accepting results from the miner. | ||
| A server MAY aggregate acknowledgements for multiple successful share-submission messages from the same channel into a single `SubmitShares.Success` response. | ||
| A server MUST NOT hold the acknowledgement of a successful submission for longer than 60 seconds after receiving it. A server that aggregates by count sends whatever it has batched when that time runs out. The only exception is a submission whose validation is delayed as described in §5.3.14; for it, the time is counted from when the result is known. |
There was a problem hiding this comment.
How can validation be delayed? A share includes a job_id, not a prev_hash, and only the server assigns a job_id. The server either knows the job and validates the share at once, or rejects it. If there's no real case, I'd remove this exception and the sentence in 5.3.14.
If there is one, the delay should still count against the 60 seconds, since the client can't see when validation finishes.
There was a problem hiding this comment.
Agreed, there is no case. A client learns a prevhash only from SetNewPrevHash (5.3.17), so it cannot be ahead of the server on a server-issued job, and a Custom Job is validated when the server accepts SetCustomMiningJob, before any share for it exists (5.3.18, 5.3.19). The 5.3.17 "details" the sentence pointed at never existed. Dropped the exception from 5.3.13 and the two sentences from 5.3.14; an error is immediate and the chapter 3 response timeout bounds it like any reply.
| Response to `SubmitSharesStandard` or `SubmitSharesExtended`, accepting results from the miner. | ||
| A server MAY aggregate acknowledgements for multiple successful share-submission messages from the same channel into a single `SubmitShares.Success` response. | ||
| A server MUST NOT hold the acknowledgement of a successful submission for longer than 60 seconds after receiving it. A server that aggregates by count sends whatever it has batched when that time runs out. The only exception is a submission whose validation is delayed as described in §5.3.14; for it, the time is counted from when the result is known. | ||
| This bound is the sign of life the protocol gives a client. No message asks the server for a reply on demand, and new Jobs arrive only when the server has one (§5.1), so a client that is submitting shares and hears nothing has only this bound to tell a slow server from a dead one. |
There was a problem hiding this comment.
I'd state timing rules like this one from both sides. The responder MUST reply within a stated time, and the requester SHOULD NOT give up before a longer one. TCP pairs its times this way, with a delayed ACK under 0.5 s (RFC 1122 4.2.3.2) and a retransmission timeout of at least 1 s (RFC 6298 2). For share acknowledgements, I'd add:
A client SHOULD NOT treat the server as unresponsive until 90 seconds after sending a submission that has received neither a
SubmitShares.Successcovering it nor aSubmitShares.Error.
The extra 30 seconds cover network delay, coarse timers, TCP resending a lost acknowledgement (1, 2, 4, and 8 s for four losses), etc. The 90 seconds also nicely matches SV1, where it has worked for years.
There was a problem hiding this comment.
Agreed, and taken further than 5.3.13: every rule on the branch had the same number on both sides. Chapter 3 now says the requester SHOULD NOT treat the peer as unresponsive until 30 seconds after the reply was due, counted from when it sent the request, with the margin named once there. 5.3.13 carries your sentence, 90 seconds after sending a submission with neither a Success covering it nor an Error, and 5.3.21's grace period becomes the response timeout plus the margin. I left SV1 out of the text, since the 90 seconds there is a job interval rather than an acknowledgement bound.
| When the server then accepts the `UpdateChannel`, §5.3.7 requires it to send a new `SetTarget` that respects the new `max_target`, and the server MUST send it within the time §5.3.13 allows for a share acknowledgement. A server that does not accept the `UpdateChannel` responds with `UpdateChannel.Error` instead, and the bound stays at the previous `max_target`. | ||
|
|
||
| An accepted `UpdateChannel` has no response (§5.3.8), so a client cannot tell whether a `SetTarget` was sent before or after the server accepted it. A client that lowers `max_target` SHOULD allow the server a reasonable grace period to send the new `SetTarget` or the `UpdateChannel.Error` before treating a target above the new value as a violation of the bound (for instance: by falling back to a different server). The grace period ends when the `UpdateChannel.Error` arrives, or when the time the client allowed runs out. A `SetTarget` within the new `max_target` does not end it, since it may have been sent before the server accepted the `UpdateChannel`. The target of a `SetTarget` received during the grace period still applies until the next one, like any other. | ||
| An accepted `UpdateChannel` has no response (§5.3.8), so a client cannot tell whether a `SetTarget` was sent before or after the server accepted it. A client that lowers `max_target` SHOULD allow the server at least that time as a grace period to send the new `SetTarget` or the `UpdateChannel.Error` before treating a target above the new value as a violation of the bound (for instance: by falling back to a different server). The grace period ends when the `UpdateChannel.Error` arrives, or when the time the client allowed runs out. A `SetTarget` within the new `max_target` does not end it, since it may have been sent before the server accepted the `UpdateChannel`. The target of a `SetTarget` received during the grace period still applies until the next one, like any other. |
There was a problem hiding this comment.
The client's grace period should be longer than the server's bound, not equal to it, for the same reasons I gave on 5.3.13. I'd make it the same 90 seconds.
There was a problem hiding this comment.
Agreed, and it follows from the two-sided rule: the grace period is now the server's deadline plus the chapter 3 margin. The number differs from your 90 because the branch you reviewed is stale; the SetTarget owed after an UpdateChannel is a reply to a request and falls under the 30 s response timeout, not the 60 s acknowledgement bound, so the grace period is 60 s.
f49c90e to
445c397
Compare
|
I had some local changes that I had forgotten to push before @rkuester and @GitGab19 left their reviews. apologies for that! I've pushed them and would appreciate a new round of reviews. the changes I had forgotten to push are:
|
380f561 to
582b2c6
Compare
582b2c6 to
cd6d8c1
Compare
bit-aloo
left a comment
There was a problem hiding this comment.
Once we have these time bounds, especially for share submission and acknowledgement, do we still need the current share count based batching for acknowledgements?
Right now, batching is based on the number of shares, but with these changes, it also becomes time dependent. For example, an acknowledgement should be sent within 30 seconds of the oldest unacknowledged share being submitted. Could we drop the share count dependency entirely and make batching purely time based?
I have a few more questions about the request-response time bounds for other messages as well. Need to think a bit more about the edge cases there.
batching criteria is up to implementations, it's not defined on spec. the spec does not impose that it needs to be based on count, so this is not true:
|
At low target share rate (<10 spm) the share count is too low to trigger the counter arm of this so the timer bound will always trigger. The real question is, what is the value to the client of the freshness of acknowledgments? If we only want acks to be sufficiently/minimally fresh then the timeout is enough, if we prefer fresher information for some reason then including the count-based rule does that for us. Personally, I can't think of a good reason why the firmware would care about freshness (beyond the liveness check) but the client isn't always a miner, it could be a translator or jdc. |
8e3104d to
edd1e35
Compare
gimballock
left a comment
There was a problem hiding this comment.
ACK.
One suggestion, here or as a follow-up: the timeouts say when to give up but not how to come back, so every client of a server that stops answering reaches the same deadline together and reconnects in lockstep. For §3:
A node that gives up on a peer under these timeouts and reconnects SHOULD delay each reconnect attempt after the first by a randomized backoff (for example, exponential with jitter), so that nodes that lost the same server do not reconnect in lockstep.
10,000 clients retrying together arrive within about a round trip; spread over 30 s, that is about 330 a second, for about 15 s of added delay on average.
…replies The spec has a dozen request/response pairs (SetupConnection, the Open*MiningChannel messages, SubmitShares*, SetCustomMiningJob, AllocateMiningJobToken, DeclareMiningJob, ProvideMissingTransactions, RequestTransactionData, CoinbaseOutputConstraints) and says how long the reply may take for none of them. 5.3.19 says "by timeout" and 6.2 says "in a reasonable time" without a number. Every implementation turns that into a constant of its own, and the side being waited on has to satisfy the strictest one; SV1 ended up with an unwritten 90-second rule that way. State the rule once, in chapter 3, for every request: the receiver MUST reply within 30 seconds. DeclareMiningJob may be answered first by ProvideMissingTransactions; 6.4.5 says that counts as the reply, and that Success or Error is then due within the response timeout of ProvideMissingTransactions.Success. A reply that a section lets be held, as 5.3.13 does for acknowledgement batching, is bounded by a second number, the liveness timeout of 60 seconds, the value the Working Group call of 2026-10-29 aligned on for share acknowledgements. UpdateChannel has an .Error counterpart but an accepted one gets no response (5.3.8), so the rule exempts a message whose section says so; 5.3.21 bounds the SetTarget that 5.3.7 requires instead. 5.3.19 and 6.2 name the requester's wait, twice the response timeout from sending, in place of their wording, since both describe when the miner or the JDC falls back. Both timeouts bind the node that produces the reply. A proxy that passes an OpenStandardMiningChannel further (5.3.2) cannot answer before its own upstream node does, and a compliant pool may take the whole 30 seconds, so a per-hop deadline would be unmeetable; per-hop budgets would instead stack past the patience below. A relaying node forwards and relays without delay, which adds only transit. A shared deadline makes every client of a server that stops answering give up together and reconnect together, within about a round trip of each other. Ask for a randomized backoff between reconnect attempts so that they spread out instead. State the rule from both sides. The responder's deadline and the requester's patience are measured from different moments on different clocks, so the requester SHOULD NOT give up until twice the response timeout has passed since it sent the request, unless the request's section gives it a different wait, as extension 0x0001 does for RequestExtensions, which a server without the extension ignores, and as 5.3.13 and 5.3.21 do. TCP pairs its times the same way: a delayed ACK under 0.5 seconds (RFC 1122, 4.2.3.2) against a retransmission timeout of at least 1 second that doubles per loss (RFC 6298). The second half covers transit, coarse timers and a few retransmissions. 30 seconds is where request/response deadlines settle in protocols that set one: SIP gives a transaction 64 times its 500 ms T1, 32 seconds (RFC 3261), ICE consent expires after 30 seconds without an answer (RFC 7675), SCTP heartbeats every 30 seconds (RFC 4960). It is also what the Working Group call of 2026-10-29 converged on from the client side: nothing under 10 seconds, so that clients on flaky links are not cut off, and giving up at 30. A server has every reason to answer at once, so the number mostly tells a client how long it must wait before it may give up. stratum-mining#245
5.3.13 lets a server aggregate SubmitShares.Success across submissions and says nothing about how long it may hold them. The illustration batches every 10 successful submissions, which at an ordinary vardiff share rate is minutes of silence from a healthy server. A client that is submitting shares and hears nothing has no sentence to cite for when the server is wedged, and the protocol offers no other application-level sign of life: no message asks for a reply on demand, and jobs are not promised at any rate. Bound the hold by the chapter 3 liveness timeout, 60 seconds from receipt of each submission, the value the Working Group call of 2026-10-29 aligned on. A server that batches by count flushes what it has when the time runs out; at the common target of 6 shares per minute that still leaves about 6 acknowledgements per batch. State the client's side too: it SHOULD NOT give up before the liveness timeout plus the response timeout have passed after sending a submission that has received neither a Success covering it nor an Error. A client that has found no share is owed nothing, and a dead host is left to the transport. A note gives the one-timer way to meet the floor without a send time per outstanding submission: follow one submission, and on its acknowledgement move to the most recent outstanding one, so the server can only stop the timer by clearing the backlog. 5.3.14 said an error may be delayed when the server cannot validate the submission at once, when the miner learns a new prevhash before the server does. That cannot happen: a client learns a prevhash only from SetNewPrevHash (5.3.17), and a Custom Job is validated when the server accepts SetCustomMiningJob, before any share for it exists (5.3.18, 5.3.19). A delay the client cannot see would also be no bound for it. Drop the two sentences; an error is immediate, and the chapter 3 response timeout bounds it like any reply. stratum-mining#245
Nothing in the spec requires a server to send jobs at any rate beyond what 5.1 and 5.1.1 already say: fast enough that the client does not roll nTime past MAX_FUTURE_BLOCK_TIME. A server whose template does not change can leave a channel without a new job for a whole block interval. Clients coming from SV1 expect otherwise, because SV1 pools send a job every minute or so to stay under the liveliness limit one client set a decade ago, and a client that carries that expectation over drops healthy connections whenever the mempool is quiet: mujina#65 shipped a 2-minute job timeout before review caught it, and the Bitaxe firmware has had one since December 2024, which on an SRI testnet4 pool fired on 12% of template gaps in an hour. Say in 5.1 that the hashrate-driven rate is the only one promised, that a client holding a job MUST NOT read job silence as a dead server, and that the response and liveness timeouts of chapter 3 are the signs of life to use instead. A client that holds no job yet has neither sign: the first job is not a reply to a request, and there are no shares to acknowledge. 5.3.15 and 5.3.16 make the first job the first message after the channel opens and 5.3.17 sends a prevhash when a channel opens, with no deadline on either, so a server that answers Open*MiningChannel and then goes quiet is undetectable. Bound both by the response timeout from receipt of the Open*MiningChannel, alongside the .Success, so the generic wait of chapter 3 covers them without a second clock. stratum-mining#245
5.3.21 asks a client that lowers max_target to allow the server "a reasonable grace period" before treating a target above the new value as a violation. Reasonable has no number, and the server cannot learn what the client picked, so a strict client falls back from a server that did nothing wrong. An accepted UpdateChannel has no response of its own, but 5.3.7 requires a SetTarget when the new max_target is below the channel's current target, which makes that SetTarget the reply chapter 3 bounds. Say so: the server MUST send it within the response timeout, and the client's grace period is twice that. The max_target row of 5.3.7 said the delay depends on "new job readiness on the server". A target applies to every job received after it (5.3.21), so the server need not wait for a job; the row now points at the bound instead. stratum-mining#245
5.3.2 lets a server close a connection that opens no channel "within a reasonable period". The client cannot learn the period, so a client that has a reason to connect before it has a channel to open (a proxy whose first downstream device has not arrived yet) cannot know whether it will still be connected when it does. Give the period the chapter 3 liveness number: a server SHOULD NOT close a channel-less connection before 60 seconds have passed since its SetupConnection.Success, and MAY after. A client that needs the connection longer opens a channel or expects to reconnect. stratum-mining#245
The extension lets a client wait "2x the initial connection time" for a RequestExtensions reply, retry, and consider a fallback pool after about 1 second. The connection time is a number only the client has, so no server can be held to it, and the base protocol now gives a server 30 seconds for any request (chapter 3), which would stall every connection to a server without the extension, since such a server ignores the message and the client may not proceed before the reply. State the deadline from both sides, in seconds. A supporting server MUST answer within 5 seconds; answering is a lookup in its own list of extensions. A client that hears nothing for 10 seconds treats the server as not supporting the extension, reconnects, and proceeds without RequestExtensions; the reconnect is what the ordering rule requires, since the response must arrive before any other message. The retry adds nothing on an ordered transport and the fallback-pool advice is client policy, so both go. Chapter 3 names this wait as the example of a section giving the requester a different one. stratum-mining#245
edd1e35 to
2f80096
Compare
|
@gimballock thanks, done. |
Closes #245
The spec has a dozen request/response pairs and says how long the reply may take for none of them; 5.3.19 says "by timeout" and 6.2 "in a reasonable time". It lets a server aggregate
SubmitShares.Successwith no limit on the hold and promises jobs at no rate. A client has no sentence to cite for when a server is dead, so each client picks its own limit and servers end up satisfying the strictest one. SV1 got its unwritten 90-second rule that way; mujina#65 and the Bitaxe firmware both shipped a 2-minute job-gap timeout that fires on healthy servers whenever the mempool is quiet.Following the Working Group call of 2026-10-29, the request rule is stated once in chapter 3 and 5.3.13 carries the one exception. Four commits, one concern each:
.Success/.Errorcounterpart or whose section requires a particular reply; the receiver MUST reply within 30 seconds of receiving it, unless the section says otherwise (an acceptedUpdateChannelgets no response, 5.3.8, only aSetTargetwhen the newmax_targetis below the channel's current target, 5.3.7 and 5.3.21). The sender SHOULD NOT treat the peer as unresponsive until twice the response timeout has passed since it sent the request, unless the request's section gives it a different wait (extension 0x0001 does forRequestExtensions); the second half covers transit, coarse timers and retransmission (rkuester's two-sided rule, as TCP pairs delayed ACK and RTO). 6.4.5 says aProvideMissingTransactionssent in time counts as the reply toDeclareMiningJob, withSuccess/Errorthen due within the timeout ofProvideMissingTransactions.Success. 5.3.19 and 6.2 name the requester's wait, twice the response timeout from sending, in place of their wording. Both timeouts bind the node that produces the reply; a proxy that passes a request further relays without delay (GitGab19).Successcovering it nor anError; a client with no shares is owed nothing. A note gives the one-timer implementation (follow one submission, on its acknowledgement move to the most recent outstanding one) that meets the floor without a send time per share. The delayed-validation sentences of 5.3.14 are dropped: a client learns a prevhash only fromSetNewPrevHashand a Custom Job is validated atSetCustomMiningJob, so the case never arises, and an error falls under the response timeout like any reply.SetNewPrevHashare due within the response timeout of receiving theOpen*MiningChannel, like the.Success, so a server that answers and then goes quiet is caught (GitGab19).SetTargetowed after an acceptedUpdateChannelis due within the response timeout of receiving theUpdateChannel, and the client's grace period is twice that. Themax_targetrow of 5.3.7 no longer says the delay depends on job readiness, since a target applies to every job received after it; it points at the bound.RequestExtensionsgets its own two-sided deadline, 5 s for a supporting server and 10 s for the client, in place of the "2x the initial connection time" strategy that only the client can measure (GitGab19).The call aligned on 60 seconds for share acknowledgements and left the generic deadline open for research. The research puts the two in different bands, which is why the acknowledgement bound is not the response timeout. Request/response deadlines settle around 30 seconds: SIP gives a transaction 64×T1 = 32 s (RFC 3261), ICE consent expires after 30 s without an answer (RFC 7675), SCTP heartbeats every 30 s (RFC 4960). Session liveness settles at 60 to 90: SCTP caps its RTO at 60 s (RFC 4960), BGP suggests a 90 s hold with keepalives at a third of it (RFC 4271), PCEP a 30 s keepalive with a dead timer a few multiples of it (RFC 5440). 30 seconds is also what the call converged on from the client side: nothing under 10 so flaky links are not cut off, giving up at 30. The 30-second value is the part put forward for review.