Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 34 additions & 32 deletions cmd/livepeer/testdata/cli/orchestrator-help.golden
Original file line number Diff line number Diff line change
Expand Up @@ -11,38 +11,40 @@ Available Commands:
redemptions Inspect recipient redemption state or explicitly retry stored transaction bytes

Flags:
-c, --config string Configuration file; empty disables discovery (env: LIVEPEER_ORCHESTRATOR_CONFIG) (default "config.toml")
-n, --network string Network name (custom networks require explicit chain settings) (env: LIVEPEER_ORCHESTRATOR_NETWORK) (default "offchain")
-d, --data-dir string Component data directory; defaults to ~/.lpData/<network>/<component> (env: LIVEPEER_ORCHESTRATOR_DATA_DIR) (default "$HOME/.lpData/offchain/orchestrator")
-a, --account string Keystore account address (env: LIVEPEER_ORCHESTRATOR_ACCOUNT)
-o, --orchestrator-address string Registered orchestrator address receiving ticket payments; defaults to the keystore account (env: LIVEPEER_ORCHESTRATOR_ORCHESTRATOR_ADDRESS)
-e, --eth-usd-feed string ETH/USD oracle address, alternative to a fixed wei-per-usd rate (env: LIVEPEER_ORCHESTRATOR_ETH_USD_FEED)
-p, --price-max-age duration Maximum age of the oracle observation (env: LIVEPEER_ORCHESTRATOR_PRICE_MAX_AGE) (default 2h0m0s)
-l, --listen string Public HTTP listener (env: LIVEPEER_ORCHESTRATOR_LISTEN) (default "127.0.0.1:8935")
-m, --metrics-listen string Loopback metrics listener (env: LIVEPEER_ORCHESTRATOR_METRICS_LISTEN) (default "127.0.0.1:8936")
-s, --service-url string Public orchestrator base URL; defaults to http://listen (env: LIVEPEER_ORCHESTRATOR_SERVICE_URL)
-r, --runner-service-url string Runner-facing base URL for callbacks and trickle; defaults to service-url (env: LIVEPEER_ORCHESTRATOR_RUNNER_SERVICE_URL)
--proxy-url-template string Generated proxy URL with {proxy} in a hostname label or final path segment (env: LIVEPEER_ORCHESTRATOR_PROXY_URL_TEMPLATE)
--bootstrap-secret-file string File containing runner bootstrap credential (env: LIVEPEER_ORCHESTRATOR_BOOTSTRAP_SECRET_FILE)
--runner-config string Static runner TOML path (env: LIVEPEER_ORCHESTRATOR_RUNNER_CONFIG)
--redeemer-db string (env: LIVEPEER_ORCHESTRATOR_REDEEMER_DB)
-k, --keystore-file string Encrypted geth redemption account JSON file (env: LIVEPEER_ORCHESTRATOR_KEYSTORE_FILE)
--keystore-password-file string Owner-only file containing the exact keystore password bytes (env: LIVEPEER_ORCHESTRATOR_KEYSTORE_PASSWORD_FILE)
--rpc-url-file string (env: LIVEPEER_ORCHESTRATOR_RPC_URL_FILE)
--chain-id uint (env: LIVEPEER_ORCHESTRATOR_CHAIN_ID)
--redeemer-max-fee-per-gas string Optional maximum redemption fee in wei per gas (env: LIVEPEER_ORCHESTRATOR_REDEEMER_MAX_FEE_PER_GAS)
--controller-address string (env: LIVEPEER_ORCHESTRATOR_CONTROLLER_ADDRESS)
-w, --wei-per-usd string (env: LIVEPEER_ORCHESTRATOR_WEI_PER_USD)
-t, --ticket-face-value string (env: LIVEPEER_ORCHESTRATOR_TICKET_FACE_VALUE)
--ticket-win-prob string (env: LIVEPEER_ORCHESTRATOR_TICKET_WIN_PROB)
--runner-grants strings Private runner IP[:port] or CIDR grants (env: LIVEPEER_ORCHESTRATOR_RUNNER_GRANTS)
--session-proxy-grants strings Private session proxy IP[:port] or CIDR grants (env: LIVEPEER_ORCHESTRATOR_SESSION_PROXY_GRANTS)
--health-grants strings Private static runner health IP[:port] or CIDR grants (env: LIVEPEER_ORCHESTRATOR_HEALTH_GRANTS)
--heartbeat-interval duration Runner heartbeat interval (env: LIVEPEER_ORCHESTRATOR_HEARTBEAT_INTERVAL) (default 5s)
--heartbeat-ttl duration Runner heartbeat expiry (env: LIVEPEER_ORCHESTRATOR_HEARTBEAT_TTL) (default 30s)
--print-config Print audited redacted TOML configuration
-h, --help help for livepeer-orchestrator
-v, --version version for livepeer-orchestrator
-c, --config string Configuration file; empty disables discovery (env: LIVEPEER_ORCHESTRATOR_CONFIG) (default "config.toml")
-n, --network string Network name (custom networks require explicit chain settings) (env: LIVEPEER_ORCHESTRATOR_NETWORK) (default "offchain")
-d, --data-dir string Component data directory; defaults to ~/.lpData/<network>/<component> (env: LIVEPEER_ORCHESTRATOR_DATA_DIR) (default "$HOME/.lpData/offchain/orchestrator")
-a, --account string Keystore account address (env: LIVEPEER_ORCHESTRATOR_ACCOUNT)
-o, --orchestrator-address string Registered orchestrator address receiving ticket payments; defaults to the keystore account (env: LIVEPEER_ORCHESTRATOR_ORCHESTRATOR_ADDRESS)
-e, --eth-usd-feed string ETH/USD oracle address, alternative to a fixed wei-per-usd rate (env: LIVEPEER_ORCHESTRATOR_ETH_USD_FEED)
-p, --price-max-age duration Maximum age of the oracle observation (env: LIVEPEER_ORCHESTRATOR_PRICE_MAX_AGE) (default 2h0m0s)
-l, --listen string Public HTTP listener (env: LIVEPEER_ORCHESTRATOR_LISTEN) (default "127.0.0.1:8935")
-m, --metrics-listen string Loopback metrics listener (env: LIVEPEER_ORCHESTRATOR_METRICS_LISTEN) (default "127.0.0.1:8936")
-s, --service-url string Public orchestrator base URL; defaults to http://listen (env: LIVEPEER_ORCHESTRATOR_SERVICE_URL)
-r, --runner-service-url string Runner-facing base URL for callbacks and trickle; defaults to service-url (env: LIVEPEER_ORCHESTRATOR_RUNNER_SERVICE_URL)
--proxy-url-template string Generated proxy URL with {proxy} in a hostname label or final path segment (env: LIVEPEER_ORCHESTRATOR_PROXY_URL_TEMPLATE)
--bootstrap-secret-file string File containing runner bootstrap credential (env: LIVEPEER_ORCHESTRATOR_BOOTSTRAP_SECRET_FILE)
--runner-config string Static runner TOML path (env: LIVEPEER_ORCHESTRATOR_RUNNER_CONFIG)
--redeemer-db string (env: LIVEPEER_ORCHESTRATOR_REDEEMER_DB)
-k, --keystore-file string Encrypted geth redemption account JSON file (env: LIVEPEER_ORCHESTRATOR_KEYSTORE_FILE)
--keystore-password-file string Owner-only file containing the exact keystore password bytes (env: LIVEPEER_ORCHESTRATOR_KEYSTORE_PASSWORD_FILE)
--rpc-url-file string (env: LIVEPEER_ORCHESTRATOR_RPC_URL_FILE)
--chain-id uint (env: LIVEPEER_ORCHESTRATOR_CHAIN_ID)
--redeemer-tx-timeout duration Inclusion timeout for each redemption transaction attempt (env: LIVEPEER_ORCHESTRATOR_REDEEMER_TX_TIMEOUT) (default 5m0s)
--redeemer-max-tx-replacements int Maximum fee replacements per redemption attempt (env: LIVEPEER_ORCHESTRATOR_REDEEMER_MAX_TX_REPLACEMENTS) (default 1)
--redeemer-max-fee-per-gas string Optional maximum redemption fee in wei per gas (env: LIVEPEER_ORCHESTRATOR_REDEEMER_MAX_FEE_PER_GAS)
--controller-address string (env: LIVEPEER_ORCHESTRATOR_CONTROLLER_ADDRESS)
-w, --wei-per-usd string (env: LIVEPEER_ORCHESTRATOR_WEI_PER_USD)
-t, --ticket-face-value string (env: LIVEPEER_ORCHESTRATOR_TICKET_FACE_VALUE)
--ticket-win-prob string (env: LIVEPEER_ORCHESTRATOR_TICKET_WIN_PROB)
--runner-grants strings Private runner IP[:port] or CIDR grants (env: LIVEPEER_ORCHESTRATOR_RUNNER_GRANTS)
--session-proxy-grants strings Private session proxy IP[:port] or CIDR grants (env: LIVEPEER_ORCHESTRATOR_SESSION_PROXY_GRANTS)
--health-grants strings Private static runner health IP[:port] or CIDR grants (env: LIVEPEER_ORCHESTRATOR_HEALTH_GRANTS)
--heartbeat-interval duration Runner heartbeat interval (env: LIVEPEER_ORCHESTRATOR_HEARTBEAT_INTERVAL) (default 5s)
--heartbeat-ttl duration Runner heartbeat expiry (env: LIVEPEER_ORCHESTRATOR_HEARTBEAT_TTL) (default 30s)
--print-config Print audited redacted TOML configuration
-h, --help help for livepeer-orchestrator
-v, --version version for livepeer-orchestrator

Use "livepeer-orchestrator [command] --help" for more information about a command.

Expand Down
2 changes: 2 additions & 0 deletions configs/orchestrator/config.example.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ HealthGrants = []
# Both files must be owner-only; the password is read as exact bytes (no newline).
# Or use LIVEPEER_ORCHESTRATOR_KEYSTORE_PASSWORD and omit KeystorePasswordFile.
# RPCURLFile = "/run/secrets/livepeer-payment-rpc-url"
# RedeemerTxTimeout = "5m" # Inclusion timeout per transaction attempt.
# RedeemerMaxTxReplacements = 1 # Same-nonce fee replacements before retrying the ticket.
# RedeemerMaxFeePerGas = "1000000000" # Optional ceiling, wei per gas
# ChainID = 42161
# Controller = "0xD8E8328501E9645d16Cf49539efC04f734606ee4" # Arbitrum default.
Expand Down
6 changes: 3 additions & 3 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,9 +113,9 @@ or a new chain invocation.
The signer account is the payer, supplying the TicketBroker deposit and reserve.
The orchestrator's keystore account signs redemptions and pays gas. Ticket payments
credit `OrchestratorAddress`, which defaults to that account. The chain account
must match the keystore account when signing. Keep a redemption key exclusive to
its orchestrator worker; concurrent CLI or other use can conflict with nonce
tracking.
must match the keystore account when signing. Multiple orchestrator processes
may share a redemption key. Failed submissions are retried automatically.
See [transaction recovery](payment-recovery.md).

### Prepare a keystore

Expand Down
4 changes: 2 additions & 2 deletions docs/orchestrator.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,8 +122,8 @@ every hour; `PriceMaxAge` must be positive and defaults to two hours. Startup
requires a valid observation. Stale rates suspend new sessions; existing
sessions retain their initial price.

`RedeemerMaxFeePerGas` optionally caps redemption fees in wei per gas. The
cap is per gas, not a total spending budget. Winning tickets wait until payment
See [transaction recovery](payment-recovery.md#orchestrator-transaction-recovery)
for timeouts, replacements and fee settings. Winning tickets wait until payment
parameters expire before redemption exposes their randomness on-chain. Only a
successful, canonical, finalized receipt settles their liability.

Expand Down
63 changes: 38 additions & 25 deletions docs/payment-recovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,27 +103,48 @@ use these states:
| State | Meaning and recovery |
| --- | --- |
| Queued (no attempt) | Wait for parameter expiry. Preparation failures leave the ticket eligible for another attempt. |
| `prepared` | Signed bytes, nonce and locally computed hash are durable; the worker resumes the first send after restart. |
| `broadcast` | A send may have reached RPC. Reconcile receipts automatically; an operator may explicitly resend identical bytes. |
| `submitted` | RPC acknowledged the hash. Keep checking receipts; explicit identical-byte retry is available if necessary. |
| `confirmed` | Receipt is successful, at or below the finalized head, and matches the canonical block hash. Liability is settled. |
| `reverted` | A canonical finalized receipt failed. Retain the audit row; do not resubmit automatically. |
| `expired` | Ticket creation round is outside the contract validity window. Retain the audit row. |

An uncertain broadcast blocks preparation of a new transaction
for this worker, preventing accidental nonce reuse. The redemption key must be
exclusive to this worker; do not submit concurrent CLI transactions from it.
Only a canonical finalized receipt marks a ticket redeemed. A pending receipt,
reorg, unavailable finalized head or RPC failure retains uncertainty and liability.
| `prepared` | Signed bytes, nonce and hash are durable before sending; restart resumes the saved transaction. |
| `broadcast` | Send in progress or interrupted before recording its outcome; restart resumes the saved transaction. |
| `submitted` | RPC acknowledged the hash; monitor inclusion and replace on timeout. |
| `retryable` | Submission, waiting or replacement failed; retry the ticket automatically. Older hashes remain under receipt reconciliation. |
| `mined` | Canonical inclusion observed; wait for finalization without fee replacement. A reorg resumes recovery. |
| `awaiting_finality` | Latest state says the ticket is used; pause submission until finalized state confirms use or a reorg removes it. |
| `confirmed` / `used` | Successful canonical finalized receipt or finalized ticket-use observation settles liability. |
| `reverted` / `invalid` | Finalized transaction failure or an invalid ticket; stop submitting and reconcile unresolved hashes. |
| `expired` | Ticket creation round is outside its validity window; reconcile unresolved hashes. |
| `superseded` | Finalized inclusion consumed the same sender/nonce, or finalized ticket settlement made this transaction unnecessary. |

Only canonical finalized state settles a ticket. RPC failures, missing receipts,
unfinalized inclusions and reorgs retain uncertainty and liability. Each ticket
counts once even when it has several signed transactions. Previously exposed
recipient randomness stays closed to new payments throughout retries.

`RedeemerTxTimeout` (`--redeemer-tx-timeout`) defaults to five minutes per
inclusion attempt. `RedeemerMaxTxReplacements` (`--redeemer-max-tx-replacements`)
defaults to one; zero disables replacements. Replacements retain the nonce and
calldata and raise both fee caps by geth's default 10% threshold, rounding up
with a strict increase.
Geth estimates initial fees using the RPC header's base fee. A missing base fee
leaves the ticket queued with an error. `RedeemerMaxFeePerGas` optionally caps
initial and replacement fees in wei per gas. Finalization waits do not consume
the inclusion timeout.

Restart restores deadlines, replacement budgets and nonce reservations for
acknowledged sends. Rejected submissions do not reserve a nonce. A rotated key can
resend saved bytes and reconcile receipts, but cannot replace the previous account's transactions.

Migration 002 preserves existing transactions. Its rollback refuses to discard
transactions or metadata that the old schema cannot represent. Retain the upgraded
database for a compatible binary when rollback is refused.

Inspect status without a key or RPC connection:

```sh
livepeer orchestrator redemptions
```

After checking the hash on the configured chain, explicitly retry the stored
transaction if it is absent or needs rebroadcast:
Automatic recovery normally handles rebroadcast. To explicitly resend the
current retryable transaction using its saved bytes and fees:

```sh
livepeer orchestrator redemptions \
Expand All @@ -138,9 +159,8 @@ pass the same `--config`, or `--network` and `--data-dir`, as the service.
`--redeemer-db` selects a different database. Recovery reads `RPCURLFile` and `ChainID`
for retries; `--rpc-url-file` and `--chain-id` override them. It does not unlock
the account or read unrelated service credentials. RPC endpoints use normal
system TLS trust. This operation uses the exact saved signed bytes, nonce and
hash; it cannot create a replacement
or change fees. A retry error retains the hash and uncertain state.
system TLS trust. Historical hashes remain inspectable; manual retry cannot
reopen a terminal ticket or displace a newer active transaction.

In-memory control state is bounded: payment sessions have a 100,000-entry quota
and 24-hour inactivity retention; ticket replay guards have a 1,000,000-nonce
Expand All @@ -149,11 +169,4 @@ session activity current. Expiry is monotonic within a recipient lifetime so
a backward chain observation cannot revive parameters after their replay guards
are removed. Auth-token refresh never resets nonce tracking. Winning tickets
and uncertain transactions are retained for audit/reconciliation, so back up
and monitor database disk use independently of these quotas.

Redemption planning uses geth's dynamic-fee selection and requires a base fee
in the RPC header; a missing base fee leaves the ticket queued with an error.
`RedeemerMaxFeePerGas` optionally caps the fee in wei per gas; a plan above
that ceiling remains queued without signing or broadcasting. Prepared and
submitted identities reserve their nonces across restarts. Explicit retry
resends the stored transaction without changing its fees.
and monitor database disk use independently of these control-state quotas.
25 changes: 18 additions & 7 deletions eth/contracts.go
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,12 @@ func (c *Contracts) PrepareAndStore(ctx context.Context, plan TransactionPlan, k
}

func (c *Contracts) prepare(ctx context.Context, plan TransactionPlan, key *Key, chainID *big.Int, save func(SignedTransaction) error) (SignedTransaction, error) {
return c.prepareAtNonce(ctx, plan, key, chainID, save, nil)
}

// prepareAtNonce lets the ported redemption backend own nonce allocation through
// submission. The CLI's prepare-and-reserve behavior remains unchanged.
func (c *Contracts) prepareAtNonce(ctx context.Context, plan TransactionPlan, key *Key, chainID *big.Int, save func(SignedTransaction) error, explicit *uint64) (SignedTransaction, error) {
if key == nil || key.Address() != plan.From || chainID == nil || chainID.Sign() <= 0 {
return SignedTransaction{}, errors.New("transaction account does not match signing key")
}
Expand Down Expand Up @@ -297,12 +303,15 @@ func (c *Contracts) prepare(ctx context.Context, plan TransactionPlan, key *Key,
state := c.nonceState(plan.From)
state.Lock()
defer state.Unlock()
nonce, err := c.RPC.ethereum.PendingNonceAt(ctx, plan.From)
if err != nil {
return SignedTransaction{}, safeRPCError(err)
}
if nonce < state.next {
nonce = state.next
var nonce uint64
if explicit != nil {
nonce = *explicit
} else {
nonce, err = c.RPC.ethereum.PendingNonceAt(ctx, plan.From)
if err != nil {
return SignedTransaction{}, safeRPCError(err)
}
nonce = max(nonce, state.next)
}
if nonce == math.MaxUint64 {
return SignedTransaction{}, errors.New("transaction account nonce exhausted")
Expand All @@ -325,7 +334,9 @@ func (c *Contracts) prepare(ctx context.Context, plan TransactionPlan, key *Key,
}
// Once saved or returned to a caller, this identity can be broadcast even
// if its outcome is uncertain. Never reuse its nonce in this client.
state.next = nonce + 1
if explicit == nil {
state.next = nonce + 1
}
return prepared, nil
}

Expand Down
6 changes: 3 additions & 3 deletions eth/contracts/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Contract bindings

The ten generated Go files in this directory and `chainlink/` are copied
unchanged from `livepeer/go-livepeer` commit
unchanged from source commit
`bd645a09266833fb859053445d9ac85846330756`:

- `controller.go`
Expand All @@ -17,7 +17,7 @@ unchanged from `livepeer/go-livepeer` commit

Their filtered source history is a parent of the binding import commit. It
preserves the authors, committers, dates, and messages, with `Original-Commit`
trailers linking each retained commit to `livepeer/go-livepeer`. Contributors
trailers recording each retained commit's original identity. Contributors
include Eric Tang, Yondon Fu, Nico Vergauwen, Rafał Leszko, Rick Staa, and
Victor Elias. The original MIT notice is in
[`LICENSE`](../../LICENSE).
Expand All @@ -27,7 +27,7 @@ copy these files from the selected committed revision, review its contract/API
changes, and update this provenance. For example, from a checkout of this repo:

```sh
git -C /path/to/go-livepeer show REVISION:eth/contracts/bondingManager.go > eth/contracts/bondingManager.go
git -C /path/to/protocol-source show REVISION:eth/contracts/bondingManager.go > eth/contracts/bondingManager.go
```

Application policy lives in the parent `eth` package: canonical snapshots,
Expand Down
Loading
Loading