A Go service that monitors a configured selection of Symbiotic vaults and runs a pluggable solver against them. A solver is a self-contained integration with an external protocol that sources, prices, or routes liquidity on top of a Symbiotic vault adapter; the bot handles discovery, pricing/signing, on-chain reads, reconciliation, and settlement for it.
The framework is solver-agnostic: each integration lives in its own package, registers itself, and is selected by config — adding one never touches the generic engine. The available integrations are listed under Solvers.
Status: early build. Engineering guidelines:
CLAUDE.md. Per-solver scope, architecture, and roadmap live underdocs/.
cmd/vault-solver— process bootstrap: flags, logging, signal-driven shutdown.internal/solver— genericSolverinterface, registry, and engine.internal/solvers/<name>/— one self-contained package per integration; all protocol-specific logic lives here.internal/{config,chain,signer,txmanager}— solver-agnostic infra: two-stage config, vault / Multicall3 reads, a pluggable signer, and a nonce-serialized transaction broadcaster that shares one unresolved signed lifecycle across solvers.api/— committed codegen: contractbindings/(abigen) and protocol API clients, each refreshable from upstream.
State is intentionally minimal — positions, liquidity, and readiness are read from on-chain views and the relevant protocol API on each tick; no database.
Solvers are listed in config under solvers: — one or more, at most one entry per solver type.
Every solver shares the chain client and signer. Transaction-sending solvers also share the single
nonce-serialized txManager, so multiple solvers on one EOA never race on nonces. Solvers whose
settlement is submitted externally do not start it. Each entry's config block is typed and validated
by its own solver. Adding a solver touches no framework code — see the recipe in
CLAUDE.md.
Sharing is deliberately process-scoped. Deploy solvers that use a different signer, read-RPC set, or private
write endpoint as a separate process with its own config subset and txmanager. Assign each scrape target a
unique Prometheus instance (and optionally a stable lane target label). One EOA must never be configured
in two processes: independent txmanagers would race on its nonce even when their RPC URLs differ. Solvers
that share an EOA belong in one process so they retain one serialized nonce lane.
solver.name |
Integration | Docs | Example config |
|---|---|---|---|
3f-bridge-facilitator |
3F (Grunt) bridge-loan auctions | plan | yaml |
rfq-filler |
Symbiotic RFQ quoting + order filling | plan | yaml |
redstone-oev |
RedStone OEV liquidations | plan | yaml |
lifi-samechain |
LI.FI same-chain intents over LiquidLane | plan | yaml |
uniswapx-filler |
UniswapX V2 RFQ quoting and LiquidLane filling | plan | yaml |
All solvers expose a pluggable
strategy — the built-in default or an external webhook you run; see
Strategies.
Acts as a Bridge Facilitator in 3F (Grunt)'s bridge-loan auctions, on top of one
or more Symbiotic BridgeFacilitatorAdapters. 3F auctions the right to front a bridge loan; this solver bids on behalf
of its adapters, funds the loans it wins just-in-time, and permissionlessly redeems repaid loans back
to the vault with yield.
It holds no API key: each adapter is registered with 3F by its vault creator, who authorizes this
solver's signer as the adapter's offer signer — directly (an EOA) or via an EIP-1271 contract signer —
so offers are authorized by signature alone. Design, config,
and roadmap: docs/3F-PLAN.md. When adapters is present, the solver operates only
on that explicit list. Otherwise it discovers all entries of the configured on-chain IAdapterFactory,
refreshing before each auction-discovery pass with a hard 2,000-entity safety limit; a larger reported
count is an error. Either source is filtered to non-zero vault/asset targets that authorize this
solver's signer (validated via the adapter's ERC-1271 isValidSignature). An empty factory is valid and
is polled until eligible adapters appear. Example:
config/3f.example.yaml.
Named mainnet instances are configured in vault-solver-deploy, using this same
rfq-filler integration. Presto is vault-solver-presto (symbiotic_presto in RFQ
backend); its executor and single adapter are set in that repository's mainnet
deploy matrix. A new instance does not require a new solver type in this application.
Public solver IDs use symbiotic_<name>; workloads, services, and Vault paths use
vault-solver-<name>.
An externally-owned solver/executor for Symbiotic RFQ, on top of per-vault
LiquidLaneAdapters. It runs a POST /quote server that prices swaps for the RFQ backend and a poller
that fills the orders it is awarded, settling on-chain through the adapter.
It runs either in external mode (the open-source filler; quoting and filling scoped to the operator's
own adapters) or internal mode (Symbiotic-internal; adds the private discounts flow). The caller EOA
must be an authorized caller of the RFQ Executor (its setCallers allowlist, granted by the owner).
External mode also fails startup unless that executor has direct owner/marketMaker/isFiller
authorization on every configured adapter; the fatal startup log includes the executor, configured adapters,
and underlying authorization error.
When tokensToQuote: permissioned, admitted inputs are never aggregated: the selected strategy must
use one candidate route. Other scopes keep the existing multi-route behavior.
minAmountsIn adds an optional per-input-token floor on request size (base units, decimal strings):
a request below its token's minimum is not quoted (HTTP 204), while an amount equal to the minimum
still quotes; unlisted tokens have no floor.
Pareto's mainnet AA_FalconXUSDC tranche (0xC26A…f99C) uses this existing generic path and needs no
token-specific solver code. The production deployment config already includes it in
permissionedTokens with a one-token minAmountsIn floor. A solver instance can route it only after
its configured LiquidLane adapter has onboarded the token. Execution also requires either direct
owner/marketMaker/isFiller authorization or a live signed discount in internal mode.
When an exact-input request exceeds the advertised adapter capacity, the default strategy caps the
quoted output at the available maxAssets instead of declining in every token scope; the excess input
is reflected as worse execution price and price impact. Awarded orders are planned again from current
LiquidLane state at fill time; the solver does not retain quote-time route plans.
RFQ keeps quoting while fills are queued or pending. As soon as a won order is polled, its planned output
is reserved against the vault capacity it spends, including capacity reached through a discount. Quotes and
later fill plans subtract every reservation until the order expires or fails; an order the backend stops
reporting expires locally at its own deadline. When the backend reports the block an adapter's maxAssets
was read at (adapters[].blockNumber on /quote), a confirmed fill is subtracted only from snapshots older
than its inclusion block, for a few hours after the order is filled, and never from newer ones, which
already reflect it. Fills are still
sent one at a time on the shared nonce lane. Reservations are local to the process and are not restored
after a restart.
Design, config, and roadmap:
docs/RFQ-PLAN.md · example
config/rfq.example.yaml.
An off-chain bidder for RedStone Atom OEV auctions. When a price update makes a Morpho Blue position liquidatable, RedStone runs a sub-second WebSocket auction for the right to be the liquidator; this solver bids, and on winning, its signed payload is bundled atomically with the price update and the liquidation.
On settlement it liquidates the position and exits the seized collateral through a single Symbiotic
LiquidLaneAdapter, realizing the spread and paying its bid. It signs and bids but never submits the
settlement transaction — RedStone's auctioneer does. The solver config owns the RedStone Executor,
LiquidLane adapter, and callback address; the selected strategy owns the callback-specific
operationData. Operators can set maxBidWei as a per-auction spend ceiling over any strategy; it is
required for the external webhook strategy and optional for the built-in default. The common gas:
block is optional, and its shared oracle facts are passed to the selected strategy. The built-in strategy
uses them for after-cost economics; without them, it selects gross-profitable bundles while retaining the
signed gas-price cap and native funding checks. When gas: is configured, startup requires a feed for the
resolved adapter loan asset and a readable initial oracle snapshot. Design, config, and roadmap:
docs/OEV-PLAN.md · example
config/redstone-oev.example.yaml.
A same-chain LI.FI Intents solver for LiquidLane-backed RWA → underlying routes. It publishes standing quotes
from current adapter liquidity with optional gas accounting and receives matched, already-opened escrow orders over the
LI.FI WebSocket feed. On startup and reconnect it catches up active matches through GET /orders before
publishing quotes; while disconnected it suspends renewal and retries withdrawal of known curves
until the order server acknowledges removal.
After REST recovery completes, WebSocket closes 1000/1001/1005/1006/1012/1013 are logged at Info.
Earlier disconnects and other errors remain Error; reconnect backoff resets only after recovery. Before each fill it
rechecks the canonical order status, adapter state, configured gas cost, and strategy decision, then atomically claims
the input, redeems it through LiquidLane, and fills the output via
LiquidLaneLifiExecutor. Capacity reserved by already-submitted fills is deducted from both later fill
decisions and standing quotes until those transactions complete. Each token pair advertises its allocated share of
available capacity when several pairs share one vault; accepting a fill reserves its shared CapacityID
and immediately refreshes every affected quote. The reservation remains until the shared tx manager returns a
terminal result under the shared confirmation policy.
Pre-sign or definitive broadcast failures release the reservation without a receipt. Before
signing and after a receipt sweep without a valid receipt, the tx manager rechecks the LI.FI order status.
An observed Claimed or Refunded then switches the owned nonce to cancellation instead of
retaining liquidity until pendingTimeoutMs. None, an unrecognized status, or an unavailable status read
leaves the current lifecycle unchanged and is retried, so a lagging latest-state RPC cannot cancel a fresh fill.
Orders
that the built-in strategy proves fillable without, but blocked by, pending reservations enter a bounded FIFO
without blocking later deliveries. The worker retries them after every reservation release and returns a still-
blocked order to the tail. During startup/reconnect recovery, quote publication remains suspended until each
recovered order leaves the FIFO, either resolved or returned to the recovery sweep. Overflow drops the newest
retry. A webhook null decision and an order-specific 400/422 fill rejection stay terminal; other
strategy failures get at most three attempts per order during each recovery session. On graceful
shutdown the solver keeps the feed alive while it expires active curves with the configured order-server HTTP
timeout, then stops accepting orders and waits for already-accepted fills until completion or the finite process
hard stop.
If a newly opened order reaches the feed before the RPC endpoint exposes its deposit, the worker retries the
status-None read with bounded exponential backoff capped at 5 seconds until the 30-second window or earlier
order deadline. The final scheduled read is clamped to 250 milliseconds before that boundary. Duplicate
deliveries are coalesced during the wait; claimed, refunded, and unknown statuses remain terminal. Stopping
intake drops these unaccepted retries immediately.
The published quote ladder is not replayed at fill time: the
solver greedily rebuilds the best current route plan, and redeemed output above the order requirement remains
executor surplus. The default strategy trims an uneconomic range prefix to the first input whose conservative
floor yields a positive output, then prices the published suffix by running the shared LiquidLane exact-input
quote solver at both endpoints. It caps the lower of the two endpoint rates by that floor for interior route
transitions, rounding, and, when configured, worst-case route gas.
strategy.config.rangeCount sets the geometric curve resolution (default 8, maximum 16).
Omitting LI.FI's gas: block disables gas accounting in quote/fill decisions and skips gas-state and
Chainlink reads; the tx manager still prices and pays the actual transaction gas.
The executor contract is the registered LI.FI solver account. It is registered once through EIP-1271 using
a caller signature bound to the executor's EIP-712 domain, appears as exclusiveFor in quotes, and calls the
settler's direct finalise path. The framework signer is an authorized executor caller and transaction sender;
fills do not carry a per-order AllowOpen signature.
The owner manages callers, while ERC-1271 validates domain-separated registration signatures against the
current callers.
Our deployment convention is one LI.FI API key per registered executor contract. LI.FI can register multiple accounts under one key, but this deployment deliberately does not share a key across executors. All processes using one executor therefore share its API key and LI.FI reputation; active/active operation also requires external order coordination. The API key, executor owner key, and caller transaction key are distinct credentials.
Only on-chain escrow orders are supported; gasless Compact, Permit2/3009, Dutch auctions, and future-order
scheduling are out of scope. Dutch (0x01) and exclusive Dutch (0xe1) orders are ignored at order-feed
admission and logged as unsupported. Fully valid feed orders routed to another origin or output chain are
expected noise and logged at info; malformed identifiers and operational failures remain errors.
solverMode: external serves direct filler-authorized adapters.
solverMode: internal also enables signed private discounts through the shared backend. tokensToQuote uses the same all,
permissioned, and permissionless scopes as RFQ; permissioned inputs must execute through one physical
route. The order-server REST/WS endpoints are explicit required config. When gas: is configured, each
Chainlink feed has its own required max age. The default strategy evaluates bounded geometric exact-input ranges across
available capacity. See the plan for settlement, pricing, concurrency, and onboarding details:
docs/LIFI-PLAN.md · example
config/lifi.example.yaml.
The opened-order settler must report governanceFee() == 0. The solver checks this at startup and again for
every admitted order. Startup fails closed; at runtime an unreadable or non-zero fee skips the order with an
error log before planning or submission.
The implementation is ready for the opened-order path. The next live E2E requires deploying the current executor build, registering it with LI.FI, and granting it filler authorization on the target adapter.
An Ethereum-mainnet UniswapX solver backed by LiquidLane routes. It serves the RFQ POST /quote
webhook, polls the Uniswap order API for exclusive and public V2 orders, resolves
their Dutch amounts from current chain time, and fills executable orders through a configured
LiquidLaneUniswapXExecutor. The executor uses the same owner-managed caller list as the RFQ executor and
remains the Reactor-facing filler. Before serving traffic, the solver validates executor bytecode, finds the
tx-sending EOA in the executor's indexed callers list, and, in external mode, checks every configured
route's direct authorization. Failures log the relevant executor, caller, or adapters and the underlying
reason before startup returns. The executor ABI has no Reactor getter, so matching the configured Reactor to
the deployed immutable remains a deployment assertion. solverMode: external is the default, requires a
non-empty adapters list plus direct authorization, and forbids the discounts block. solverMode: internal
requires that block; direct routes are authorization-filtered from each snapshot while valid signed-discount
routes remain usable. discounts.baseUrl supports HTTP for backends within local infrastructure, as in
RFQ and LI.FI, as well as HTTPS. orderServer.baseUrl still requires HTTPS outside loopback. In internal
mode adapters is optional: a non-empty list scopes quotes and direct fills, while fill-time signed-discount
recovery may use any adapter advertised by the backend. Without a list the solver quotes and fills
discount-only. Every fill is simulated again immediately before submission. The wall-clock anchor for
a fill is captured before reading chain time, so RPC and planning latency consume the order's remaining
validity instead of extending it.
Set strategy.name: single to use the local UniswapX single-source strategy.
It uses the same pricing config as default and the normal sources allowed by solverMode.
The solver prefers the quoted source at fill time, then tries alternatives covering the awarded output.
Fresh signed terms and simulation validate the chosen source before submission.
The optional quoteServer.selectionTtl (default 10m) and quoteServer.maxSelections (default 4096)
bound the in-memory source preferences. Eviction or restart causes fresh selection from the signed order.
Once the order is known, the preference applies only through its exclusivity deadline; later fills select afresh.
Quotes do not reserve capacity. Single-source fills retain the default strategy's configured price
buffer but do not require additional output to repay gas; the sender pays gas and fee caps still apply.
The existing default retains its configured gas-coverage requirement for fills.
The quote path uses a refreshed on-chain inventory snapshot so it stays within Uniswap's
response deadline. Each request is priced once for its concrete amount: the strategy returns one
amountIn/amountOut pair after price buffer and, when configured, estimated fill gas, with no precomputed
ladders, amount ranges, or quote-time route reservation. Omitting the entire gas: block disables gas
accounting in both quote and fill decisions and skips gas-state and Chainlink reads. The tx manager still
prices and pays actual transaction gas, so that cost is then subsidized by the solver. Uniswap deliberately
makes indicative and hard RFQ requests
indistinguishable, so the solver echoes quoteId but does not guess the phase. Fill planning reserves
selected capacity before signature resolution, simulation, and transaction admission for all strategies
(default, single, and webhook). Quoting continues during planning, queueing, and confirmation;
source fallback replaces the reservation. Until a plan installs its reservation, its capacity remains
quotable. Pending reservations conservatively reduce every source limit in the same vault/output token.
A subsequent fill can wait for the preceding transaction and miss its admission or exclusivity deadline;
see fill capacity and retries.
Transaction completion invalidates cached inventory before releasing its reservation; quoting resumes
after a latest-state refresh. Unsubmitted attempts release capacity without invalidating inventory.
A quote is returned only if inventory, reservations, and blocking conditions remain unchanged during
calculation. Quoting fails closed during startup warmup, stale or unknown exclusive-order delivery,
an unavailable nonce lane, an active Uniswap blockUntilTimestamp, or the configured local fade breaker. A claimed order is requeued before chain reads,
signed-discount resolution, calldata construction, or preflight while the nonce lane is paused. A txmanager
result rejected before the worker lifecycle does not count toward the local fill breaker and is reported as
solver_bot_txmanager_admission_rejections_total{label="uniswapx-fill"} rather than a terminal fill
failure. GET /ready exposes that state and
also returns not-ready when the latest snapshot has no quotable inventory;
GET /health and its probe-friendly alias GET /healthz remain liveness-only.
Every valid exclusive order assigned to the executor is tracked through decayStartTime. After that
deadline, tracked hashes are reconciled in batches against the order API and canonical transaction receipts.
A successful on-chain fill at or before the deadline clears the obligation, including another filler's soft
override. A fill by any filler only after the deadline—including our executor—or any known non-filled
terminal state for an obligation observed live or recovered after a runtime poll gap opens the separate
local fade breaker, matching Uniswap's
fade definition.
An already-terminal miss found only by initial startup history reconciliation is logged and terminalized
without opening a fresh local breaker.
If terminal status or receipt time cannot be established, quoting stops without opening the breaker until
reconciliation succeeds.
The upstream /orders endpoint returns only the newest 50 rows and no longer paginates. A full 50-row open
snapshot is processed but treated as incomplete; exclusive quoting and readiness stay blocked. Recovery
filters the newest all-status snapshot locally and clears the unknown state only when that snapshot reaches
the configured lookback cutoff.
In internal mode, advertised LiquidLane routes are resolved on-chain and checked against their advertised
asset and decimals, current physical capacity/rate, adapter minimum discount, token policy, and configured
gas feeds. Configured adapters scope quoting when present; fill-time discount recovery remains unrestricted,
matching RFQ solver-mode semantics. A selected discount is resolved again immediately before simulation and
encoded as a typed discountSwap; its adapter, token, output floor, signatures, and expiry window are
checked fail-closed.
The order API key is required and read indirectly through orderServer.apiKeyEnv. Uniswap's public quote
contract specifies source-IP allowlisting rather than an application header, so restrict the quote endpoint
to the published Beta/production source IPs at the ingress. The order API URL must use HTTPS except for
loopback development servers. Each V2 order carries its swapper-authorized cosigner; the solver verifies its
cosignature directly, so there is no static cosigner setting to rotate. Exclusive V2 polling is mandatory
while the quote server is enabled; public V2 filling remains independently opt-in. Legacy V1 limit orders
are not supported. The generated order client follows upstream order-service spec version 2.0.0 and decodes
the current DutchV2OrderEntity, including nested cosignerData, cosignature, and createdAt.
Native-asset outputs are currently declined because the supported LiquidLane routes settle ERC-20 vault
assets.
Exact-input and exact-output Dutch auctions are supported. Exact-output quotes directly size enough input
for the requested output, buffer, and gas; rounding or execution output above that requirement remains
executor surplus. If a Dutch exact-output input grows between planning and execution, the executor consumes
the planned route input and retains the positive input difference as filler surplus. The Reactor atomically
enforces the order's aggregate outputs. Multiple outputs are supported when every output uses the same
ERC-20; mixed-token outputs fail closed because one
LiquidLane route produces one vault asset. Quote webhook protocols v1 and v2 are accepted, while V3
orders and secondary-DEX routes are not supported. Design,
config, onboarding, and deployment prerequisites:
docs/UNISWAPX-PLAN.md · example
config/uniswapx.example.yaml.
The solvers split protocol plumbing (reads, signing, submission — fixed) from the decision — how to size, price, and select — which is a pluggable strategy, chosen in config:
default— the built-in in-process strategy for that solver.single— the local UniswapX strategy that selects one source for the entire request. It chooses the highest final output for exact input, or the lowest required input for exact output, including configured quote buffer and gas costs. A better price with insufficient capacity is rejected; the strategy does not combine sources to cover the request. Each source can use the vault's available capacity up to its own limit, after inventory reserves and pending fills.webhook— delegates each decision to an external HTTP service you run: the solver sends it the raw facts as JSON and executes the validated plan it returns, so your service owns the logic. LI.FI and UniswapX own separate strategy contracts and independently reject returned fills that exceed current capacity or do not cover the order plus configured gas. UniswapX delegates each concrete quote toPOST /decide-quoteand each current fill plan toPOST /decide-fillunder the configured webhook URL.
The single-source calculation is shared in LiquidLane; the strategy and its configuration belong to
UniswapX. RFQ and LI.FI reject strategy.name: single at startup.
Each solver controls its eligible sources, order requirements, and execution. Selecting single changes
how liquidity is chosen within that source set.
This is the seam for customizing a solver without forking. Contract and trust model:
docs/strategy-plan.md.
The shared txManager serializes transaction-sending solvers on one EOA. While a transaction is queued
or active, UniswapX declines new quotes, LI.FI retires standing curves, and 3F stops new offers;
reconciliation continues. RFQ keeps quoting and accounts for pending fills through reservations; it stops
only while the nonce lane is conflicted. Pending calls can be replaced or cancelled with the same nonce. Each pending
receipt RPC has its own timeout and does not block the lifecycle loop's replacement/cancellation timers.
Configure maxFeeGwei for every transaction-sending process. It also caps cancellation; tipGwei sets a
priority-fee floor, or selects fee-history pricing when zero. replacementIntervalMs, pendingTimeoutMs,
broadcastTimeoutMs and shutdownTimeoutMs control replacement, cancellation and shutdown bounds.
The manager remains alive while solvers drain accepted work; orchestrator SIGTERM grace must cover both
solver preparation/drain and manager shutdown. A timeout does not guarantee that a signed call cannot land.
Defaults, fee headroom, request/result semantics, nonce recovery and internal ownership are documented in the transaction manager plan. Integration-specific deadline and capacity rules remain in each solver's plan.
For liquidity commitments, the built-in strategies apply these limits:
- 3F counts all live offer principals and request slots before creating offers for another auction.
Expiration is at least
now + offerExpiryBuffer. WebhookliveOffers[]now includes decimal-stringprincipal; remote strategies must reserve it as well as the live request slot. - RFQ external mode excludes discount inventory at quote time. Excess input can be absorbed only by a
direct swap, whose calldata caps output. After a successful cancellation reaches the configured
confirmations, a still-open, unexpired order can be retried with a fresh fill plan and newly resolved
discount signatures.
solvers[].config.maxCancellationRetriesdefaults to3additional attempts (0disables them); retries wait at least onepollIntervalMsinterval before a fresh open-order poll can re-arm the order. Reverted transactions are not retried, and uncertain fill or cancellation inclusion is reconciled through the backend. Retry counts are local to each process and reset on restart. - LI.FI and UniswapX split shared vault capacity across token pairs before quoting. A pair can therefore quote less than the vault's total free liquidity. This does not reserve every repeated quote request.
- The default OEV strategy permits one pending bundle per adapter. New auction frames arriving during a decision are skipped; liquidity and gas use the full estimated callback output, while the configured haircut applies to profit. Search shortlists at most 512 candidates using the selected profit objective.
- Go (toolchain version pinned in
go.mod; auto-fetched by recent Go releases). - For regenerating codegen:
make tools(installs pinnedabigen,golangci-lint). OpenAPI clients use the Java openapi-generator, downloaded on demand byhack/openapi-generator-cli.sh(needs a JRE). - A reachable EVM RPC endpoint and a signing key (see Configuration).
make build # build ./bin/vault-solver
./bin/vault-solver version
make test # go test -race -cover ./...
make test-txmanager-anvil # real pending replacement/cancellation against local Anvil
make lint # golangci-lint
./bin/vault-solver run --config config/3f.example.yamlThe CLI is built with Cobra; run vault-solver --help for the
command list (run, version). Debug logging is off by default; enable it with
observability.debug: true in config or the --debug flag (the flag wins):
./bin/vault-solver run --config config/3f.example.yaml --debugThe observability listener (default :9090) serves /metrics, /healthz, and /readyz. No extra
config is required for the collectors below. /readyz reports nonce safety: it fails before startup
completes, during shutdown and while a nonce conflict pauses the shared transaction manager, but not while
a transaction is merely pending, so quote servers stay in rotation. During graceful shutdown readiness
drops first, while liveness and metrics remain available until the shared transaction manager finishes its
bounded drain.
Tracing is off unless OTEL_EXPORTER_ENABLED is 1, true, yes, on, or enabled (the same
switch the RFQ backend uses). Everything else is the standard OpenTelemetry environment, read by the
SDK: OTEL_EXPORTER_OTLP_ENDPOINT (default http://localhost:4318, OTLP over HTTP/protobuf),
OTEL_SERVICE_NAME (default vault-solver; set it per deployment, e.g. vault-solver-rfq),
OTEL_RESOURCE_ATTRIBUTES, OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG (default
parentbased_always_on; use parentbased_traceidratio to thin background-loop traces without
dropping backend-initiated ones), OTEL_EXPORTER_OTLP_HEADERS, and the OTEL_BSP_* batch settings.
OTEL_TRACES_EXPORTER and OTEL_EXPORTER_OTLP_PROTOCOL are ignored: the exporter is always OTLP/HTTP.
There is no YAML equivalent; tracing is configured only by these variables.
# deploy/docker-compose.yml, or -e flags on docker run
environment:
OTEL_EXPORTER_ENABLED: "true"
OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4318"An inbound /quote continues the caller's trace; every outbound HTTP call, JSON-RPC call, and
transaction carries traceparent onward. Log lines under traced work carry trace_id and span_id,
and Sentry events are tagged with trace_id. Spans carry solver, quote.id, order.id, tx.hash,
and the other identifiers listed in docs/TRACING-PLAN.md. A fill links back to
the quote that produced it when the process still remembers that quote (best effort, in-memory);
after a restart the fill simply starts a new trace. LI.FI fills are not linked, because its standing
quotes have no per-request quote event to link from.
A websocket or IPC RPC endpoint is traced at the call level, one span per JSON-RPC call plus one for
the dial; it receives traceparent only on the websocket handshake, so the provider can tie the
connection to your trace but not an individual call.
No secret is ever recorded on a span: RPC endpoints appear as ordinals rather than URLs, and an outbound client span records the request URL without its query string, so a webhook URL that carries a token in the query does not put it in the trace.
Tracing never blocks a quote or a fill: spans are exported in the background from a bounded queue,
export failures are logged at Info, and a bad OTEL_* setting disables tracing at startup instead of
failing it. The observability listener itself is never traced.
LI.FI order rejection logs include available orderId, onChainOrderId, orderType,
inputSettler, originChainId, a stable reason_code, and the failing field/field_value.
Context is extracted independently of typed decoding. Allowlisted scalar values are capped at
160 UTF-8 bytes; containers are summarized, and signatures, full payloads, callback data and
auction context are excluded. Zero identifiers are reported without assuming their asset meaning.
The envelope's type, chain and input settler are checked before interpreting token identifiers.
Foreign chains use unsupported_chain; a different nonzero EVM input settler is an expected
unsupported_settler debug skip, including clean foreign output settlers/oracles. Native input
(token identifier zero) is rejected with unsupported_native_input at Info, without Sentry.
Multiple inputs/outputs, nonempty callbacks, and unsupported pricing-context types are also expected
skips at the existing Debug level. Dutch auctions and foreign chains retain their Info logs.
These observations use the existing unsupported/other_chain workflow outcomes, never invalid.
Zero still cannot execute through the solver's ERC-20 path.
Malformed identifiers, missing inputs/outputs, invalid known-context lengths, RPC failures and
operational invariant violations remain Error/Sentry. Permanent rejection alone does not silence an
error: only explicitly classified unsupported formats skip Sentry. Dirty input identifier bits retain
invalid_token_identifier.
Receipt lookup errors retain streak suppression and include rpcBudgetTotalMs, sweepElapsedMs,
hashesChecked, hashesTotal, rpcChecks, lastRPCDurationMs and cancelCause. Each RPC has its own
rpcTimeout; the budget total sums effective budgets of completed calls, including priority reads,
and is not a sweep deadline. Checked hashes are distinct; rpcChecks also counts repeated reads.
hash and cancelCause describe the first failed read, while the last-RPC duration describes the
last call. Shutdown still stops the reader without adding a partial-sweep error log.
Existing RPC count/duration metrics measure the scale of RPC failures.
RFQ backend calls propagate the existing X-Request-Id from request context, generating an ID
with the same mechanism when polling has no inbound request. Public orders and private discount
calls retain this outbound ID in typed client errors and the structured backendRequestId log/Sentry
context, without adding it to the error text or Sentry title. HTTP request IDs are separate from business
requestId/quoteId fields in JSON. End-to-end correlation requires the backend to accept and
log the same header; solver-side propagation alone does not establish that guarantee.
Discount responses use X-Request-Id for correlation; the client reads that header
and does not require a JSON requestId. Deploy this client update together with
the backend change that removes the duplicate body field.
Sentry groups these diagnosed errors by (solver, message, reason_code); other errors retain
(solver, message). Dynamic identifiers remain event context. No additional log sites are introduced.
The txmanager metric reference covers transaction outcomes, admission, replacements, phase timing and account snapshots, including labels and units.
The registry also includes standard Go/process collectors,
solver_bot_build_info{version,commit}, and solver_bot_solver_info{solver}. The first identifies the exact
binary behind a sample; the second exposes bounded config-time process membership so fleet dashboards can
map each scrape instance/execution lane to its solvers without inferring ownership from traffic.
| Scope | Metric family | Labels | What it shows and why it is useful |
|---|---|---|---|
| LI.FI | solver_bot_workflow_events_total{event="order_parse"} |
solver, strategy, event, outcome |
Rejected feed observations: invalid, unsupported (including Dutch auctions), or other_chain. REST recovery replays count again; this is not a unique-order count. Uses the existing workflow event family and its last-event timestamp. |
| Framework | solver_bot_service_ready |
— | 1 exactly when the shared /readyz gate reports ready, otherwise 0. This is process and nonce-safety readiness; a pending transaction does not clear it. It is not a claim that every solver upstream is healthy; combine it with solver freshness and connectivity. |
| Framework | solver_bot_solver_info |
solver |
Constant 1 for each solver configured in this process. Prometheus target labels such as instance/lane make process membership explicit without adding deployment-specific labels in application code. |
| Framework | solver_bot_external_operation_duration_seconds |
solver, strategy, operation, outcome |
Count and latency of allowlisted recurring solver operations such as polls and authoritative refreshes. Outcomes are bounded to success, degraded, skipped, or error; errors and request-derived values never become labels. |
| RPC | solver_bot_rpc_requests_total |
role, method, outcome |
Logical HTTP JSON-RPC calls. Roles are read, write, cancel, or shared; methods and outcomes are bounded, with transport, HTTP 3xx/4xx/5xx, rate-limit, decode, context, and JSON-RPC errors separated. Redirects are not followed; 3xx responses fall through to the next read endpoint. |
| RPC | solver_bot_rpc_attempts_total |
role, endpoint, method, outcome |
Per-endpoint attempts, including failed primary and successful fallback attempts. endpoint is only a role-local ordinal (0, 1, …); configured URLs and error text are never labels. |
| RPC | solver_bot_rpc_inflight |
role |
Calls whose response bodies have not completed; a sustained value exposes a hung endpoint or consumer. |
| RPC | solver_bot_rpc_request_duration_seconds |
role, method, outcome |
End-to-end HTTP JSON-RPC latency through response-body consumption. |
| RPC | solver_bot_rpc_last_successful_request_timestamp |
role |
Last successful logical call by endpoint role. |
| RPC | solver_bot_rpc_last_successful_attempt_timestamp |
role, endpoint |
Last successful endpoint attempt, so an idle or dead fallback can be distinguished from a healthy primary. |
| Workflow | solver_bot_workflow_events_total |
solver, strategy, event, outcome |
Bounded solver events. Event/outcome pairs are fixed at construction; request data and errors cannot create labels. |
| Workflow | solver_bot_workflow_dropped_observations_total |
solver, strategy, reason |
Observations rejected because code used an undeclared event/outcome, amount kind, or state view. Any increase is an instrumentation contract drift signal; reasons are bounded. |
| Workflow | solver_bot_workflow_last_event_timestamp |
solver, strategy, event, outcome |
Last occurrence of the matching bounded event, including successful fills, quotes, wins, settlements, and refreshes. |
| Workflow | solver_bot_workflow_amount_atomic_units_total |
solver, strategy, event, asset, kind |
Event amounts in asset atomic units. Never aggregate unlike asset values; planned_surplus is gross planning output, not realized PnL. |
| Workflow | solver_bot_workflow_observed_items |
solver, strategy, view |
Last complete authoritative item count for a bounded state view. |
| Workflow | solver_bot_workflow_last_observation_timestamp |
solver, strategy, view |
Freshness paired with each retained workflow state count. |
| RFQ | rfq_filler_http_request_duration_seconds |
method, route, status |
Quote-server request count (_count), status funnel, and latency. Routes are allowlisted and methods are normalized to GET, POST, or other to bound cardinality. |
| RFQ | rfq_filler_http_requests_total |
method, route, status |
Deprecated one-release compatibility counter for existing alerts; migrate to rfq_filler_http_request_duration_seconds_count. |
| RFQ | rfq_active_orders |
— | Current queued, submitting, submitted, or cancellation-retry obligations awaiting terminal backend state. |
| RFQ | rfq_oldest_active_order_age_seconds |
— | Age of the oldest active obligation; catches a single stuck order that a count-only alert can miss. |
| LI.FI | lifi_active_quotes |
— | Process-local quote count from the last successful publication or suspension reconciliation. It can remain nonzero after the remote quotes expire at quoteTtl, so use it with refresh freshness rather than as backend state. |
| LI.FI | lifi_active_quote_ranges |
— | Number of currently active standing-quote ranges from the last successful reconciliation. |
| LI.FI | lifi_active_quote_max_input_atomic_units |
token_in, token_out, token_in_decimals, token_out_decimals |
Largest currently advertised input range ceiling per token pair. Alternative curves are maxed rather than summed, so the gauge does not double-count shared capacity. |
| LI.FI | lifi_last_successful_refresh_timestamp |
— | Freshness of standing-quote publication or suspension reconciliation; distinguishes an authoritative zero from a dead reconciliation loop. |
| LI.FI | lifi_order_feed_connected |
— | 1 only while the order-feed loop owns an established WebSocket; 0 while disconnected, dialing, or backing off. |
| LI.FI | lifi_order_recovery_ready |
— | 1 only when the current established order-feed connection has completed convergent REST recovery; every disconnect or reconnect resets it to 0. |
| LI.FI | lifi_order_backlog |
stage |
Current process-local orders waiting in inbox, recovery_retry, capacity_retry, or deposit_retry. An item actively being processed is not queued. |
| LI.FI | lifi_order_nearest_deadline_timestamp |
stage |
Nearest protocol order deadline among work waiting in each stage; 0 when that stage is empty or its queued orders have no deadline. |
| UniswapX | uniswapx_quote_duration_seconds |
— | End-to-end quote-handler latency across all request outcomes. |
| UniswapX | uniswapx_exclusive_obligations_outstanding |
— | Live-observed or recovered obligations still awaiting terminal classification. |
| UniswapX | uniswapx_exclusive_nearest_deadline_timestamp |
— | Nearest outstanding exclusivity deadline; alerts on urgent or stuck obligations. |
| UniswapX | uniswapx_block_until_timestamp |
— | Maximum deadline among remote, local-fill, exclusive-fade, and startup-warmup time-based quote blockers. |
| UniswapX | uniswapx_ready |
— | Scrape-time availability: 1 only when current quote state, breakers, exclusive delivery, and the transaction nonce lane permit quoting. |
| UniswapX | uniswapx_last_quote_refresh_timestamp |
— | Last atomic quote-state publication. A successful publication may contain no inventory, so freshness alone is not readiness. |
| UniswapX | uniswapx_last_exclusive_poll_timestamp |
— | Last successful exclusive poll plus recovery/obligation reconciliation. |
| UniswapX | uniswapx_pending_fills |
— | Admitted fills holding LiquidLane capacity while awaiting a txmanager terminal result. |
| OEV | oev_won_inflight |
strategy |
Locally observed winning bids still awaiting settlement. |
| OEV | oev_oldest_won_inflight_age_seconds |
strategy |
Age since the oldest still-inflight win was locally observed; 0 when no locally won reservation remains. |
| OEV | oev_hotpath_seconds |
strategy |
End-to-end handling latency for parsed auction frames against the auction's short decision budget. |
| OEV | oev_deposit_wei |
strategy |
Executor deposit from the last complete state refresh; use it to monitor settlement runway. |
| OEV | oev_deposit_below_floor |
strategy |
1 when the Executor deposit is below the settlement floor, without requiring dashboards or alerts to duplicate the contract threshold. |
| OEV | oev_feed_connected |
strategy |
1 only after the WebSocket is connected and every configured subscription frame has been sent; 0 before dial, during backoff, and from teardown onward. |
| 3F | threef_backlog_nonempty_since_timestamp |
view |
Process-local timestamp when complete authoritative snapshots first began continuously reporting a non-empty active_requests or redeemable backlog. 0 also means no authoritative non-empty observation has occurred yet, so pair it with view freshness. It resets on restart and is deliberately not presented as an individual request age. |
Bounded workflow dimensions:
| Solver | Events and outcomes | Amount/state dimensions |
|---|---|---|
| RFQ | quote/<decision>, order/won, order_poll/success, fill/{success,failure,not_admitted} |
quote/{input,output} and successful fill/{input,output,planned_surplus} by asset |
| LI.FI | order_processing/<result>, queue_drop/<stage>, fill/success |
Fill amounts by asset and kind |
| UniswapX | quote/<decision>, {exclusive,public}_order_poll/{ok,failed}, exclusive_obligation/{won,settled_in_time,missed}, fill/{success,failure,not_admitted,declined} |
Quote and successful-fill amounts by asset and kind; quote amount assets are restricted to the immutable route snapshot used for that decision |
| OEV | auction/<decision>, bid/{enqueued,won,settled_success,settled_failed,would_bid,unresolved}, breaker/failure, state_refresh/success |
Native bid amounts use asset="native"; kind is the bid stage, including dry-run would_bid |
| 3F | offer/{success,error}, redeem/success; state views are targets, offers, active_requests, redeemable |
Offer principal and expected_yield by deposit asset |
OEV bid/unresolved records a local settlement timeout, not a mutually exclusive terminal state. A later
result also advances settled_success or settled_failed, so lifecycle dashboards must not reconcile
unresolved and settlement outcomes as disjoint counters.
Event timestamps reset to 0 on restart; use max_over_time(...[$__range]) when a dashboard should
retain a pre-restart observation inside its selected range.
3F skips adapters with a zero offer signer, vault, or asset without blocking snapshot freshness.
Discovery failures and errors reading required vault(), offerSigner(), or asset() values still
prevent a complete target snapshot. A reverted or malformed isValidSignature() response instead
excludes the adapter as unauthorized without making discovery incomplete.
External-operation labels are fixed at construction: 3F exposes target_refresh, offer_refresh,
active_request_refresh, and redeemable_refresh; RFQ exposes order_poll; LI.FI exposes
quote_refresh, quote_suspend, and order_recovery; UniswapX exposes quote_refresh,
exclusive_order_poll, and public_order_poll; OEV exposes state_refresh. degraded means a safe
partial or last-known-good path remained usable, while skipped means a deliberate gate, stale-plan
discard, or shutdown cancellation. Transaction sends are outside these timers.
LiquidLane counters include successful receipts reported as included_unconfirmed; they are operational telemetry rather than an accounting ledger, and amounts for
different token labels must not be added without price/decimal normalization. An inclusion observed only
during shutdown can still be reorged after process exit, so it may overcount a success and, for UniswapX,
clear the local fill-failure breaker; accounting systems must use canonical on-chain data instead.
Six native Grafana Dashboard Schema v2 templates are committed under
dashboards/: one unified dashboard each for Runtime, 3F, RFQ, LI.FI, UniswapX, and OEV.
Choose Namespace to select the environment. Solver dashboards aggregate all replicas in that
namespace. Runtime also has a Pod (RPC / resources) filter for RPC and process diagnostics;
All includes every replica, and the selector includes pods observed earlier in the selected time
range. Execution totals remain aggregated across the namespace regardless of the pod selection.
Counter increases are calculated per process before summing, percentiles use combined histogram
buckets, and shared account balances are not summed across replicas.
Expanded amount tables show observed increases during the selected period. Retained counter peaks are in collapsed diagnostics: they include the first observed nonzero sample but may also include activity before the selected period, so they are not period volume. Token amounts are shown in atomic units and must not be summed across different tokens without decimal and price normalization.
Each JSON file contains a Dashboard Schema v2 specification. Assign its dashboard identity through
the Dashboard resource's metadata.name when provisioning; neither a dashboard UID nor a datasource
UID is embedded in the template. Grafana resolves the empty ${datasource} selection on load.
Discovery uses the scrape labels namespace and kubernetes_pod; adapt these labels to your
collector if needed. Runtime RPC queries normalize exported_role to role when the scraper has
renamed the exporter's role label, and otherwise retain the native role label.
For LI.FI, UniswapX, and RedStone OEV gas accounting, choose feed age limits using the
shared oracle freshness rules.
A chain.multicallAddress override must support aggregate3 and getCurrentBlockTimestamp;
incompatible contracts fail startup when gas accounting is enabled.
Config is YAML with a two-stage decode: the framework reads solver.name to select the
implementation and hands the opaque solver.config block to that solver to type. Each solver has its
own fully annotated example under config/ (see the Example config column above) — every field,
including the applicable shared chain/signer/txManager/observability blocks, is documented
inline there.
The chain block takes a primary rpcUrl plus optional rpcFallbackUrls — HTTP(S) endpoints tried
in order for reads when the primary is unavailable. chain.rpcAttemptTimeoutMs bounds each HTTP(S)
endpoint attempt, including reading its response body, for read, write and cancellation RPCs.
It defaults to 20000 (20 seconds) when omitted or zero; negative or overflowing values are rejected.
For example, rpcAttemptTimeoutMs: 5000 allows up to 5 seconds per endpoint attempt. This is not a
total fallback-chain budget: a shorter caller deadline is divided across the remaining endpoints.
The txmanager's shorter fee/receipt budgets and broadcastTimeoutMs still apply; WebSocket/IPC calls
are unaffected. When using eRPC, this bounds the solver's wait for eRPC, including eRPC's internal
retries; upstream timeouts and retries inside eRPC must be configured separately.
Normal signed broadcasts and both startup nonce reads
are pinned to writeRpcUrl, or the primary rpcUrl when it is omitted, and never fall over across
endpoints. Optional cancelRpcUrl routes only same-nonce self-cancellations, including their fee replacements
and exact rebroadcasts, to a separate endpoint. Set cancelRpcUrl: ${CANCEL_RPC_URL} and, for mainnet,
CANCEL_RPC_URL=https://boost.rpc.mevblocker.io/fast. When omitted or empty, cancellation uses the ordinary
write RPC. A configured cancellation RPC failure is returned without broadcasting to another endpoint.
Sender balance and nonce telemetry (the periodic account snapshot behind the solver_bot_txmanager_account_*
metrics) always uses the read RPC, never writeRpcUrl, so a submission relay that rate-limits reads cannot
stall it; only broadcasts, startup nonce reads and replacement nonce checks reach the write endpoint. Receipt confirmation uses the
canonicality checks independently of endpoint
affinity, while retaining normal read fallbacks. An HTTP 3xx response is not followed and falls through to the next read
endpoint. A non-final endpoint's JSON-RPC null receipt or header result falls through
to the next read endpoint; the final endpoint's null remains the ordinary not-found result. Unavailable
multi-read snapshots retry on a later poll; OEV compares both number and hash around each latest-state
snapshot and retries a changed head once immediately. A second crossing fails startup or retains the runtime's
last-known-good snapshot until the next poll. Explicit write and cancellation endpoints must report the same chain ID as the
read endpoint.
For transaction-sending solvers, startup fails closed when the write endpoint's pending nonce differs
from its latest mined nonce because txManager cannot recover an unknown signed lifecycle. The EOA
must be exclusive to this process: standard nonce reads cannot reveal a future transaction queued
beyond a gap. Before upgrading from a build that allowed several unresolved signed nonces, drain that
EOA's write-endpoint pool. After an unclean exit, nonce equality alone cannot rule out a private
submission hidden by its relay. The packaged Docker Compose deployment restarts automatically with
unless-stopped, so it can resume and reuse that nonce before the hidden submission becomes visible. If
the old attempt later consumes the nonce, txManager pauses admissions and readiness and remains
fail-closed for operator investigation; automatic restart does not recover the lost in-memory ownership.
For controlled maintenance, stop the service and reconcile outstanding private submissions before bringing
the EOA back.
Before replacing or rebroadcasting a pending transaction, the manager checks the latest mined nonce.
If it has already been consumed, broadcasting stops and tracked receipts are reconciled even when the
submission RPC previously returned success. A failed nonce read defers the replacement until a later
attempt. Unexplained nonce consumption pauses admission/readiness until ownership is established. See
nonce conflict and restart behavior
for exact-hash reconciliation and reorg handling. LiquidLane state reads always use RPC latest; an archive node
is not required.
Never commit a real key or live config — keys are supplied via env/file behind the Signer
interface; *.local.* and .env are gitignored.
Generated code is committed for hermetic builds; refresh from upstream on demand:
make refresh-abi FORGE_OUT=../rfq/out # re-vendor contract ABIs from a Foundry build
make refresh-openapi # re-pull the live 3F OpenAPI spec
make refresh-rfq-openapi # re-pull the RFQ backend OpenAPI spec
make generate # regenerate bindings + API clientEngineering conventions — the modular framework/integration boundary, config-driven configuration,
modern Go 1.27 style, the required test/lint/format gate, and secure-coding rules — are in
CLAUDE.md (AGENTS.md is a symlink to it). Every change must keep
make format && make test && make lint green and unit-test new logic.