[Hollujay]

Summary

• Add recovery for submitted requests that do not yet have an authoritative
  submission hash.
• Reconcile against authoritative Stellar network evidence.
• Persist the confirmed submission hash when positive confirmation is obtained.
• Leave uncertain/network-failure cases unchanged for later retry.
• Keep existing pending submission retry behavior intact.
• Add regression coverage for recovery and safety cases.

Problem

A request can enter submitted state before the real submission hash is
persisted. If the process fails at the wrong point, the request can remain
submitted with no submission hash and is not currently selected by the normal
pending retry path.

Fix

A new background reconciliation pass (src/reconcile.ts, wired into the
existing scheduler in src/background.ts alongside expiry and retry) selects
a bounded batch (RECONCILIATION_BATCH_SIZE, default 100) of submitted rows
still missing submission_hash, oldest first.

For each row, it looks the transaction up on the network by tx_hash — the
hash of the unsigned transaction envelope, computed at creation time. This is
a safe, deterministic lookup key: Stellar's transaction hash is computed over
the transaction body only (not the signatures), and the stored envelope is
never mutated between creation and submission, so tx_hash is exactly the
hash the network will report for this submission.

Three outcomes:

• Found and successful — the network's own reported hash (never a locally
  derived value) is persisted to submission_hash via a guarded
  UPDATE ... WHERE status = 'submitted' AND submission_hash IS NULL. This
  guard is what makes the write safe under concurrency (see Safety).
• Not found, or a transient/network failure — the row is left completely
  unchanged so a later pass can retry it.
• Found but not successful (the network positively recorded the
  transaction as failed) — also left unchanged; this is logged at warn for
  operator review. There is no existing terminal status for "submitted but
  failed on-chain", and introducing one was out of scope for this change, so
  no automatic state transition or resubmission happens here.

This PR does not implement or claim automatic resubmission of any ambiguous
transaction.

A new partial index (migrations/003_add_reconciliation_index.sql) backs the
bounded batch query so it never sequential-scans the full table.

The Horizon lookup is abstracted behind TransactionLookupGateway
(src/types.ts, implemented by HorizonTransactionLookupGateway in
src/horizon.ts), the same pattern already used for account state and
submission. Tests substitute FakeTransactionLookupGateway
(test/helpers.ts), which can simulate a confirmed transaction, a not-found
result, or a thrown network failure.

Safety

• No false confirmation: submission_hash is only ever written from the
  network's own response to a successful, positive lookup — never derived
  locally, never written on a "not found" or on a found-but-failed record.
• Uncertain results never mutate state: a 404 ("not found") is explicitly
  not treated as proof of non-submission (propagation/indexing delay could
  explain it), and any other lookup failure (timeout, 5xx, connection error)
  is caught and logged, leaving the row untouched for the next pass.
• Concurrency: the persisting write is a single atomic
  UPDATE ... WHERE status = 'submitted' AND submission_hash IS NULL. If two
  reconciliation passes (this instance or another) race on the same row, only
  the first commit applies; every later one is a no-op — proven under a real
  Postgres-backed concurrent-write test, not mocked.
• No interference with existing behavior: reconciliation only ever
  touches rows already at status = 'submitted' with submission_hash IS NULL; it never changes status, so it cannot race with expiry (which only
  acts on pending rows) or with the normal submission/retry path.

Testing

• npm test — 64/64 passed (56 pre-existing + 8 new reconciliation tests)
• npm run typecheck — passed
• npm run lint — passed
• npm audit --omit=dev — 0 vulnerabilities
• git diff --check — clean
• Real Testnet smoke test: not performed. There is no existing
  project tooling for a live Testnet smoke test, and building a new E2E
  harness for this task was explicitly out of scope. This change is
  unit/integration tested only, against a real Postgres database and a fake
  network gateway (FakeTransactionLookupGateway) — real Horizon behavior
  (404 mapping, response shape) was verified by reading the installed
  @stellar/stellar-sdk source, not exercised live.

Scope

This PR does not change:

• coordinator-web
• decoder behavior
• dependency versions
• rate limiting
• API error-response normalization
• cancel/notification/metrics features