feat: resolve issues #340, #343, #344, and #347

[ibrahimbabatundeibrahim8-alt]

Resolves 4 issues simultaneously across test coverage, operational documentation, and observability:

1. Issue Add unit tests asserting OTP codes are uniformly distributed and never predictable across repeated calls #340: Statistical smoke test for OTP code distribution

• Verified that crates/email/src/lib.rs uses rand::rngs::OsRng (OS CSPRNG) to generate OTPs.
• Added generate_otp_produces_a_roughly_uniform_distribution_of_digits_across_a_large_sample testing 100,000 generated OTP codes across all 6 digit positions to assert no systematic digit bias or modulo distortions.
• Added generate_otp_duplicate_rate_across_a_large_sample_is_consistent_with_true_uniform_randomness testing duplicate collisions across 100,000 draws from the 1,000,000 code space against the birthday paradox expectation (~95,163 expected unique codes).
• Documented that these tests serve as gross implementation bug smoke tests rather than full cryptographic certifications.
• Closes Add unit tests asserting OTP codes are uniformly distributed and never predictable across repeated calls #340

2. Issue Add a fuzz test for operation_index_from_toid across the full TOID input range #343: Proptest fuzz corpus for operation_index_from_toid

• Added property-based fuzz tests using proptest! in crates/ingest/src/lib.rs.
• Added operation_index_from_toid_never_panics_on_arbitrary_input fuzzing with arbitrary string inputs to guarantee crash freedom.
• Added operation_index_from_toid_extracted_value_is_always_within_the_documented_valid_range_when_some across valid TOID patterns, whitespaced inputs, and boundary numbers asserting that any extracted operation index is non-negative and <= i32::MAX.
• Closes Add a fuzz test for operation_index_from_toid across the full TOID input range #343

3. Issue Document the bin/migrate-keys and bin/backfill-operation-index runbooks #344: Operational runbooks for migrate-keys and backfill-operation-index

• Created docs/runbook-migrate-keys.md providing end-to-end guidance for the dual-key rotation window, pre-flight checks, invocation examples (full rotation and cipher upgrade), healthy log indicators, store method references (Store::list_wallets_needing_reseal, Store::reseal_wallet), and interruption recovery/rollback procedures.
• Created docs/runbook-backfill-operation-index.md providing guidance for historical deposit TOID operation index backfills, pre-flight candidate estimation, dry-run and live invocations, expected logs, store/ingest code cross-references (octo_ingest::operation_index_from_toid), and transactional idempotency guarantees.
• Linked both runbooks in README.md under the Documentation section.
• Closes Document the bin/migrate-keys and bin/backfill-operation-index runbooks #344

4. Issue Add tracing spans around every Horizon call for latency visibility #347: Named tracing spans around Horizon calls

• Added distinct #[tracing::instrument] spans around public Horizon interaction methods in crates/api/src/horizon.rs (balances, account_sequence, account_info, submit_transaction, friendbot_fund) and crates/ingest/src/horizon.rs (payments_after).
• Configured spans to record call metadata (call_type, account, cursor, limit) and dynamic outcome (success, circuit_open, not_found, rejected/tx_failed, failure) while skipping sensitive payloads (such as raw transaction XDR envelopes).
• Added unit tests in crates/api/src/horizon.rs and crates/ingest/src/horizon.rs verifying named span emissions using custom test tracing subscriber layers.
• Added tracing-subscriber to dev-dependencies for octo-api and octo-ingest.
• Closes Add tracing spans around every Horizon call for latency visibility #347

Summary

Related step / issue

Checklist

• cargo fmt --all -- --check passes
• cargo clippy --workspace --all-targets -- -D warnings passes
• cargo test --workspace passes
• No secrets (seeds/keys) logged or persisted in plaintext
• Added/updated tests (test vectors for crypto/derivation changes)
• Updated docs / CHANGELOG if behavior changed

How to test

[drips-wave]

@ibrahimbabatundeibrahim8-alt Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits