runner-watch

Runner-watch is a free-data market intelligence service. It watches Stocks, Solana Memecoins and Sports for early movers, records the evidence behind each call, and shows the result on shared List, Detail and Map screens.

The public product runs at https://runners.rati.chat and https://sports.rati.chat. A desktop client (desktop/) ships the local scanner and can attach to the service through a signed node identity.

What runs where

Process	Entry point	Role
Web	uvicorn runner_web.main:app	FastAPI + Jinja screens, JSON APIs, WebAuthn login
Worker	stonks-worker	~20 background collectors, scans, settlement and delivery tasks
Trainer	stonks-trainer	Ranker training and historical backfill
Desktop	desktop/ (Tauri)	Local scanner, node API and swarm transport
Edge	cloudflare-router/	Public host routing in front of Fly

PROCESS_ROLE selects the role (web, worker, trainer, or all for local development). Split roles require REDIS_URL for shared rate limits, jobs and cache.

Repository layout

src/runner_web/     Online service: routes, data workers, product logic
src/runner_watch/   Market scanner, EDGAR, risk and scoring
src/runner_node/    Desktop node API and local scan runtime
src/runner_swarm/   Signed node identity, transport and reputation
rust/stonks-ranker/ Integer ranker compiled into the production image
ml/sec-qwen/        SEC fine-tuning sources
web/                Jinja templates and static assets
desktop/            Tauri desktop client
cloudflare-router/  Edge worker
tests/              Unit and Playwright browser suites
docs/               Product and design documents (see docs/README.md)
scripts/            Operational and release scripts

Local development

Requirements: Python 3.11–3.13, uv, and Node for the desktop client.

uv sync --extra dev            # install the service and test tools
uv run stonks-migrate          # create/migrate the SQLite database
uv run uvicorn runner_web.main:app --reload --port 8080

The full stack (Postgres, Redis, migrate, web, worker, trainer) runs with Docker:

docker compose -f compose.local.yml up --build

compose.local.yml is the reference for the production process boundaries and for local-only values. Production values live in fly.toml.

Testing

uv run pytest                                     # unit suite (browser tests excluded)
uv run pytest -m browser -o addopts=              # Playwright browser suite
uv run ruff check src tests ml/sec-qwen/src       # lint

The two suites can run in one process: tests/conftest.py suspends Playwright's private event loop around non-browser tests. Running the browser suite alone remains the fastest path.

Configuration

Configuration is environment-only. 163 variables are read across src/; the values below are the ones that change behavior or are required in production. Unset optional features simply stay off.

Runtime and host

• APP_ORIGIN, RUNNERS_ORIGIN, SPORTS_ORIGIN, LEGACY_ORIGIN — public hosts.
• RP_ID, LEGACY_RP_ID, COOKIE_SECURE, COOKIE_DOMAIN — WebAuthn and cookies.
• PROCESS_ROLE, BACKGROUND_WORKERS_ENABLED, WORKER_EXPECTED_INSTANCES.
• TRUST_FLY_CLIENT_IP, EDGE_PROXY_SECRET, REQUIRE_EDGE_PROXY_SECRET.

Storage

• DATABASE_URL or DATABASE_PATH, REQUIRE_DATABASE_URL, REQUIRE_DATABASE_TLS, ALLOW_NEWER_DATABASE_SCHEMA.
• REDIS_URL, REQUIRE_REDIS_TLS, RATE_LIMIT_HASH_KEY, REQUIRE_RATE_LIMIT_HASH_KEY.

ALLOW_NEWER_DATABASE_SCHEMA=1 is deliberate in production: the release command migrates before the new image serves traffic, and a rollback must tolerate a schema written by the newer release. Keep it off elsewhere.

Secrets (never commit; set through flyctl secrets or the desktop vault)

• OPENROUTER_API_KEY, HELIUS_API_KEY, MASSIVE_API_KEY, ODDS_API_KEY, FINTEL_API_KEY, COURTLISTENER_API_TOKEN, SAM_API_KEY, STRIPE_SECRET_KEY, STRIPE_WEBHOOK_SECRET, TELEGRAM_BOT_TOKEN, TELEGRAM_WEBHOOK_SECRET, OPERATIONS_TOKEN, SWARM_NODE_PRIVATE_KEY.

Use scripts/configure-production-security to generate the edge secret, operations token and invite codes before the first production deploy.

Feature switches default to off and are turned on per environment: DISCOVERY_SOURCES_ENABLED, SPORTS_INGESTION_ENABLED, ODDS_API_ENABLED, SEC/HOUSE_DISCLOSURES_ENABLED, MEMECOINS_ENABLED, FINTEL_SHORT_DATA_ENABLED, ROBINHOOD_CHAIN_ENABLED (read-only Stock Token metadata on the stock detail), TELEGRAM_*_ALERTS, and the RANKER_* training knobs. Budget caps and intervals (HELIUS_DAILY_CREDITS, ODDS_API_MONTHLY_WORKING_LIMIT, BACKGROUND_SCAN_INTERVAL_SECONDS, …) are documented in the module that owns them.

Scaling roadmap

Runner-watch scales by separating write-heavy collection from read-heavy product traffic. Four contracts stay fixed as the system grows:

• every derived result keeps its source, event time and collection time;
• entity matches remain dated claims with review and correction history;
• retries remain safe and every terminal state remains visible;
• model promotion requires replayable data, metrics and an exact artifact ID.

Target data path

The target shape separates collection, durable evidence, derived views and public reads. Each source family can scale without changing the evidence contract used by the rest of the service.

flowchart LR
    subgraph Ingest[Ingestion]
        Sources[Market, filing, chain and sports sources]
        Queue[Partitioned ingestion queue]
        Workers[Source worker pools]
        Raw[(Raw document archive)]
        DB[(Postgres evidence store)]
        Sources --> Queue --> Workers
        Workers --> Raw
        Workers --> DB
    end

    subgraph Compute[Derived state]
        Identity[Entity reconciliation]
        Trainer[Model training and replay]
        Models[Versioned model artifacts]
        Views[Materialized read models]
        DB --> Identity --> Views
        DB --> Trainer --> Models --> Views
    end

    subgraph Serve[Serving]
        Client[Web and desktop clients]
        Edge[Edge router]
        Web[Web and API pool]
        Cache[(Redis cache and leases)]
        Client --> Edge --> Web
        Web --> Cache --> Views
        Web --> DB
    end

Loading

Area	Current design	Next scale step	Proof before promotion
Ingestion	One worker schedules bounded collectors and records source runs, hashes, counts and errors.	Move each source family to queue-backed consumers. Partition work by source and market. Archive large raw payloads in object storage while keeping their hashes and metadata in Postgres.	Peak queue age stays below one collection interval. Replaying the same source documents produces the same normalized rows.
Entity model	Entities, references, dated claims and resolution events keep observations separate from accepted identity.	Add an append-only relationship event log, bounded reconciliation workers and materialized read models for the List, Detail and Map screens.	The same evidence produces the same entity revision. Corrections preserve old reports, links and receipts.
Database	Postgres owns durable production state. Redis carries short-lived jobs, caches and rate limits. SQLite supports local development.	Move Postgres to a managed high-availability service with point-in-time recovery and read replicas. Partition the largest event tables by market and time. Send analytics to a separate store.	Restore drills meet the recovery target. Replica delay, query latency and partition growth stay within their budgets.
Failure handling	Leases, bounded attempts, explicit retry states, heartbeats and worker supervision recover common failures.	Add per-source circuit breakers, dead-letter queues, replay tools, distributed traces and source-level service objectives.	Kill-and-restart tests lose no receipts, create no duplicate deliveries and recover before the stated deadline.
Model evaluation	Training uses chronological train, calibration and untouched test groups. Predictions keep the model ID and input evidence. Deterministic risk rules retain final veto power.	Add walk-forward replay, champion/challenger shadow runs, cohort-level calibration checks and drift alerts by session and market regime.	A candidate beats the declared baseline on frozen metrics and cohorts. Its data, code, configuration and result hashes reproduce the decision.
Deployment controls	CI runs lint, unit, browser, audit and image checks. Releases migrate first, verify exact builds, smoke-test public screens and roll back on failure.	Add signed images, a software bill of materials, schema compatibility gates and separate canaries for web, worker and trainer roles. Stage provider changes behind environment switches.	Canary health, schema compatibility, worker heartbeats and public receipts pass before wider traffic moves.

Recovery path

Every queued operation uses a stable idempotency key and a lease token. A worker can stop at any point without losing the original evidence or making a second delivery.

stateDiagram-v2
    state "Dead letter" as DeadLetter
    [*] --> Queued
    Queued --> Leased: worker claims lease
    Leased --> Completed: commit output and receipt
    Leased --> Queued: lease expires
    Leased --> Retry: bounded failure
    Retry --> Queued: backoff ends
    Retry --> DeadLetter: attempt limit reached
    DeadLetter --> Review: operator inspects evidence
    Review --> Queued: replay approved
    Completed --> [*]

Loading

Scale sequence

flowchart LR
    Observe[1. Observe] --> Isolate[2. Isolate]
    Isolate --> Store[3. Store]
    Store --> Serve[4. Serve]
    Serve --> Evaluate[5. Evaluate]
    Evaluate --> Distribute[6. Distribute]

Loading

The scale sequence is:

1. Observe. Track queue age, collector duration, database latency, cache hit rate, worker heartbeats, replay time and model calibration.
2. Isolate. Split source families into independent workers while keeping one job contract and one idempotency rule.
3. Store. Add table partitions, raw-document archives, high-availability Postgres and read replicas.
4. Serve. Build bounded read models, coalesce duplicate refreshes and protect hot keys from cache stampedes.
5. Evaluate. Shadow new models and providers before they affect public scores, reports or alerts.
6. Distribute. Keep one ordered writer for each feed or market and place read replicas closer to users. Add multi-writer paths only after ordering, idempotency and conflict rules have executable tests.

Each scale change should land as a small release with a load receipt, a replay receipt and a recovery receipt. Latency, queue growth, storage growth or missed recovery targets should trigger the next step.

flowchart LR
    Change[Candidate change] --> Load[Load receipt]
    Load --> Replay[Replay receipt]
    Replay --> Recovery[Recovery receipt]
    Recovery --> Canary[Role-specific canary]
    Canary --> Gate{All gates pass?}
    Gate -->|Yes| Promote[Promote]
    Gate -->|No| Rollback[Roll back]

Loading

Deployment

Pushes to main go through .github/workflows/fly.yml:

1. Lint, type-free unit tests, browser tests, dependency audit and Docker build.
2. Save the current release image for rollback.
3. Deploy, deploy the edge worker, keep worker/trainer processes alive and heartbeating.
4. Smoke-test both origins and render every production screen.
5. Roll back automatically and fail the run if any step fails.

Rollback is also manual: flyctl deploy --image <previous-image> --skip-release-command.

Documentation

Start at docs/README.md. It marks which design documents are current and which are historical records.