Skip to content

Pass 2 (Ruby LLM) Batching & Rate Limiting #63

Description

@b08x

Sequence: 4 (depends on #67, parallel with #58/#59)

Pass 2 (LLM::Engine/LLM::Classifier/LoaderDrafter, all already Core::Ports::Breaker-wrapped per-call) has no batch-level rate-limit awareness — under bulk ingest, many sfl-worker jobs hitting the same provider concurrently will trip provider-side rate limits faster than any single job's own circuit breaker will catch.

Scope: a worker-level (not per-call) rate limiter/backoff shared across concurrent IngestJob Pass 2 dispatches within one sfl-worker process — e.g. a token-bucket or fixed-window limiter wrapping the chat_factory.for(:ingest_classification)/:loader_drafting calls, configurable per-provider. Distinct from the existing Core::Ports::Breaker (which handles per-call failure isolation, not proactive throughput shaping).

Depends on: #67.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    async-workerRedis/Ohm background worker layer for bulk ingestenhancementNew feature or request

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions