Skip to content

Background Worker Schema & Infrastructure #64

Description

@b08x

Sequence: 1 (foundation — no dependencies)

Persistence layer for the async ingest track (#56, #59, #67). Issues like the Async Ingestion Queue (#56) and Dead-Letter Queue (#59) need a place to track in-flight job state that isn't the ingest_review_entries Postgres table (that table is for human review of low-confidence/unrecognized files — see Store::PgIngestReviewRepository — not job scheduling).

Architectural decision: use the pure-Ruby Ohm library mapped to the existing Redis instance (docker-compose port 6380, already provisioned — see AGENTS.md). Avoids pulling in ActiveJob or a heavyweight Postgres-polling queue.

Required work

  1. Boot.rb wiring: add a require_redis: flag to SFL::Boot.call, following the exact same pattern as the existing require_llm:/require_db: flags (see lib/sfl/boot.rb) — opt-in, no side effects when unset, so specs and non-ingest commands are unaffected. Add REDIS_URL to .env/.env.example. Implement connect_redis(env)Ohm.redis = Redic.new(url).
  2. Ohm model: scaffold SFL::Store::IngestJob < Ohm::Model with attributes document_id, file_path, state (pending/pass_one/embedding/pass_two/done/failed), error_trace; indices on state and document_id.
  3. Add gem "ohm", gem "redic" to the Gemfile.

Out of scope here

Anything that consumes IngestJob records (the worker loop is #67; retry/DLQ semantics are #59).

Depends on: nothing. Blocks: #56, #59, #67, #68.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    async-workerRedis/Ohm background worker layer for bulk ingestenhancementNew feature or request

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions