Skip to content
ilovepixelartPublic

About

Async-first, Redis-backed job queue for Python: atomic Lua state transitions, at-least-once delivery, fully typed asyncio API. pip install toro-queue

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

3 stars

Watchers

0 watching

Forks

Repository files navigation

toro 🐂

An async-first, Redis-backed job queue for Python. Every state transition is an atomic Lua script; producing and processing are asyncio end to end.

Python
PyPI Downloads License
Coverage Quality Gate Status
Reliability Rating Maintainability Rating Security Rating

pip install toro-queue      # the import name is `toro`

Installed as toro-queue on PyPI (the name toro was taken), but you import toro. See the docs for the architecture, the reliability model, and the detailed guides. Coming from another Python task queue? Migrating maps the vocabulary. Wondering whether you need a queue at all? The FAQ answers "why not just use Postgres" honestly. Upgrading? Breaking changes are listed per release in Upgrading.

Pairs with matador, a live web dashboard for your queues.

Why toro

  • Async-native. Enqueue and process with async/await on one event loop, a natural fit for FastAPI, aiohttp, or any asyncio app. A plain def processor works too, run in the worker's own thread pool.
  • Atomic by construction. Claims, retries, promotions and finishes are Lua scripts, so a job can't be lost or double-committed between two round trips.
  • At-least-once delivery. Per-job locks + a background mark-and-sweep recover jobs from workers that crashed - without the visibility-timeout double-delivery trap of some other queues.
  • Typed. Ships py.typed; the public API is fully annotated.

Features

Enqueue delayed jobs, global priorities (FIFO within a band)
Retries fixed or exponential backoff, capped attempts
Schedules repeatable cron and fixed-interval (every) jobs
Flows parent/child job trees: fan-out/fan-in, failure policies, flow-aware retry
Rate limiting queue-wide token bucket shared across all workers
Global concurrency one cap on jobs active at once, across every worker process
Dedup custom (idempotent) job ids + a throttle window ({id, ttl})
Serialize by key concurrency_key: jobs sharing a key run one at a time, in order, without holding a worker
Cancellation cancel_job() stops a job wherever it is, a running one included: an async processor is cancelled where it awaits, a sync one ends when its thread returns
Bounded history keeps the newest 1000 completed / 5000 failed by default; or the last N, an age, or everything
Reliability per-job locks, lock renewal, stalled-job recovery
Observability progress, per-job logs, lifecycle events, await result()
Metrics metrics_text(): OpenMetrics for any scraper, backed by counters that survive a restart
Sync handlers a plain def processor runs in the worker's own thread pool, so the loop stays free
Blocked-loop alarm a processor that starves the event loop is named, with the jobs in flight, the moment the loop runs again
Transactional enqueue pending() collects jobs and sends them as one pipelined write once your write commits
Lifecycle pause / resume, graceful shutdown that drains in-flight jobs
Dashboard matador - a live web UI

Quick start

import asyncio
from toro import Queue, Worker

async def main():
    queue = Queue("emails")
    await queue.add("welcome", {"to": "ada@example.com"})

    async def process(job):
        print("sending", job.data)
        return {"ok": True}

    worker = Worker("emails", process, concurrency=8)
    worker.on("completed", lambda job, result: print("done", job.id))
    await worker.run()

asyncio.run(main())

A taste of the options

# Priorities, delay, and retry-with-backoff
await queue.add("report", data, priority=10, delay=5000,
                attempts=5, backoff={"type": "exponential", "delay": 1000})

# Idempotent custom id (a second add with the same id is ignored)
await queue.add("charge", data, job_id="order-1234")

# A repeatable schedule (cron or every-N-ms); "run now" with trigger_scheduler
await queue.add_scheduler("nightly-rollup", cron="0 0 * * *")

# A flow: children run first (fan-out), the parent runs on their results (fan-in)
from toro import FlowChild as c
report = await queue.add_flow("report", {"q": 3},
                              children=[c("fetch", {"shard": i}) for i in range(3)])

# Queue-wide rate limit: at most 100 jobs / second across every worker
await queue.set_limits(rate_limit={"max": 100, "duration": 1000})

# Wait for a result from the producer side
job = await queue.add("resize", {"src": "a.png"})
print(await job.result(timeout=30))

Flows

A flow enqueues a parent and its children as one atomic tree. The children run first (fan-out, nested arbitrarily); the parent parks until every child has settled, then runs and reads their results (fan-in). One primitive covers fan-out/fan-in and chained steps, with per-child failure policies and flow-aware retry that recovers a whole failed flow in one shot. Full guide: docs/flows.md.

Develop

Managed with uv; the Astral toolchain throughout.

uv sync                          # venv + deps + dev group
uv run ruff check .              # lint  (strict: select = ALL)
uv run ruff format .             # format
uv run ty check                  # type check
REDIS_URL=redis://localhost:6391 uv run pytest -m "unit or integration"   # integration needs a Redis you own
uv run python examples/basic.py

The suite is a pyramid - -m unit (fast, no Redis), -m integration (Redis), and -m load (the open-loop benchmark harness in tests/load/).

License

MIT

About

Async-first, Redis-backed job queue for Python: atomic Lua state transitions, at-least-once delivery, fully typed asyncio API. pip install toro-queue

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages