Version: 1.0
Status: Draft
Author: Bernard Ograh
Last updated: 2026-05-20
TaskForge is a self-hosted distributed task queue service inspired by Google Cloud Tasks. It provides a durable, observable, and configurable system for background job execution across distributed services. It exposes an HTTP API for enqueuing tasks, manages retries and scheduling, and routes work to a pool of stateless workers.
The primary goal is a production-grade internal tool that a small engineering team can deploy, operate, and extend without depending on a managed cloud service.
As a backend system scales across multiple services, the need for reliable asynchronous work execution increases. Common pain points without a centralised task queue:
- Work is lost when a service crashes mid-execution.
- Retry logic is duplicated across codebases, inconsistently.
- There is no observability into what jobs are running, failing, or stuck.
- Rate-limiting against downstream services is handled ad hoc.
- Developers have no way to inspect, cancel, or replay failed jobs.
TaskForge solves all of the above in a single deployable service.
- Provide a reliable, at-least-once delivery guarantee for all enqueued tasks.
- Support configurable retry policies with exponential backoff and jitter.
- Handle delayed/scheduled task delivery with second-level precision.
- Detect and reclaim tasks from crashed or stuck workers via heartbeat expiry.
- Enforce per-queue rate limits to protect downstream systems.
- Expose a real-time admin dashboard for queue health and task inspection.
- Be deployable via Docker Compose with Postgres and Redis as the only dependencies.
- TaskForge is not a message broker (no pub/sub fan-out, no topic subscriptions).
- It does not execute tasks itself — workers are separate processes owned by the caller.
- It does not provide a managed SaaS offering or multi-tenancy.
- It does not guarantee exactly-once delivery in the initial version.
- It does not support task dependencies or DAG-style workflows in v1.
| User | Description |
|---|---|
| Backend engineer | Enqueues tasks via the HTTP API from application services. |
| Platform/infra engineer | Deploys and operates TaskForge; configures queues and rate limits. |
| On-call engineer | Uses the admin dashboard to inspect failures and replay DLQ tasks. |
API server — Exposes the REST API for queue management, task enqueuing, inspection, and cancellation. Stateless; horizontally scalable.
Dispatcher — Internal loop that moves tasks from pending into a Redis sorted set for fast dispatch to workers. Respects priority and rate limits.
Scheduler — Sweeps Postgres for tasks whose scheduled_at has passed and promotes them to the Redis working buffer. Runs on a configurable tick (default: 1 second).
Worker pool — Stateless worker processes external to TaskForge that poll the dispatch API, execute payloads, and report results via ack/nack.
Heartbeat watchdog — Background loop that identifies tasks whose heartbeat_at has not been updated within the timeout window and reclaims them.
Admin dashboard — Web UI served by the API server. Shows live queue stats, task history, and a dead-letter queue browser with replay controls.
| Store | Role |
|---|---|
| Postgres | Durable source of truth for all tasks, queues, and DLQ entries. |
| Redis | Hot dispatch buffer (sorted sets by priority), rate-limit counters (token bucket via Lua), scheduler working set. |
pending → claimed → executing → done
↘ retry → pending (loop)
↘ dead-letter (max retries exceeded)
↘ timeout → retry
A task is created as pending. The dispatcher claims it (atomically, via SELECT FOR UPDATE SKIP LOCKED) and assigns it to a worker. The worker begins execution and sends heartbeats. On success, it calls ack; on failure, nack. Failed tasks are rescheduled with exponential backoff until max_retries is exceeded, at which point they are moved to the dead-letter queue.
| Column | Type | Notes |
|---|---|---|
id |
UUID | Primary key |
name |
TEXT | Unique, human-readable identifier |
max_retries |
INT | Default: 3 |
rate_limit |
INT | Max tasks dispatched per second; null = unlimited |
created_at |
TIMESTAMPTZ | — |
| Column | Type | Notes |
|---|---|---|
id |
UUID | Primary key |
queue_id |
UUID | FK → queues |
payload |
JSONB | Arbitrary caller-defined data |
status |
TEXT | pending, claimed, executing, done, retry, dead |
priority |
INT | 1 (highest) to 10 (lowest); default 5 |
scheduled_at |
TIMESTAMPTZ | Earliest dispatch time |
claimed_at |
TIMESTAMPTZ | When dispatcher last claimed |
claimed_by |
TEXT | Worker ID |
heartbeat_at |
TIMESTAMPTZ | Last heartbeat from worker |
attempts |
INT | Number of execution attempts so far |
last_error |
TEXT | Error message from most recent failure |
result |
JSONB | Worker-supplied result on success |
created_at |
TIMESTAMPTZ | — |
Indexed on (queue_id, status, priority, scheduled_at).
| Column | Type | Notes |
|---|---|---|
id |
UUID | Matches original task ID |
queue_id |
UUID | — |
payload |
JSONB | — |
last_error |
TEXT | — |
attempts |
INT | — |
died_at |
TIMESTAMPTZ | — |
replayed_at |
TIMESTAMPTZ | Set when manually re-enqueued |
POST /queues Create a queue
GET /queues List all queues
DELETE /queues/:name Delete a queue (must be empty)
POST /queues/:name/tasks Enqueue a task
GET /queues/:name/tasks/:id Inspect a task
DELETE /queues/:name/tasks/:id Cancel a pending task
GET /queues/:name/stats Queue depth, throughput, error rate
Enqueue request body:
{
"payload": { "user_id": "abc123", "action": "send_welcome_email" },
"priority": 3,
"scheduled_at": "2026-05-19T15:00:00Z"
}POST /workers/heartbeat Extend execution lease (every 10s)
POST /workers/ack Report success + optional result
POST /workers/nack Report failure + error message
GET /queues/:name/dlq List dead-letter tasks
POST /queues/:name/dlq/:id/replay Re-enqueue a dead task
DELETE /queues/:name/dlq/:id Discard a dead task permanently
Backoff delay for attempt n is calculated as:
delay_ms = min(base_ms * 2^n + rand(0, base_ms), max_delay_ms)
Default values: base_ms = 1000, max_delay_ms = 3_600_000 (1 hour).
Jitter (the rand term) is added to prevent thundering herd — without it, all retrying tasks would attempt re-execution at the same moment.
After max_retries attempts, the task is moved to the dead-letter queue and will not be retried automatically.
Rate limits are enforced per queue at the dispatcher level using a Redis token bucket. A Lua script atomically checks and decrements the bucket on each dispatch decision, ensuring no race conditions under concurrent dispatchers.
Workers are never aware of rate limiting — it is transparent at the dispatch layer.
Workers must send a POST /workers/heartbeat with their task ID every 10 seconds while executing. The watchdog loop checks every 15 seconds for tasks where:
status = 'executing' AND heartbeat_at < now() - interval '30 seconds'These tasks are returned to retry status and rescheduled. The worker that was executing them will receive a rejection if it later attempts to ack or nack a task it no longer owns.
Timeout threshold is configurable per deployment (default: 30 seconds).
Current implementation note: The admin dashboard is deferred from the current v1 backend implementation. The backend exposes queue stats, DLQ, task inspection, and active worker endpoints that a dashboard can consume, but /admin is not currently served by the API.
The dashboard is a single-page web UI served at /admin. It requires no separate deployment.
Queue overview panel: Shows all queues with live metrics — depth, tasks/sec, error rate, oldest pending task age. Updates via server-sent events.
Task inspector: Search or filter tasks by queue, status, worker ID, or time range. View full payload, all status transitions, heartbeat history, and error messages.
Dead-letter browser: Lists DLQ tasks with filter by queue and time. Supports single and bulk replay. Shows how many times a task has been replayed.
Worker status: Lists active workers by ID, last heartbeat, and current task. Highlights workers that have not sent a heartbeat recently.
| Milestone | Description | Estimated effort |
|---|---|---|
| 1 — Naive queue | Postgres-only, single dispatcher, basic ack/nack | 1–2 days |
| 2 — Retry engine | Attempt counter, backoff formula, DLQ promotion | 1 day |
| 3 — Scheduler loop | Delayed task delivery, two-tier Redis buffer | 1 day |
| 4 — Heartbeat watchdog | Worker crash detection, task reclaim | 1 day |
| 5 — Priority lanes | Redis sorted set dispatch by priority score | 1 day |
| 6 — Rate limiting | Per-queue token bucket via Redis Lua script | 1 day |
| 7 — Admin dashboard | SSE metrics stream, task inspector, DLQ browser | 2 days |
Total estimated build time: 8–10 days for a working v1.
The API server exposes a /metrics endpoint in Prometheus format with the following metrics:
taskforge_tasks_enqueued_total— counter by queuetaskforge_tasks_completed_total— counter by queue and statustaskforge_task_execution_duration_seconds— histogram by queuetaskforge_queue_depth— gauge by queue and statustaskforge_dlq_depth— gauge by queuetaskforge_worker_heartbeat_age_seconds— gauge by worker ID
Structured JSON logs are emitted for every state transition with task_id, queue, worker_id, from_status, to_status, and duration_ms.
Minimum viable deployment via Docker Compose:
services:
api:
build: .
ports: ["8080:8080"]
environment:
DATABASE_URL: postgres://...
REDIS_URL: redis://redis:6379
depends_on: [postgres, redis]
postgres:
image: postgres:15
environment:
POSTGRES_DB: taskforge
redis:
image: redis:7-alpineWorkers are separate processes deployed by the caller and configured to point at TASKFORGE_API_URL.
- Should task payloads be encrypted at rest in Postgres for sensitive use cases?
- Is a webhook delivery mode (TaskForge pushes to a URL rather than workers polling) in scope for v2?
- Should queue configuration support per-task-type retry overrides, or only queue-level defaults?
- What is the retention policy for completed tasks? Should there be an archival or TTL mechanism?
- Exactly-once delivery semantics
- Task dependency graphs (fan-in/fan-out DAGs)
- Webhook push delivery mode
- Multi-tenant / team isolation
- Task result streaming (currently result is only available after
ack)