Proposal: support queue-scoped worker instances

[obckelbley]

Problem

A Sidequest instance currently processes jobs from every active queue in its backend.

The queues configuration controls settings for named queues, but it does not restrict which queues the instance processes. QueueManager.getActiveQueuesWithRunnableJobs() retrieves every queue represented in the jobs table. If a discovered queue is absent from the configured queues array, Sidequest applies the queue defaults and may process it anyway:

https://github.com/sidequestjs/sidequest/blob/master/packages/engine/src/execution/queue-manager.ts

This means independently deployed worker groups cannot share a backend while processing disjoint sets of queues.

Use case

An application may run separate worker groups for different workload classes while keeping one job database.

For example:

• one worker group processes emails;
• another processes reports;
• the groups have different dependencies, resource requirements, scaling policies, or deployment lifecycles.

An emails worker should never claim a reports job, and vice versa.

Existing workaround

This can be implemented today with a custom backend. The backend can filter getQueuesFromJobs() to expose only permitted queues and refuse claimPendingJob() calls for queues outside that set.

That workaround demonstrates that queue-scoped processing fits the existing execution model, but it has drawbacks:

• queue selection becomes backend-specific even though it is engine behavior;
• the scope must be enforced in multiple low-level methods;
• equivalent implementations are needed for each database backend;
• application code must track both the backend contract and the engine's queue-discovery behavior;
• related maintenance operations require separate consideration.

A first-class engine option, or a backend-neutral queue-selection hook, would express the intent directly and allow the built-in backends to enforce it consistently.

Proposed behavior

Allow an engine to opt into processing a static set of queue names.

Conceptually:

await Sidequest.start({
  queues: [{ name: "emails", workers: 4 }],
  processQueues: ["emails"],
});

The property name is illustrative rather than a proposed final API.

Expected semantics:

1. Existing configurations retain the current all-queues behavior.
2. An explicitly scoped engine discovers and claims jobs only from its configured scope.
3. Jobs on excluded queues remain available to other worker instances.
4. Queue scope controls processing, not enqueueing.
5. Dashboard and administrative operations may remain backend-wide.
6. Automatic maintenance operations have clearly defined queue-scoping behavior.

A static string allowlist should be sufficient; a dynamic predicate does not appear necessary.

Implementation considerations

Filtering queue discovery would prevent excluded queues from reaching the dispatcher. Enforcing the same scope at the claim boundary would protect the invariant if queue discovery changes later.

The implementation should remain backend-neutral and preserve the current all-queues behavior when no scope is supplied.

Acceptance cases

With two engine instances sharing one backend:

• engine A is scoped to queue-a;
• engine B is scoped to queue-b;
• jobs are enqueued on both queues;
• engine A never claims a queue-b job;
• engine B never claims a queue-a job;
• an engine without an explicit scope retains the current all-queues behavior.

The behavior should be consistent across PostgreSQL, MySQL, SQLite, and MongoDB.

The FAQ mentions this capability as future Pro functionality. Is that still the intended boundary, or would an OSS implementation or extension point be considered?

I would be happy to work on an implementation after the desired semantics are agreed.