Background Jobs & Queues Guide
Anything slow, flaky or retryable does not belong in the request path: sending email, resizing images, calling LLMs, syncing webhooks. Accept the work, enqueue it, return fast.
Queues come in two families — hosted services like Cloudflare Queues, AWS SQS, Google Cloud Tasks and Upstash QStash, and Redis-backed libraries like BullMQ, Sidekiq and Celery that need a long-lived worker process somewhere.
Picking a queue for your stack
Match the queue to where your compute runs:
- Cloudflare Queues — Workers produce and consume, with batching and a dead-letter queue built in
- AWS SQS + Lambda — event source mappings, visibility timeouts and partial-batch failure handling
- Upstash QStash — delivers jobs as HTTP calls, ideal for serverless apps with no worker to run
- BullMQ on Redis — rich scheduling and flows, but requires a persistent worker on Render, Railway or Fly.io
Write handlers that survive retries
Delivery is at-least-once everywhere, so idempotency is the core discipline: key each job with a deterministic ID and skip work already done. Retry with exponential backoff plus jitter, cap attempts, and route exhausted jobs to a dead-letter queue that alerts someone. Distinguish permanent failures (bad input — do not retry) from transient ones (timeouts — retry), and keep handlers short by chunking long work.
Scheduling and cron
Cron belongs next to your compute: Cloudflare Cron Triggers, Vercel Cron Jobs and Amazon EventBridge Scheduler all fire on schedules. Keep the cron handler tiny — it should enqueue work, not perform it — and guard against overlapping runs with a lock. Inngest and Trigger.dev add multi-step workflows, delays and fan-out when plain cron plus a queue stops being enough.
Observe your jobs
Track job status in a table — queued, running, done, failed with the error — keyed by a correlation ID that threads through your logs. Alert on queue depth and oldest-message age, not just failures: a stuck consumer looks healthy until customers notice. Build replay tooling early; re-running a failed job safely is the payoff of idempotent design.
Frequently asked questions
Can I run BullMQ on Vercel or Workers?
Not well — BullMQ workers need a persistent process. On serverless platforms use QStash, Cloudflare Queues or Inngest, which deliver jobs over HTTP.
What is a dead-letter queue for?
It catches jobs that exhausted their retries so they are inspectable and replayable instead of silently lost. Alert on anything landing there.
How do I stop duplicate job execution?
Design for at-least-once delivery: derive an idempotency key from the job payload and check it before doing work or sending side effects.
Where do scheduled tasks fit in?
Use platform cron — Cron Triggers, Vercel Cron, EventBridge Scheduler — to enqueue jobs on a schedule, keeping the heavy work in queue consumers.