Making Postgres Queues Scale: The 2026 Guide to Reliable Job Queues on PostgreSQL

Teaser: PostgreSQL can serve as a serious job queue — with SKIP LOCKED, partial indexes, and careful vacuuming it handles millions of jobs reliably. This article walks through the queue patterns that scale, the failure modes that break naive implementations (thundering herds, table bloat, priority starvation), and when to graduate to dedicated infrastructure.

Introduction

In July 2026, a Hacker News front-page article (from DBOS) revisited a perennial topic: making Postgres queues scale. The article demonstrated that PostgreSQL — with the right patterns — can handle queue workloads that many teams assume require Redis, RabbitMQ, or SQS.

The claim deserves scrutiny. Naive Postgres queues (a jobs table with a status column and a polling worker) break down at surprisingly low throughput — 100–1000 jobs/s — due to row contention, index bloat, and thundering-herd polling. But with three core techniques — SKIP LOCKED, partial indexes, and worker coordination — Postgres queues reach 10K–100K jobs/s for typical workloads.

This article explains why naive queues fail, the patterns that fix them, the performance numbers, and the decision framework for when Postgres is the right queue and when it isn't.

Why Naive Postgres Queues Fail

A typical naive implementation looks like this:

-- Naive queue table
CREATE TABLE jobs (
    id          BIGSERIAL PRIMARY KEY,
    payload     JSONB NOT NULL,
    status      TEXT NOT NULL DEFAULT 'pending',
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Naive worker polling loop
SELECT * FROM jobs
WHERE status = 'pending'
ORDER BY created_at
LIMIT 10;

This fails for three reasons:

1. Race Conditions (Lost Jobs)

Two workers can SELECT the same rows before either updates them. Both process the same job — or worse, the second UPDATE fails and the job is lost. Classic SELECT-then-UPDATE race.

2. Thundering Herd

Every worker polls the same query every few hundred milliseconds. All workers scan the same index, contend on the same pages, and generate heavy lock traffic. At 50 workers × 3 polls/s, that's 150 index scans/s hammering the status index.

3. Index Bloat

The naive pattern updates status from pending → processing → done. Under PostgreSQL's MVCC, every UPDATE creates a new row version and invalidates the old one. The pending index entry becomes dead space that must be vacuumed. Under sustained load, the table and index balloon, and query performance degrades until a disruptive VACUUM FULL.

Pattern 1: SKIP LOCKED — The Foundation

SKIP LOCKED is the single most important technique. It makes the queue safe for concurrent workers:

-- Claim jobs atomically, skipping rows locked by other workers
WITH claim AS (
    SELECT id
    FROM jobs
    WHERE status = 'pending'
    ORDER BY created_at
    LIMIT 10
    FOR UPDATE SKIP LOCKED
)
UPDATE jobs
SET status = 'processing',
    claimed_at = now(),
    claimed_by = current_setting('app.worker_id', true)
FROM claim
WHERE jobs.id = claim.id
RETURNING jobs.id, jobs.payload;

How it works:

Worker A: SELECT ... LIMIT 10 FOR UPDATE SKIP LOCKED
          → locks rows 1-10, starts processing

Worker B: SELECT ... LIMIT 10 FOR UPDATE SKIP LOCKED
          → rows 1-10 are locked → SKIPS them
          → claims rows 11-20

Worker C: → claims rows 21-30

No worker blocks on another's claim. Each worker gets a disjoint set of rows, and the atomic UPDATE-with-SELECT ensures a job is claimed exactly once.

sequenceDiagram
    participant W1 as Worker 1
    participant W2 as Worker 2
    participant DB as PostgreSQL

    W1->>DB: CTE: SELECT ... FOR UPDATE SKIP LOCKED
    DB-->>W1: locks rows 1-10
    W1->>DB: UPDATE → processing

    W2->>DB: CTE: SELECT ... FOR UPDATE SKIP LOCKED
    DB-->>W2: rows 1-10 locked → skip → claims 11-20
    W2->>DB: UPDATE → processing

    DB->>DB: no blocking, no lost jobs

Pattern 2: Partial Indexes — Keep the Queue Small

The second critical technique: index only what the queue needs. A partial index on pending rows keeps the hot index tiny:

-- Partial index: only pending jobs are indexed
CREATE INDEX idx_jobs_pending
ON jobs (created_at)
WHERE status = 'pending';

-- Optional: priority-ordered pending index
CREATE INDEX idx_jobs_pending_priority
ON jobs (priority DESC, created_at)
WHERE status = 'pending';

Why it matters:

Approach	Index Size (1M jobs, 5% pending)	Query Cost
Full index on status	~50 MB + bloat from every UPDATE	Scans all statuses, then filters
Partial index on pending	~2.5 MB	Only pending rows — 20× smaller

Because the partial index contains only pending rows, it stays small even as the done history grows. Index maintenance cost (from UPDATEs) drops correspondingly.

Priority Queues

With the priority partial index, claiming becomes:

WITH claim AS (
    SELECT id
    FROM jobs
    WHERE status = 'pending'
    ORDER BY priority DESC, created_at
    LIMIT 10
    FOR UPDATE SKIP LOCKED
)
UPDATE jobs
SET status = 'processing', claimed_at = now()
FROM claim
WHERE jobs.id = claim.id
RETURNING jobs.id, jobs.payload;

High-priority jobs are always claimed first. Note the trade-off: priority ordering under concurrency is approximate (a lower-priority job claimed by an idle worker will run), which is correct for most queue workloads.

Pattern 3: Worker Coordination and Backpressure

Claimed-By Tracking

Record which worker claimed each job:

ALTER TABLE jobs
ADD COLUMN claimed_by TEXT,
ADD COLUMN claimed_at TIMESTAMPTZ;

-- Reclaim jobs whose worker died (no heartbeat)
WITH stale AS (
    SELECT id FROM jobs
    WHERE status = 'processing'
      AND claimed_at < now() - interval '5 minutes'
    FOR UPDATE SKIP LOCKED
)
UPDATE jobs SET status = 'pending', claimed_by = NULL
FROM stale WHERE jobs.id = stale.id;

Heartbeat + Lease

A lease prevents a crashed worker's jobs from hanging forever:

-- Worker heartbeat (every 30s while processing)
UPDATE jobs
SET lease_expires_at = now() + interval '2 minutes'
WHERE id = $1 AND claimed_by = $2
RETURNING id;

graph LR
    A[Worker claims job] --> B{Heartbeat every 30s}
    B -->|OK| C[Lease renewed +2min]
    B -->|Worker crash| D[Lease expires]
    D --> E[Job requeued to pending]
    E --> F[Another worker claims]
    F --> C

    classDef ok fill:#54A24B,stroke:#3a7a35,color:#fff
    classDef fail fill:#E45756,stroke:#b33d3d,color:#fff
    class A,B,C ok
    class D,E,F fail

Backpressure via Limit + Adaptive Polling

Don't claim more than you can process. Use per-worker limits and adaptive polling intervals:

import psycopg2
import time

def worker_loop(conn, batch_size=10, max_poll_interval=5.0):
    poll_interval = 0.05  # start at 50ms
    while True:
        with conn.cursor() as cur:
            cur.execute("""
                WITH claim AS (
                    SELECT id FROM jobs
                    WHERE status = 'pending'
                    ORDER BY priority DESC, created_at
                    LIMIT %s FOR UPDATE SKIP LOCKED
                )
                UPDATE jobs SET status='processing', claimed_at=now()
                FROM claim WHERE jobs.id = claim.id
                RETURNING jobs.id, jobs.payload
            """, (batch_size,))
            claimed = cur.fetchall()
            conn.commit()

        if claimed:
            poll_interval = max(0.05, poll_interval * 0.5)  # speed up
            process_batch(conn, claimed)
        else:
            poll_interval = min(max_poll_interval, poll_interval * 2)  # slow down
            time.sleep(poll_interval)

Exponential backoff on empty polls prevents the thundering herd: with 100 workers all backing off, total polling rate stays bounded.

Performance Numbers

DBOS's benchmarks on a single Postgres instance (16 vCPU, 64 GB, NVMe):

Configuration	Throughput	Notes
Naive poll (status index)	~200 jobs/s	Lock contention, thundering herd
+ SKIP LOCKED	~1.5K jobs/s	Correctness fixed, contention remains
+ Partial index	~8K jobs/s	Small hot index
+ Claim batching (LIMIT 50)	~25K jobs/s	Fewer round-trips
+ Multiple claimers per poll	~40K jobs/s	Single query claims for N workers
+ Connection pooling (pgbouncer)	~50K jobs/s	Sustained ceiling for 1 instance

For comparison: SQS standard queues are optimised for 3K–10K messages/s in a single queue; Redis List-based queues handle 50K–200K operations/s depending on persistence settings. Postgres at 25–50K jobs/s is squarely in production-viable territory for most applications.

Failure Modes to Watch

Failure Mode	Symptom	Prevention
Table bloat	Slow queries after weeks of operation	Autovacuum tuning, partitioned history, pg_squeeze
Deadlock	40P01 errors under heavy concurrency	Consistent claim order, smaller batches
Hot row contention	High lock waits on jobs page	Randomise claim start (OFFSET jitter) or hash partitioning
Clock skew	claimed_at lease bugs in multi-node	Use DB time (now()) not app time
Unbounded retries	Poison messages loop forever	Max-attempt counter + dead letter status

When Postgres Is the Wrong Queue

Postgres queues are not universal:

Scenario	Prefer
1M+ jobs/s throughput	Redis / Kafka / SQS
FIFO with strict ordering guarantees	SQS FIFO / RabbitMQ
Long-lived streams (event sourcing)	Kafka / NATS JetStream
Pub/sub fan-out with consumer groups	Redis Streams / RabbitMQ
Multi-region distributed queues	SQS / Pulsar

The decision rule: if your queue fits in one Postgres instance and you need transactional consistency between queue state and application data, Postgres is not just adequate — it's the best choice. The same database that stores your orders can atomically create the row and enqueue the job. No dual-write problem.

Conclusion

Postgres queues scale — when built with the right patterns. SKIP LOCKED provides the atomicity that makes concurrent workers safe, partial indexes keep the hot path small, and worker coordination (leases, heartbeats, backoff) prevents the operational failures that plague naive implementations. The result: 25–50K jobs/s on a single instance, with transactional consistency that dedicated queue systems cannot offer.

The biggest insight is architectural: keeping the queue in the same database as the data eliminates the dual-write problem — the classic failure mode of "write to DB, then publish to queue, and crash between the two." For transactional workloads, that consistency guarantee outweighs the throughput headroom of dedicated queue infrastructure.

Source: DBOS — Making Postgres queues scale (July 2026).