Why Request Queues Hit 429s: Token-Aware Backpressure for Production LLMs

Set limiter: { max: 50, duration: 60000 } in BullMQ, deploy ten background worker pods, and watch your production queue blow up at 2:00 PM anyway. Upstream LLM providers start throwing HTTP 429 Too Many Requests errors even though your dashboard swears you are running at 8% of your Requests Per Minute (RPM) ceiling. Most teams get this wrong because they treat LLM inference like a standard REST API. It isn't. A single webhook job carrying a 64-page vendor contract burns 78,000 input tokens in one shot, while the next job is a 400-token classification ping. Request-based rate limiters are completely blind to payload weight.

Why Fixed Worker Concurrency Fails the TPM Ceiling

Every commercial model endpoint enforces two separate ceilings simultaneously: Requests Per Minute (RPM) and Tokens Per Minute (TPM). When you configure standard BullMQ worker concurrency or fixed-window job limiters, you only gate the number of open HTTP sockets. That works fine when every database write or Stripe API call costs roughly the same compute.

Document extraction and multi-step agent pipelines break that symmetry immediately. Suppose your workspace sits on an 800,000 TPM tier for claude-3-7-sonnet or gpt-4o. Ten concurrent workers each pull a heavy multi-document extraction job averaging 85,000 prompt tokens. Within 1.4 seconds, your cluster dispatches 850,000 tokens. Request #10 fails instantly with rate_limit_error. Worse, if your workers use naive exponential backoff without pausing the upstream queue, the other nine workers grab new jobs and hammer the already exhausted token window. You spend the next four minutes burning CPU cycles on dead retries while real-time user requests time out.

Pre-Flight Token Estimation and Atomic Redis Lua Buckets

Fixing this requires shifting your queue from job-count gating to pre-flight token accounting. Before a BullMQ worker opens an HTTPS connection to OpenAI, Anthropic, or Vertex AI, it must reserve the estimated token cost from a shared Redis token bucket. Do not run a full BPE tokenizer inside the hot worker loop if you are processing massive text blobs in Node.js; synchronous tiktoken calls on 2MB strings will block your event loop for 40+ milliseconds. Instead, compute the token count once during job ingestion (or use a fast byteLength / 3.6 heuristic with a 10% safety buffer for non-ASCII text) and attach estimatedInputTokens directly to the BullMQ job payload.

Inside Redis, maintain two leaky buckets per provider model tier using a single atomic Lua script executed via EVALSHA. One key tracks rpm_available, and the second tracks tpm_available. When a worker picks up a job, the Lua script calculates the continuous token refill based on redis.call('TIME') elapsed since the last check, verifies whether tpm_available >= estimatedInputTokens + reservedOutputTokens, and decrements both counters atomically. If the bucket is dry, the script returns the exact millisecond wait time required for the bucket to refill. The worker calls job.moveToDelayed(Date.now() + waitMs, token) and yields immediately without throwing a failed attempt.

The `max_tokens` Reservation Trap and Post-Flight True-Ups

Here is what actually breaks once you deploy a naive token bucket in production: over-reservation starvation.Upstream providers often count your max_tokens parameter against your TPM limit the moment the request lands, or your own bucket reserves estimatedInputTokens + max_tokens to prevent mid-stream cutoff. If your engineers lazily set max_tokens: 16384 on every structured JSON extraction call—even when the actual JSON response averages 650 tokens—your Redis bucket thinks each job consumed 16,384 output tokens. Your effective throughput drops by 80%.

You have to implement a post-flight true-up phase. The moment the LLM call resolves, inspect usage.input_tokens and usage.output_tokens from the response payload, along with the provider's response headers (anthropic-ratelimit-tokens-remaining or x-ratelimit-remaining-tokens). Run a second lightweight Redis command that credits back the unused reservation delta: reservedTotal - actualTotal (clamped to the bucket ceiling). Even better, overwrite your local Redis bucket counter whenever the upstream x-ratelimit-remaining-tokens header reports a lower number than your local state. That keeps your cluster synchronized even if a separate staging service or CLI script shares the same API key.

Isolating Runaway Tenants Before They Starve Real-Time Traffic

Global token buckets protect your API key, but they won't stop one enterprise customer from wrecking your SLA for everyone else. We learned this building high-volume resume and document parsing pipelines for Enorexa Recruiter. When a staffing agency uploads a batch of 400 PDFs at 9:05 AM, a single FIFO queue will starve interactive users waiting on single-record parses.

Split your ingestion into two distinct lanes with hierarchical Redis quotas: an interactive priority queue that holds a guaranteed 30% TPM reserve, and a bulk background queue capped at 70% of your global TPM ceiling. Additionally, enforce a per-tenant sliding window key (tpm:tenant:{orgId}) so a single customer bulk upload pauses its own jobs once it hits 150,000 TPM, leaving the rest of the worker pool free to process other tenants. If a job can wait 20 minutes—like nightly CRM enrichment or historical audit logs—skip the real-time queue altogether and route it to the provider's asynchronous Batch API at a 50% cost discount with separate rate pools.

What to Audit in Your Worker Cluster Today

Pull your APM logs from the last seven days and filter for HTTP 429 responses and BullMQ stalled job events. If your workers retry 429 errors using fixed 2-second delays instead of parsing the Retry-After or x-ratelimit-reset-tokens header, you are amplifying your own outages. Start by logging the ratio between reserved max_tokens and actual completion_tokens across your top three prompts—tightening max_tokens bounds alone frequently recovers 35% of wasted TPM capacity in a single afternoon. If your operations or engineering team is fighting brittle AI queues, noisy webhook retries, or runaway inference bills, explore how we architect resilient production systems at Enorexa or talk to us to review your pipeline.