Why Next.js LLM Streams Freeze in Production (And How to Fix Proxy Buffering)

Every engineering team building an AI product hits the exact same wall the week they deploy to production. On localhost:3000, your LLM response streams token-by-token at 60 frames per second. Then you push to AWS behind an Application Load Balancer or a containerized Nginx reverse proxy. Suddenly, users stare at a frozen spinner for 24 seconds until 1,200 words dump onto the screen in a single violent repaint—or worse, the connection dies mid-sentence with TypeError: network error and an HTTP 504 Gateway Timeout.

Why Next.js Server Actions Sabotage Token Streaming

The first mistake we audit in broken Next.js codebases is routing real-time LLM token streams through Server Actions ('use server'). Developers love Server Actions because they eliminate boilerplate API wiring. That convenience is a trap for long-lived streams. Under the hood, Server Actions are built around React Flight payloads and POST-based RPC mutations. They are designed to resolve state transitions, not hold open bidirectional or long-lived chunked pipes. Worse, Next.js queues sequential Server Action invocations per client session; while a 45-second reasoning model is streaming through an action, every other mutation in that tab hangs. Drop Server Actions for token delivery. Use standard App Router Route Handlers (app/api/stream/route.ts) returning a raw Response wrapped around a Web ReadableStream, and fire the async token iterator inside an immediately invoked function expression (IIFE) inside start(controller) so the HTTP headers flush in under 15 milliseconds.

The 64KB Reverse Proxy Buffer Trap

Moving to a Route Handler only solves half the problem if your infrastructure silently hoards bytes. By default, Nginx, Cloudflare, and cloud ingress controllers optimize for standard HTML and JSON payloads by buffering upstream responses until a memory threshold—typically 4KB to 64KB—fills up. A single LLM token is roughly 4 bytes. At 50 tokens per second, your stream generates just 200 bytes per second. That means an Nginx proxy with proxy_buffers 8 16k will sit on your LLM output for over 20 seconds before flushing a single byte to the browser. You have to explicitly force every hop in the network path to flush immediately. In your Next.js response headers, set 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache, no-transform' (the no-transform directive stops CDNs from applying gzip compression that re-buffers chunks), and 'X-Accel-Buffering': 'no'. If you run custom Nginx containers on ECS or Kubernetes, hardcode proxy_buffering off; and chunked_transfer_encoding on; directly on your streaming location blocks.

Surviving 30-Second Reasoning Pauses Without 504 Timeouts

In 2026, reasoning models routinely spend 20 to 40 seconds generating hidden chain-of-thought tokens before emitting the first visible character. To an AWS Application Load Balancer or corporate firewall with a default 30-second idle timeout, a silent TCP socket looks like a dead upstream server. The load balancer severs the connection, and your client parser crashes with SyntaxError: Unexpected end of JSON input. Never leave an open SSE stream silent. Inside your Route Handler's ReadableStream, attach a setInterval heartbeat that enqueues an SSE comment frame (controller.enqueue(encoder.encode(': heartbeat\n\n'))) every 12 seconds. Per the official SSE specification, browsers and compliant fetch parsers ignore lines starting with a colon, but the underlying TCP packets reset the idle timer on every load balancer and NAT gateway between your container and the user's laptop.

Decoupling Client State From Ephemeral HTTP Pipes

Even with zero buffering and 12-second heartbeats, mobile Safari will ruthlessly kill your fetch stream the second a user switches apps or drives into an elevator. If your LLM generation lives entirely inside the memory of an ephemeral HTTP request handler, that user just lost both their answer and the $0.08 you paid in inference tokens. For any generation taking longer than 10 seconds, decouple inference from the HTTP socket. Push the prompt job to a worker writing tokens into a Redis Stream (XADD) with a 15-minute TTL, and have your Next.js SSE endpoint read from that stream via XREAD BLOCK using a cursor ID (Last-Event-ID). When a client drops at token 340 and reconnects three seconds later, it passes lastTokenIndex: 340 and replays the missed chunks in 8 milliseconds without re-billing the model.

Auditing Your Streaming Architecture Before Launch

You can test if your production pipeline is actually streaming in ten seconds: open Chrome DevTools, throttle the network to Fast 3G, inspect the EventStream tab on your API call, and check if individual frames arrive under 100ms apart. If they arrive in multi-kilobyte bursts, your proxy or runtime is still buffering. If your team is fighting dropped LLM connections, runaway serverless duration bills, or fragile Next.js streaming state, explore how Enorexa architects resilient full-stack AI systems—or talk to us to audit your production pipeline before your next launch.