CI/CD for Production LLMs: Stopping Silent Prompt Regressions

Last month, a three-word edit to a system prompt broke 1,400 automated vendor invoice syncs in forty minutes. A product manager had tweaked an extraction prompt to make the assistant sound more polite when flagging missing tax IDs. tsc --noEmit passed. All 180 Jest unit tests stayed green. GitHub Actions deployed the container to production in four minutes flat. Then the model started wrapping its JSON payload inside conversational markdown fences whenever a PDF lacked a VAT number, throwing SyntaxError: Unexpected token 'H' at position 0 across our ingestion workers.

Most engineering teams treat prompts like configuration strings instead of executable code. They test changes by pasting three sample inputs into a playground, squinting at the output, and merging to main. We call this deployment by vibe check. It works when you have ten daily users. It collapses the moment an upstream model provider ships a silent weights revision or an operations lead tweaks instructions for a single edge case and inadvertently tanks tool-call routing across the rest of your pipeline.

Why Green Unit Tests Lie About AI Workflows

Standard CI pipelines validate deterministic contracts. If you mock the OpenAI or Anthropic SDK response in your Jest suite, you are only testing whether your TypeScript wrapper can parse a hardcoded fixture you wrote yourself six months ago. You aren't testing the prompt at all.

When we audit broken production AI pipelines, the root cause is almost never a broken API client. It is behavioral drift. Changing a RAG chunk window from 512 to 256 tokens alters context density enough that an agent stops calling lookup_vendor_ledger before issuing create_draft_bill. Or a prompt tweak shifts numeric formatting from raw floats (1450.00) to currency strings ("$1,450.00"), triggering ZodError: invalid_type at "line_items[0].unit_price": Expected number, received string downstream. Fixing this requires treating prompt modifications and retrieval parameters with the exact same gating rigor as database migrations.

The Three-Tier Evaluation Gate We Run in GitHub Actions

You cannot run a 500-row LLM-as-a-Judge benchmark on every git push. It takes twelve minutes, burns $4.50 per commit in inference tokens, and frustrates developers until they bypass the check entirely. Instead, we split our evaluation pipeline into three distinct latency and cost tiers inside CI.

• Tier 1: Deterministic Schema & AST Gates (< 8 seconds, $0.00). Triggered on pre-commit and every push. We lint prompt templates for unescaped Mustache/Jinja variables, validate that Structured Output JSON schemas compile without unsupported recursive $ref definitions, and run regex assertions against cached replay responses to ensure banned phrases or raw system instruction leaks never slip through.
• Tier 2: Recorded Tool-Call & Routing Smoke Suite (< 75 seconds, ~$0.06 per PR). Triggered only when files inside /prompts/**, /agents/**, or /rag/** change in a pull request. We run 35 curated, high-entropy test cases against a pinned low-latency model with temperature: 0. This gate does not care about prose style; it asserts strictly on tool selection accuracy, argument types, and citation IDs. If precision on tool routing drops below 97%, the PR merge button locks.
• Tier 3: Production Failure Replay & Rubric Grading (Merge Queue & Nightly, ~$1.20). Before a release candidate cuts to production, we run a 250-sample Golden Dataset graded by a small, pinned rubric judge model evaluating faithfulness against retrieved context chunks and zero-hallucination constraints on financial or compliance fields.

Building a Golden Dataset From Real Production Scars

Synthetic test datasets generated by asking an LLM to "invent 100 tricky user queries" are mostly useless. Real users break systems in ways synthetic generators never imagine: rotated multi-page customs forms, half-Hindi half-English WhatsApp voice transcripts, or CSV uploads with UTF-16 BOM headers.

Every evaluation suite we build for our own platforms—including the candidate parsing engine in Enorexa Recruiter and the schema validators inside Enorexa Tools—is seeded exclusively from production scars. When a live trace triggers an HTTP 422 Unprocessable Entity, times out after 15,000ms, or gets flagged by an operator clicking a thumbs-down button in the UI, an asynchronous worker captures the exact input payload, the retrieved vector chunk IDs, and the git SHA of the active prompt. With one click, an engineer promotes that failed span into /evals/golden_failures.jsonl alongside the expected ground-truth JSON output. Your regression suite gets smarter every time production breaks.

Content-Addressed Prompts and Canary Rollbacks

Offline evals catch 90% of regressions, but live traffic always finds the remaining 10%. Never hardcode system prompts inside application environment variables where changing a sentence requires a full container rebuild.

We store prompts as versioned, content-addressed artifacts hashed by SHA-256 (prompt_sha: 8f3a9c1) alongside model parameters (model, temperature, top_p, schema_version). When deploying a new prompt revision, our router sends 5% of live traffic to the canary hash while logging OpenTelemetry spans to PostgreSQL. If the canary's schema validation failure rate or human-escalation rate exceeds the baseline hash by more than 1.5% over 200 requests, a background watchdog flips the routing pointer back to the previous SHA in under 50 milliseconds. Zero downtime. Zero paged engineers at 2:00 AM.

Stop Shipping Prompt Changes Blind

Audit your repository today. If an engineer on your team can edit a system prompt, change a chunking strategy, or swap an embedding model and merge to production without triggering an automated accuracy assertion against real historical failures, you don't have an AI pipeline—you have a demo waiting to break.

Start small this week: pull the last twenty failed payloads from your production logs, drop them into a versioned .jsonl file, and wire a 60-second GitHub Action that fails any pull request breaking your Zod or Pydantic output contracts. If your team is scaling custom LLM workflows, document automation, or full-stack software and wants battle-tested CI/CD guardrails wired into your stack, explore what we build at Enorexa or talk to us to review your architecture.