Engineering
Why pgvector HNSW Index Builds Crash PostgreSQL Workers (And the Fix)

Your background batch job dumps 500,000 document embeddings into PostgreSQL, triggers CREATE INDEX ... USING hnsw, and forty minutes later your primary database drops dead. The Linux kernel out-of-memory killer just terminated the backend worker process with signal 9. Teams moving off standalone vector stores hit this wall constantly: building Hierarchical Navigable Small World graphs inside PostgreSQL is memory-violent, and default Postgres memory allocations are completely unprepared for it.
The maintenance_work_mem Trap and Unindexed Graph Spills
Postgres builds standard B-Tree and GIN indexes by sorting on disk when memory runs short. HNSW does not work like that. Constructing an HNSW graph requires traversing and updating multi-layer neighbor graphs in memory for every single 1536-dimensional vector. If your maintenance_work_mem is left at the stock 64MB, Postgres cannot hold the target graph in RAM. The index build spills partial state, thrashing disk I/O at 4,000 IOPS while stalling every concurrent write query on your application tables.
If you swing the needle too far and bump maintenance_work_mem to 16GB on an RDS or Aurora instance, parallel index workers (max_parallel_maintenance_workers = 4) will each attempt to claim that entire memory block. Four workers trying to allocate 16GB each on a 32GB RAM instance triggers an instant OOM panic. Your cluster restarts into recovery mode, dropping active pool connections across your API routes.
Sizing Graph Parameters Without Starving Parallel Queries
The fix requires separating vector ingestion from live index construction. Never run CREATE INDEX on raw tables while streaming bulk chunks. Ingest vectors with the index dropped or disabled, then build the index using a bounded single-worker session: SET LOCAL max_parallel_maintenance_workers = 0; paired with an explicitly computed maintenance_work_mem equal to (dimensions * 4 bytes + 64 bytes) * row_count * 1.25. For 500k OpenAI text-embedding-3-small vectors (1536 dims), graph construction needs approximately 4.2GB of dedicated RAM.
Dial down your build parameters for cold storage. Setting m = 16 and ef_construction = 64 cuts graph construction memory by more than half compared to aggressive defaults (m = 32, ef_construction = 128), with less than a 1.8% drop in 10-nearest-neighbor recall on production semantic search workloads.
Lean Vector Architectures in Production
Running hybrid search on PostgreSQL saves thousands of dollars in vector database licensing, but it demands strict operational guardrails. When systems scale past millions of chunks, you need decoupled ingestion pipelines that isolate vector graph indexing from customer-facing transactions. At Enorexa, we engineer lean database architectures and high-throughput AI pipelines that stay performant under real-world load. If your AI backend is crashing under heavy vector indexing or scaling strain, talk to us to audit your production pipeline.
Related Engineering & AI Articles
AI
LLM Fine-Tuning vs. RAG: When to Pay for Training, When to Prompt
Choosing between LLM fine-tuning and Retrieval Augmented Generation (RAG) is a critical cost and accuracy decision. Most teams default to RAG, but that's not always the cheapest or most performant path for specialized tasks.
November 16, 2026
Engineering
Why Serverless WebSockets Drop AI Agent Connections (And How to Fix It)
Serverless functions like Next.js API routes promise infinite scale, but their ephemeral nature and connection limits silently kill long-lived WebSocket connections crucial for real-time AI agents. We've seen this manifest as dropped user sessions and incomplete agent responses.
November 15, 2026
Engineering
Why Next.js API Routes Still Cold Start (Even on the Edge)
Most teams building on Next.js assume Edge Functions solve cold starts. They don't, not entirely. We've seen critical API routes still hit 500ms+ latency spikes, even when deployed globally.
November 14, 2026