EnorexaEnorexa
Back to blog

Engineering

Why pgvector HNSW Index Builds Crash PostgreSQL Workers (And the Fix)

November 12, 20263 min readBy Enorexa Team

Your background batch job dumps 500,000 document embeddings into PostgreSQL, triggers CREATE INDEX ... USING hnsw, and forty minutes later your primary database drops dead. The Linux kernel out-of-memory killer just terminated the backend worker process with signal 9. Teams moving off standalone vector stores hit this wall constantly: building Hierarchical Navigable Small World graphs inside PostgreSQL is memory-violent, and default Postgres memory allocations are completely unprepared for it.

The maintenance_work_mem Trap and Unindexed Graph Spills

Postgres builds standard B-Tree and GIN indexes by sorting on disk when memory runs short. HNSW does not work like that. Constructing an HNSW graph requires traversing and updating multi-layer neighbor graphs in memory for every single 1536-dimensional vector. If your maintenance_work_mem is left at the stock 64MB, Postgres cannot hold the target graph in RAM. The index build spills partial state, thrashing disk I/O at 4,000 IOPS while stalling every concurrent write query on your application tables.

If you swing the needle too far and bump maintenance_work_mem to 16GB on an RDS or Aurora instance, parallel index workers (max_parallel_maintenance_workers = 4) will each attempt to claim that entire memory block. Four workers trying to allocate 16GB each on a 32GB RAM instance triggers an instant OOM panic. Your cluster restarts into recovery mode, dropping active pool connections across your API routes.

Sizing Graph Parameters Without Starving Parallel Queries

The fix requires separating vector ingestion from live index construction. Never run CREATE INDEX on raw tables while streaming bulk chunks. Ingest vectors with the index dropped or disabled, then build the index using a bounded single-worker session: SET LOCAL max_parallel_maintenance_workers = 0; paired with an explicitly computed maintenance_work_mem equal to (dimensions * 4 bytes + 64 bytes) * row_count * 1.25. For 500k OpenAI text-embedding-3-small vectors (1536 dims), graph construction needs approximately 4.2GB of dedicated RAM.

Dial down your build parameters for cold storage. Setting m = 16 and ef_construction = 64 cuts graph construction memory by more than half compared to aggressive defaults (m = 32, ef_construction = 128), with less than a 1.8% drop in 10-nearest-neighbor recall on production semantic search workloads.

Lean Vector Architectures in Production

Running hybrid search on PostgreSQL saves thousands of dollars in vector database licensing, but it demands strict operational guardrails. When systems scale past millions of chunks, you need decoupled ingestion pipelines that isolate vector graph indexing from customer-facing transactions. At Enorexa, we engineer lean database architectures and high-throughput AI pipelines that stay performant under real-world load. If your AI backend is crashing under heavy vector indexing or scaling strain, talk to us to audit your production pipeline.