EnorexaEnorexa
Back to blog

Engineering

Kafka to ClickHouse Ingestion: Stopping Consumer Group Rebalance Storms

November 9, 20263 min readBy Enorexa Team

Most teams building real-time analytics pipelines hit an ugly cliff around 85,000 events per second. Your application pipes event telemetry through Apache Kafka, a worker pool batches those records, and an analytical columnar store like ClickHouse receives the inserts. Then traffic spikes on Monday morning. Suddenly, ClickHouse takes 12 seconds to commit a block due to heavy merge pressure, your worker blocks inside its flush loop, Kafka assumes the consumer died, and a full consumer group rebalance kicks off. Your lag doesn't just grow; it compounds exponentially.

Why Long ClickHouse Flushes Trigger Consumer Heartbeat Timeouts

ClickHouse requires wide, chunky batch inserts. If you push ten thousand single-row inserts per second, you quickly hit TOO_MANY_PARTS exceptions as background parts mergers choke on disk IOPS. So engineers do the natural thing: they buffer records in worker memory and write chunks of 50,000 rows. But here is the architectural flaw. Standard Kafka client loops (confluent-kafka-go or kafkajs) expect poll() or the main loop to yield within max.poll.interval.ms (commonly defaulted to 300,000ms, but often lowered to 30,000ms by aggressive tuning). When ClickHouse throttles synchronous writes because active parts in a partition exceed 300, your worker's write loop stalls. The Kafka coordinator marks the consumer dead, triggers a rebalance across all consumer instances, revokes partition ownership, and forces the entire consumer group to stop processing while reassignment settles.

The Uncommitted Offset Death Spiral

The disaster deepens if you commit offsets only after a successful ClickHouse write. While Worker A is waiting on an oversized table insert, the coordinator kicks it out and hands Partition 4 to Worker B. Worker B immediately seeks to the last committed offset, re-fetching 40,000 records that Worker A was already in the middle of writing. Once Worker A finally completes its delayed write or fails with a timeout, Worker B writes those exact duplicate records again. You now have double-ingestion, redundant merge tree deduplication costs, and network saturation from back-to-back rebalances that happen every few minutes under heavy operational load.

Decoupling Ingestion Loops with In-Memory Ring Buffers

The fix is separating Kafka consumer polling from analytical flush threads. Never execute a synchronous network write to ClickHouse inside your Kafka polling loop. Instead, let your Kafka consumer thread write raw message references directly into a bounded, fixed-capacity in-memory ring buffer, updating internal fetch positions while background worker threads drain the buffer into batch arrays. Set session.timeout.ms to 45,000ms and keep heartbeat.interval.ms at 3,000ms in a dedicated background goroutine or worker thread. If ClickHouse slows down and backpressure fills the memory buffer to 85% capacity, pause the consumer explicitly via consumer.pause([topicPartitions]) rather than blocking the poll thread. Kafka keeps receiving heartbeats, knows the worker is healthy, and avoids triggering an expensive cluster-wide rebalance.

What to Audit in Your Ingestion Pipeline Today

Check your consumer logs for CommitFailedException or messages indicating that offset commits failed because the group had already rebalanced. If you are running the native ClickHouse Kafka Engine, verify whether your kafka_num_consumers matches topic partitions cleanly without running into lockouts. When engineering lean real-time streaming architectures, optimizing buffering boundaries saves tens of thousands in over-provisioned cluster costs. At Enorexa, we build high-throughput telemetry pipelines, operational data engines, and resilient cloud architectures that do not fall over during peak load. If your ingestion pipeline is choking on rebalances or high-cardinality lag, talk to us and let's stabilize your data flow.