The bottleneck was never the broker

For about a year I worked on a telemetry pipeline that ingests events from connected vehicles. Hundreds of thousands of events a second on a normal day, more when a fleet wakes up in the morning. When I joined, the system was already "on Kafka", which everyone said the way you might say a house is "on a good foundation". And yet it fell over regularly, and every postmortem ended with someone proposing more partitions.

I want to write down what actually fixed it, because I keep meeting teams with the same symptoms, and because the fix had almost nothing to do with Kafka.

The shape of the problem

Each vehicle sends a stream of small events: location, speed, a few sensor readings, the occasional "something interesting happened" marker. The consumer's job was to stitch those into trips and flag the interesting bits for people to look at later.

Trip stitching needs state. To update a trip you need the previous event for that vehicle, so the consumer kept a per-vehicle record, and to keep it consistent it took a lock on the vehicle before touching the record. Simple. Correct. Slow in a way that was hard to see.

The lock itself was cheap. The cost came from what happened around it: a database read to load the trip, some logic, a write, release. Under normal load that took a few milliseconds and nobody noticed. Under a burst from one busy fleet, thousands of events for the same few hundred vehicles arrived within a second, and they all queued behind each other on those locks. The consumer's threads were mostly waiting. Lag climbed. Someone got paged.

Here is the part that took me embarrassingly long to accept: adding consumers did nothing. Kafka gives each partition to one consumer, and all events for a vehicle live on one partition, so they were already serialised before our lock ever ran. We were paying for ordering twice, once in the broker and once in our own code, and getting the throughput of neither.

What changed

The first change was to stop fighting the partition. Instead of locking per vehicle inside the consumer, we let the partition be the lock. A single consumer thread owns a partition, so events for a vehicle already arrive in order and one at a time. The lock was redundant. Deleting it was uncomfortable, because the lock felt like the thing keeping us safe, but the broker had been keeping us safe the whole time.

That alone bought maybe a third. The bigger win was moving the trip state out of the database and next to the consumer. Each consumer keeps the open trips for its partitions in memory, and writes them out on a timer or when a trip closes, not on every event. If the consumer dies, it replays from the last committed offset and rebuilds. This is the boring, well known "local state plus changelog" pattern, and it works. We went from a database round trip per event to a batch write every few seconds.

The last change was the one I would put first if I did it again. We measured what was hot. Not "Kafka lag", which was the only dashboard anyone looked at, but events per key per second. It turned out that a handful of vehicles produced ten times the traffic of the rest, because of a firmware bug that sent the same reading in a loop. No amount of architecture fixes a firmware bug. A rate limit per device and a bug report did more for stability than everything above.

After all three, the pipeline held at over 500K events per second in testing, with the same broker, the same partition count, and fewer consumer pods than before.

Things I believe now

Partitions are not a throughput knob. They are an ordering decision with a throughput side effect. If your load is skewed by key, and it almost always is, more partitions just spread the quiet keys thinner while the hot key still lands in one place.

Backpressure is something you design, not something the broker does for you. Kafka will happily buffer a week of events. That is not backpressure, that is a very patient queue. The question is what your consumer does when it cannot keep up: shed, sample, slow the producer, or scale. Pick one on purpose. We picked "slow the producer" for the firmware loop and "scale" for real bursts, and the difference between those two cases was invisible until we measured per-key rates.

Locks in a partitioned consumer are usually a smell. If you find yourself locking on the same key the topic is partitioned by, ask what the lock is protecting that the partition does not already protect. Sometimes there is a good answer. Often there isn't.

A footnote from this year

Kafka 4.2 shipped share groups, which give you queue semantics on top of a topic: several consumers can read the same partition with per-record acknowledgement. I was curious whether that would have saved us. It would not have. Share groups remove the "one consumer per partition" cap, which is great when your work is independent and just slow. Our work was not independent. Two consumers pulling events for the same vehicle would have needed the lock back, plus a way to agree on ordering, and we would have been right where we started with a newer version number.

That is the thing I most want to say to teams staring at consumer lag. Before you reach for a new feature or a bigger cluster, find out whether your slowness is "too much work" or "work that cannot run in parallel". They look identical on a lag graph. They have completely different fixes.

If you are dealing with something like this and want a second pair of eyes, that is roughly what I do for a living. Otherwise, I hope the vehicle with the chatty firmware has been patched by now.