kafka-node-memory-issue

This repo has a sample code that reproduces a memory issue with kafka-node.

To reproduce

Create the topic (assuming you have a local kafka. If not, change the script accordingly):

./create-topic.sh

This creates a topic with 100 partitions.

Next, run the producer, to populate the topic with a lot of data:

node producer.js

Now run the consumer, which should consume more and more memory, and should probably eventually die

node consumer.js

Analysis

I detected 2 separate but related issues:

1. The response we are getting is pretty big. Probably several hundred megs after decompression. The reason is that the maxFetchSize param is per partition, and since we have many partitions, the overall response is big. This was addressed by kafka but kafka-node does not seem to support it yet.
2. When messages are compressed, the 'done' event is emitted before any 'message' event, because decompressing is async. This means we are sending another fetch request before we ever get a chance to pause the consumer. This causes memory to grow, because we keep getting big fetch responses and we do not pause it fast enough.

There could be other issues, but these are the ones I observed trying to debug it.