kafka-controller Kafka connections pooling

[maschmid]

Problem

Currently, a new kafka client and an admin client is created for every reconciliation. While creating a larger deployment, this leads to hundreds of short-lived TCP connections to Kafka to be created.

We should consider using some kind of a connection pool for the kafka-controller Kafka connections.

Having a configurable connection pool would also help System Operators, as they could properly account for the number of connections from the kafka-controller when determining a proper value for Kafka max.connections setting. Currently, the number of connections is unbounded.

Persona:
Which persona is this feature for?
System Operator

Exit Criteria
kafka-controller has configurable Kafka connection pools, either global or per "bootstrap.servers".

Time Estimate (optional):
How many developer-days do you think this may take to resolve?

Additional context (optional)
there are open issues in sarama IBM/sarama#2042 , that might be needed to be either fixed first, or a workaround might be needed.