-
Notifications
You must be signed in to change notification settings - Fork 117
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize Kafka cluster connections #3397
Comments
For monitoring and viewing tcp connections for manual testing we could try to use https://www.inspektor-gadget.io/docs/v0.20.0/gadgets/top/tcp/ |
edit: I realized it's more complicated than a simple pool as different connections may need different bootstrap servers and/or config options. I'm going to test:
/assign |
Let's keep in mind that connection pools and caches are very prone to security issues, so let's have a design that reduces the risk of security issues without over-optimizing too early |
Do you have some other ideas? My initial really simple idea was to just create one admin client per Reconciler struct that needs one, but I wasn't sure if that would make too many connections overall, and also different brokers may have different bootstrap servers/security combinations which would mean the same client would not work so I thought of the pooling. WDYT @pierDipi ? |
This issue is stale because it has been open for 90 days with no |
/triage accepted |
Problem
In our controllers, we often re-create Kafka cluster connections, the original reason was this issue IBM/sarama#2060 . Now, it's fixed we need to re-review our Kafka cluster interactions in the control plane (for all resources) and see if we can optimize them by reducing the number of TCP connections created.
Persona:
Which persona is this feature for?
*
Exit Criteria
Lower number of Kafka connections for a single resource reconciliation
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
7
Additional context (optional)
/kind performance
The text was updated successfully, but these errors were encountered: