This repository provides a simple example of a Kafka producer and consumer implemented in Go, using the Sarama library. The producer sends messages to a specified Kafka topic, and the consumer subscribes to the same topic to receive and log the messages.
- Go installed on your machine.
- Apache Kafka running locally or on a reachable server.
- Sarama library (
github.com/IBM/sarama
) installed. You can install it using the following command:
go get github.com/IBM/sarama
Both the producer and consumer rely on configuration files in YAML format. The YAML files include essential Kafka configurations such as broker list. This can be extended to add other configurations like message encoding, ssl and other kafka configurations
brokerList: "localhost:9092"
# SSL Configuration
ssl:
enabled: true
caFile: "path/to/ca.pem"
certFile: "path/to/client.pem"
keyFile: "path/to/client.key"
- brokerList: The comma-separated list of Kafka broker addresses.
- ssl.enabled: Set to true to enable SSL.
- ssl.caFile: Path to the Certificate Authority (CA) file.
- ssl.certFile: Path to the client certificate file.
- ssl.keyFile: Path to the client private key file.
brokerList: "localhost:9092"
# SSL Configuration
ssl:
enabled: true
caFile: "path/to/ca.pem"
certFile: "path/to/client.pem"
keyFile: "path/to/client.key"
# Consumer Group Configuration
consumerGroup: "your_consumer_group"
- brokerList: The comma-separated list of Kafka broker addresses.
- ssl.enabled: Set to true to enable SSL.
- ssl.caFile: Path to the Certificate Authority (CA) file.
- ssl.certFile: Path to the client certificate file.
- ssl.keyFile: Path to the client private key file.
- consumerGroup: Consumer Group name to commit offsets.
Best part about golang is that the build can be easily done for required OS and Arch
-
Build the Kafka Producer
GOOS={OS_NAME} GOARCH={OS_ARCH} go build -o producer.exe producer.go
-
Build the Kafka Consumer:
GOOS={OS_NAME} GOARCH={OS_ARCH} go build -o consumer.exe consumer.go
-
Build Parameters
Parameter Linux Windows GOOS linux windows GOARCH amd64 amd64
To run the Kafka producer, use the following command:
./producer -topic topic_name -message "Your message here"
- -topic: The topic name to sent message to
- -message: The message to be sent to the Kafka topic.
To run the Kafka consumer, use the following command:
./consumer -topic topic_name
- -topic: The topic name to sent message to
To get help run
./producer -help
Usage of producer:
-message string
Message to produce to Kafka (default "Hello, Kafka!")
-topic string
Kafka topic to produce to
./consumer -help
Usage of consumer:
-topic string
Kafka topic to subscribe to