Kafka Source Node
Quick Reference
Kafka Broker
The address of your Kafka broker.
ex: localhost:9092
Kafka Topic
The name of the Kafka topic to consume from.
ex: my-topic
Consumer Group ID
A unique identifier for the consumer group.
ex: fleak-consumer-group
Encoding Type The format of the data in the Kafka topic.
Additional Properties
Extra Kafka configuration options as key–value pairs.
ex: security.protocol=SASL_SSL
Overview
The Kafka Source node allows your workflow to ingest real-time streaming data from an Apache Kafka cluster. It acts as a starting point for your data pipeline, continuously polling a specified topic for new events.
Configuration
| Field | Description | Required | Example |
|---|---|---|---|
| Kafka Broker | A comma-separated list of host/port pairs for establishing the initial connection to the Kafka cluster (Bootstrap Servers). | Yes | localhost:9092 |
| Topic | The name of the Kafka topic to consume data from. | Yes | user_signups |
| Consumer Group ID | A unique string that identifies the consumer group. This ID allows Kafka to track which messages have already been processed (offsets). Changing this ID will reset the processing position (usually to the earliest available message). | Yes | fleak-production-consumer |
| Encoding Type | Defines how the raw bytes from Kafka should be deserialized into Fleak events. More info about encoding type can be found here Map<String, String> | Yes | JSON Object |
| Additional Properties | Key-value pairs for advanced Kafka Consumer configuration (see below). | No | None |
Advanced Configuration Properties
The Additional Properties section allows you to pass standard Kafka client configuration properties directly to the underlying consumer. This is commonly used for configuring security protocols, authentication credentials, or tuning client performance.
Note: Any property supported by the standard Apache Kafka ConsumerConfig class can be entered here.
Click + Add to insert specific key-value pairs.
Common Use Cases for Additional Properties
- Security & Authentication: If your cluster requires authentication (e.g., SASL/SSL), you can provide the necessary JAAS config, security protocols, and mechanism settings here.
- Connection Tuning: You can override default timeouts or connection settings (e.g.,
request.timeout.ms,session.timeout.ms) if your network environment requires it.
Related Nodes
- kafkaSink: Send processed data back to Kafka