Kafka Source Node
Quick Reference
| Name | Description |
|---|---|
| Kafka Broker | The address of your Kafka broker ex: localhost:9092 |
| Kafka Topic | The name of the Kafka topic to consume from ex: my-topic |
| Consumer Group ID | A unique identifier for the consumer group ex: fleak-consumer-group |
| Encoding Type | The format of the data in the Kafka topic |
| Additional Properties | Extra Kafka configuration options as key–value pairs ex: security.protocol=SASL_SSL |
Overview
The Kafka Source node allows your workflow to ingest real-time streaming data from an Apache Kafka cluster. It acts as a starting point for your data pipeline, continuously polling a specified topic for new events.
Configuration
| Field | Description | Required | Example |
|---|---|---|---|
| Kafka Broker | A comma-separated list of host/port pairs for establishing the initial connection to the Kafka cluster (Bootstrap Servers). | Yes | localhost:9092 |
| Topic | The name of the Kafka topic to consume data from. | Yes | user_signups |
| Consumer Group ID | A unique string that identifies the consumer group. This ID allows Kafka to track which messages have already been processed (offsets). Changing this ID will reset the processing position (usually to the earliest available message). | Yes | fleak-production-consumer |
| Encoding Type | Defines how the raw bytes from Kafka should be deserialized into Fleak events. More info about encoding type can be found here Map<String, String> | Yes | JSON Object |
| Additional Properties | Key-value pairs for advanced Kafka Consumer configuration (see below). | No | None |
Advanced Configuration Properties
The Additional Properties section allows you to pass standard Kafka client configuration properties directly to the underlying consumer. This is commonly used for configuring security protocols, authentication credentials, or tuning client performance.
Note: Any property supported by the standard Apache Kafka ConsumerConfig class can be entered here.
Click + Add to insert specific key-value pairs.
Common Use Cases for Additional Properties
- Security & Authentication: If your cluster requires authentication (e.g., SASL/SSL), you can provide the necessary JAAS config, security protocols, and mechanism settings here.
- Connection Tuning: You can override default timeouts or connection settings (e.g.,
request.timeout.ms,session.timeout.ms) if your network environment requires it.
Related Nodes
- kafkaSink: Send processed data back to Kafka