Skip to main content

Kafka Sink Node

Quick Reference​

Kafka Broker The address of your Kafka broker. ex: localhost:9092

Topic The name of the Kafka topic to consume from. ex: my-topic

Partition Key Expression A path selection expression to extract the Kafka message Key from the record. If blank, the key will be null. ex: $.user_id

Encoding Type The format of the data in the Kafka topic.

Additional Properties Extra Kafka configuration options as key–value pairs. ex: security.protocol=SASL_SSL

💡 Tip: More info about encoding types can be found here

💡 Tip: More info about path selection expression can be found here

Overview​

The Kafka Sink Node allows you to write processed data records from your Fleak workflow directly into an Apache Kafka topic.

Configuration​

Field NameDescriptionRequired?Default
Kafka BrokerThe host:port (bootstrap servers) of your Kafka cluster (e.g., kafka-prod:9092).YesN/A
TopicThe name of the destination Kafka topic.YesN/A
Partition Key ExpressionA path expression (e.g., $.user_id) used to extract a value from the record to serve as the Kafka message Key. If left blank, messages will be keyed as null (utilizing standard Kafka partitioning logic).
More info about path expression can be found here		Nonull
Encoding TypeThe format used to serialize the record value (e.g., JSON Object).
More info about encoding type can be found here		YesN/A
Additional PropertiesKey-value pairs for advanced Kafka Producer configuration (see below).NoNone

Advanced Configuration Properties​

You can add custom properties to the Additional Properties section to override defaults or add security configurations (like SASL/SSL). Common overrides might include:

  • sasl.mechanism
  • security.protocol
  • sasl.jaas.config

The node applies the following performance defaults automatically unless overridden in the Additional Properties section:

  • Compression: lz4 (High performance compression)
  • Acks: 1 (Leader acknowledgment only)
  • Retries: 3
  • Batch Size (Kafka): 65536 bytes
  • Linger: 10 ms

Note: The default acks setting is 1. If you require stronger durability guarantees (e.g., waiting for all replicas), you should add a property acks with value all.