Skip to main content

Kafka Sink Node

Quick Reference​

Kafka Broker The address of your Kafka broker. ex: my-kafka-broker:9092

Topic The name of the Kafka topic to publish to. ex: my-topic

Partition Key Expression A path selection expression to extract the Kafka message Key from the record. If blank, the key will be null. ex: $.userId

Encoding Type The format used to serialize records before publishing to Kafka.

Additional Properties Extra Kafka configuration options as key–value pairs. ex: security.protocol=SASL_SSL

💡 Tip: More info about encoding types can be found here

💡 Tip: More info about path selection expression can be found here

Overview​

The Kafka Sink Node allows you to write processed data records from your Fleak workflow directly into an Apache Kafka topic.

Configuration​

Field NameDescriptionRequired?Default
Kafka BrokerThe host:port (bootstrap servers) of your Kafka cluster, comma-separated if multiple (e.g., kafka-1:9092,kafka-2:9092).YesN/A
TopicThe name of the destination Kafka topic.YesN/A
Partition Key ExpressionA path expression (e.g., $.user_id) used to extract a value from the record to serve as the Kafka message Key. If left blank, messages will be keyed as null (utilizing standard Kafka partitioning logic).
More info about path expression can be found here		Nonull
Encoding TypeThe format used to serialize the record value before publishing to Kafka. Options: CSV, JSON Object, JSON Array, JSON Lines.
Each record is published as an individual Kafka message. The encoding type controls the format of that single message (e.g., JSON Array wraps each record in an array).
Default: JSON Object. More info here.		YesJSON Object
Additional PropertiesKey-value pairs for advanced Kafka Producer configuration (see below).NoNone

Advanced Configuration Properties​

You can add custom properties to the Additional Properties section to override defaults or add security configurations (like SASL/SSL). Common overrides might include:

  • sasl.mechanism
  • security.protocol
  • sasl.jaas.config

The node applies the following performance defaults automatically unless overridden in the Additional Properties section:

  • Compression: lz4 (High performance compression)
  • Acks: 1 (Leader acknowledgment only)
  • Retries: 3
  • Batch Size (Kafka): 65536 bytes
  • Linger: 10 ms
  • Buffer Memory: 67108864 bytes (64 MB)
  • Max In-Flight Requests Per Connection: 5

Note: The default acks setting is 1. If you require stronger durability guarantees (e.g., waiting for all replicas), you should add a property acks with value all.

Note: Kafka delivery is asynchronous. If a message fails to be delivered (e.g., due to broker unavailability), the failure is tracked via metrics but is not returned synchronously to the caller.