Kafka Sink Node

Quick Reference

In Destination, choose Asset Type and Asset Name, then open Configure Pipeline Settings. The fields listed here are the complete set of editable inputs for this Destination type, subject to the stated visibility conditions.

• Asset Type: Choose a Data Asset; UI selection determines the sink type on Save.
• Asset Name: Choose a Data Asset; reference to the selected data asset.
• Payload Format: Configure Pipeline Settings.
• Additional Properties: Configure Pipeline Settings; Kafka producer key-value properties; expand a row to edit Key and Value.

Kafka and MSK selections use the same Destination fields and save as kafkasink. A saved node does not preserve a separate MSK selector identity. Destination has no Partition Key Expression or Enable Store and Forward control. The asset owns Topic; the Apache Kafka profile owns Broker/Bootstrap Address and authentication. Asset Information → Edit shows Topic, but Save rejects a changed topic: create a replacement asset, select it in Asset Name, then Save. Edit broker/authentication on the profile in Settings → Connection Profiles.

Configuration

Configuration keys below are relative to the node configuration. Use the field name in the panel when following instructions in the builder; required markers such as an asterisk are not part of the name.

Destination panel

Choose Asset Type and Asset Name first. Type-specific inputs appear in Configure Pipeline Settings after an asset is selected; an asset error can prevent that section from appearing. A missing or unavailable asset is a reference problem, not proof that the connector does not exist.

The fields listed here are the complete set of editable inputs for this Destination type, subject to the stated visibility conditions.

Field in the panel	Configuration key	Where to edit / when shown
Asset Type	UI only; no separate config key	Choose a Data Asset; UI selection determines the sink type on Save
Asset Name	dataAssetId	Choose a Data Asset; reference to the selected data asset
Payload Format	encodingType	Configure Pipeline Settings
Additional Properties	properties.*	Configure Pipeline Settings; Kafka producer key-value properties; expand a row to edit Key and Value — Key-value pairs for advanced Kafka Producer configuration (see below). Default: None.

Kafka and MSK selections use the same Destination fields and save as kafkasink. A saved node does not preserve a separate MSK selector identity. Destination has no Partition Key Expression or Enable Store and Forward control. The asset owns Topic; the Apache Kafka profile owns Broker/Bootstrap Address and authentication. Asset Information → Edit shows Topic, but Save rejects a changed topic: create a replacement asset, select it in Asset Name, then Save. Edit broker/authentication on the profile in Settings → Connection Profiles.

Legacy panel

Legacy localStoreMaxBytes and forwardRetryIntervalMillis are hidden settings, not editable fields.

Use this section only when the workflow opens the legacy editor. Asset-managed connection values can be read-only; see Managing Data Assets for available edits and the replacement-asset flow.

Field in the panel	Configuration key	Where to edit / when shown
Kafka Broker	broker	Legacy panel; read-only when dataAssetId is set, because the asset/profile supplies this value — The host:port (bootstrap servers) of your Kafka cluster, comma-separated if multiple (e.g., kafka-1:9092,kafka-2:9092). Default: N/A.
Topic	topic	Legacy panel; read-only when dataAssetId is set, because the asset/profile supplies this value — The name of the destination Kafka topic. Default: N/A.
Partition Key Expression	partitionKeyFieldExpressionStr	Legacy panel; read-only when dataAssetId is set, because the asset/profile supplies this value — A path expression (e.g., $.user_id) used to extract a value from the record to serve as the Kafka message Key. If left blank, messages will be keyed as null (utilizing standard Kafka partitioning logic). More info about path expression can be found here Default: null.
Encoding Type	encodingType	Legacy panel; read-only when dataAssetId is set, because the asset/profile supplies this value — The format used to serialize the record value before publishing to Kafka. Options: CSV, JSON Object, JSON Array, JSON Lines. Each record is published as an individual Kafka message. The encoding type controls the format of that single message (e.g., JSON Array wraps each record in an array). Default: JSON Object. More info here. Default: JSON Object.
Additional Properties	properties.*	Legacy panel; expand a row to edit Key and Value — Key-value pairs for advanced Kafka Producer configuration (see below). Default: None.
Enable Store and Forward	storeAndForwardEnabled	Legacy panel; read-only when dataAssetId is set, because the asset/profile supplies this value — Buffers records to a local durable queue during a connectivity outage and replays them automatically once the broker is reachable again. Prevents data loss and stops the pipeline from stalling. See below. Default: Off.

Overview

The Kafka Sink Node allows you to write processed data records from your Fleak workflow directly into an Apache Kafka topic.

The target topic is normally defined by a Kafka Topic data asset, which names the topic and links the Apache Kafka connection profile holding the broker addresses, the security protocol, and the credentials. In the legacy editor, when an asset manages the node, Fleak shows:

This node is managed by a data asset. Connection settings are read-only here; change them on the data asset, or edit the node in the Destination panel.

For Destination, edit Broker/Bootstrap Address and authentication on the Apache Kafka profile in Settings → Connection Profiles. Topic is set when creating the asset; Asset Information → Edit shows Topic, but Save rejects a changed topic. Create a replacement asset, select it in Asset Name, and Save. Re-deploy affected workflows for the change to take effect. The legacy notice describes ownership, not an additional Topic editor in Destination.

Advanced Configuration Properties

You can add custom properties to the Additional Properties section to override defaults or add security configurations (like SASL/SSL). Common overrides might include:

• sasl.mechanism
• security.protocol
• sasl.jaas.config

Note: These security properties are only accepted when the node holds its own connection settings. If the node's connection comes from a managed data asset or credential, that asset owns the security configuration, and setting security.protocol, sasl.mechanism, sasl.jaas.config, ssl.keystore.password, ssl.key.password, or ssl.truststore.password here is rejected.

The node applies the following performance defaults automatically unless overridden in the Additional Properties section:

• Compression: lz4 (High performance compression)
• Acks: all (All in-sync replicas must acknowledge)
• Idempotence: enabled automatically (see note below)
• Retries: 3
• Batch Size (Kafka): 65536 bytes
• Linger: 10 ms
• Buffer Memory: 67108864 bytes (64 MB)
• Max In-Flight Requests Per Connection: 5

Note: The node waits for broker acknowledgments before reporting a batch as written, and applies acks=all. A record whose acknowledgment fails is reported as an error rather than silently succeeding, and an unreachable broker fails the batch up front instead of blocking on every record.

Note: Idempotence is enabled automatically, but only while the effective configuration still allows it. Overriding acks to 1, retries to 0, or max.in.flight.requests.per.connection above 5 in Additional Properties turns idempotence back off, because the Kafka producer will not start with idempotence enabled under those settings.

Store and Forward

In the legacy panel, turning on Enable Store and Forward makes a connectivity failure buffer records to a local durable queue instead of dropping them. A background forwarder drains the queue back to Kafka once the broker is reachable again.

• Only transient connectivity failures are buffered — network errors and timeouts. A permanent failure (for example a malformed record) still goes through the normal error path.
• The local queue is capped at 1 GiB by default. Once the cap is reached, incoming records are dropped.
• The forwarder retries draining the queue every 5 seconds during an outage.
• Store and forward is disabled during a test run, since a test run is a single in-memory shot and a durable queue would leave an orphaned on-disk store behind. Use a deployed workflow to exercise it.

Note: Store and forward is unavailable when the node's connection is supplied by a managed data asset, since the asset owns the connection settings.

Related Nodes

• Kafka Source: Read data from Kafka topics
• Data Assets: Defining the target topic
• Connection Profiles: Kafka broker and authentication settings