Skip to main content

Kafka Source Node

Quick Reference​

  • Data Asset: Select a Kafka topic asset; brokers, topic name and authentication belong to the asset/profile.
  • Encoding Type: Defines how the raw bytes from Kafka should be deserialized into Fleak records. Options: CSV, JSON Object, JSON Array, JSON Lines, String Line, Text, XML.
    Default: JSON Array. More info here.
  • Read from: Beginning (earliest), End (latest), or Fail if no committed offset (none).
  • Additional Properties: Key-value rows; excludes auto.offset.reset because Read from controls that key; expand a row to edit Key and Value.

Additional Properties supports importing a client.properties file. Edit the offset-reset policy with Read from; do not add a second auto.offset.reset row.

Configuration​

Configuration keys below are relative to the node configuration. Use the field name in the panel when following instructions in the builder; required markers such as an asterisk are not part of the name.

Field in the panelConfiguration keyWhere to edit / when shown
Data AssetdataAssetIdSelect a Kafka topic asset; brokers, topic name and authentication belong to the asset/profile
Encoding TypeencodingTypeNode panel — Defines how the raw bytes from Kafka should be deserialized into Fleak records. Options: CSV, JSON Object, JSON Array, JSON Lines, String Line, Text, XML.
Default: JSON Array. More info here. Required: Yes. Placeholder: JSON Array.
Read fromproperties.auto.offset.resetBeginning (earliest), End (latest), or Fail if no committed offset (none) — Where to start consuming when the consumer group has no committed offset. An existing group resumes from its last committed position; changing Read from alone does not replay committed records. The current node panel has no Consumer Group ID control. Required: No. Placeholder: Beginning (earliest).
Additional Propertiesproperties.*Key-value rows; excludes auto.offset.reset because Read from controls that key; expand a row to edit Key and Value — Key-value pairs for advanced Kafka consumer configuration (see below). You can upload a client.properties file to populate the table. Required: No. Placeholder: None.

Additional Properties supports importing a client.properties file. Edit the offset-reset policy with Read from; do not add a second auto.offset.reset row.

Overview​

The Kafka Source node allows your workflow to ingest real-time streaming data from an Apache Kafka cluster. It acts as a starting point for your data pipeline, continuously polling a specified topic for new records.

The topic is defined by a Kafka Topic data asset, which names the topic and links the Apache Kafka connection profile holding the broker addresses, the security protocol, and the credentials. The node configures only how records are decoded and consumed.

Advanced Configuration Properties​

The Additional Properties section allows you to pass standard Kafka client configuration properties directly to the underlying consumer. This is commonly used for tuning client performance.

Note: Any standard Apache Kafka consumer configuration property can be entered here.

Security properties belong to the connection profile

The connection profile owns the cluster's security configuration, so security.protocol, sasl.mechanism, sasl.jaas.config, and the SSL password properties are rejected if set here. Change them on the Apache Kafka connection profile instead.

AWS MSK​

Amazon MSK clusters are consumed through this same node. MSK appears as its own tile when picking a source, but it is a label over the standard Kafka connector — the same engine command, the same Kafka Topic data asset.

Point an Apache Kafka connection profile at your MSK bootstrap brokers and configure authentication with the standard Kafka fields.

note

A dedicated AWS MSK Topic connection profile, with IAM-based authentication, is listed as coming soon and is not yet available. Until it ships, use an Apache Kafka profile.