Databricks Zerobus Sink Node
Quick Reference
In Destination, choose Asset Type and Asset Name, then open Configure Pipeline Settings. The fields listed here are the complete set of editable inputs for this Destination type, subject to the stated visibility conditions.
- Asset Type: Choose a Data Asset; UI selection determines the sink type on Save.
- Asset Name: Choose a Data Asset; reference to the selected data asset.
- Wire Protocol: Configure Pipeline Settings; Protobuf or JSON.
- Enable Store and Forward: Configure Pipeline Settings.
Destination has no Table Schema control. The asset selects the table. In Data Assets → Asset Information → Edit, update existing Catalog Name, Schema Name, and Table Name properties, then Save → Save Changes; the table location and schema are resolved from Unity Catalog. Save rejects a direct change to the displayed Table location input. Its shared Databricks connection profile owns Host URL, Client ID, Client Secret, and Zerobus Ingest Endpoint. When creating a profile, find the endpoint under + Show advanced options. It is needed for Zerobus writes; there is no separate Zerobus profile type. The retained settings localStoreMaxBytes and forwardRetryIntervalMillis are not visible controls.
Configuration
Configuration keys below are relative to the node configuration. Use the field name in the panel when following instructions in the builder; required markers such as an asterisk are not part of the name.
Destination panel
Choose Asset Type and Asset Name first. Type-specific inputs appear in Configure Pipeline Settings after an asset is selected; an asset error can prevent that section from appearing. A missing or unavailable asset is a reference problem, not proof that the connector does not exist.
The fields listed here are the complete set of editable inputs for this Destination type, subject to the stated visibility conditions.
| Field in the panel | Configuration key | Where to edit / when shown |
|---|---|---|
| Asset Type | UI only; no separate config key | Choose a Data Asset; UI selection determines the sink type on Save |
| Asset Name | dataAssetId | Choose a Data Asset; reference to the selected data asset |
| Wire Protocol | wireProtocol | Configure Pipeline Settings; Protobuf or JSON |
| Enable Store and Forward | storeAndForwardEnabled | Configure Pipeline Settings — Buffers records to a local durable queue during a connectivity outage and replays them automatically once the connection is restored. Prevents data loss and stops the pipeline from stalling. See below. Default: Off. |
Destination has no Table Schema control. The asset selects the table. In Data Assets → Asset Information → Edit, update existing Catalog Name, Schema Name, and Table Name properties, then Save → Save Changes; the table location and schema are resolved from Unity Catalog. Save rejects a direct change to the displayed Table location input. Its shared Databricks connection profile owns Host URL, Client ID, Client Secret, and Zerobus Ingest Endpoint. When creating a profile, find the endpoint under + Show advanced options. It is needed for Zerobus writes; there is no separate Zerobus profile type. The retained settings localStoreMaxBytes and forwardRetryIntervalMillis are not visible controls.
Overview
The Databricks Zerobus Sink Node streams processed records directly into a Databricks Unity Catalog managed Delta table over the Zerobus gRPC ingest API. Unlike the Databricks Sink, it does not stage files or run a COPY INTO, so there is no SQL Warehouse and no staging volume to configure.
How It Works
- Connect: The node opens a single long-lived gRPC stream to the Zerobus endpoint for the selected table.
- Encode: Each record is encoded as Protobuf (built from the table's Avro schema) or as JSON, depending on Wire Protocol.
- Ingest: Records are pushed onto the stream in batches of up to 10,000.
- Confirm: The node waits for Zerobus to confirm the last record of the batch is durable. Because Zerobus delivers in order per stream, that confirms the whole batch.
Delivery is at-least-once across reconnects. After a reconnect the stream re-sends unacknowledged records, so downstream consumers should deduplicate if they need exactly-once. Records that still fail are counted as errors, though some may already be committed if the failure came after Zerobus received them; records that arrive while a full store-and-forward queue is dropping them are lost; see Delivery Guarantees and Write Failures.
Prerequisites
Everything about where data lands comes from a Databricks Zerobus Table data asset, so you need one before adding the node.
The asset uses a shared Databricks connection profile, with Host URL, Client ID, Client Secret, and Zerobus Ingest Endpoint. Set the endpoint under + Show advanced options when creating the profile, or edit Configuration on an existing profile in Settings → Connection Profiles. A missing endpoint must be supplied for Zerobus writes; the Databricks profile type itself is accepted.
The asset must also carry an Avro schema. Unlike the Databricks Sink, Zerobus writes typed records directly, so deployment fails with a clear error if the asset has no schema.
Wire Protocol
The wire protocol is how the node talks to the Zerobus stream, so it is set on the node. A Databricks Zerobus Table data asset has a table schema and no format.
Protobuf (default) builds a protobuf descriptor from the selected table's Avro schema and encodes each record against it. A record that does not match the schema is rejected at encode time and reported as an error for that record; the rest of the batch still commits. This is where a field missing from the schema, or a value whose type cannot be converted, surfaces.
JSON opens the stream without a descriptor and serializes each record as JSON. No schema is used, so schema mismatches are not caught by the node before the record is sent.
Store and Forward
Turning on Enable Store and Forward makes a connectivity failure buffer records to a local durable queue instead of dropping them. A background forwarder replays them once the connection recovers, reopening the stream as needed.
- Only transient connectivity failures are buffered — network errors, timeouts, and the stream errors Databricks marks as recoverable. A permanent failure (schema, authentication, or permissions) is not buffered and goes through the normal error path.
- The local queue is capped at 1 GiB by default. Once the cap is reached, incoming records are dropped.
- The forwarder retries draining the queue every 5 seconds during an outage.
An outage is detected quickly: the node waits at most 20 seconds for a server acknowledgment and at most 60 seconds for a flush, well below the Databricks SDK defaults of 60 seconds and 5 minutes.
When to use the Zerobus Sink
- Choose Zerobus Sink for continuous, low-latency streaming into a Unity Catalog managed table, when you do not want to run a SQL Warehouse or manage a staging volume.
- Choose Databricks Sink when you need the staged
COPY INTOpath — for example to passCOPY INTOoptions, or to write to a table that is not a Zerobus target.
Related Nodes
- Databricks Sink: Load data into Unity Catalog tables via Parquet staging and
COPY INTO - Data Assets: Defining the target Zerobus table
- Connection Profiles: Databricks Zerobus endpoint and credentials