Databricks Sink Node

Quick Reference

In Destination, choose Asset Type and Asset Name, then open Configure Pipeline Settings. The fields listed here are the complete set of editable inputs for this Destination type, subject to the stated visibility conditions.

• Asset Type: Choose a Data Asset; UI selection determines the sink type on Save.
• Asset Name: Choose a Data Asset; reference to the selected data asset.
• Batch Size: Configure Pipeline Settings.

Destination has no Table Schema control. In Data Assets → Asset Information → Edit, the asset's existing properties appear as Catalog Name, Schema Name, Table Name, Warehouse Id, and Volume Name. Edit those properties and Save → Save Changes; the service resolves the table and schema from Databricks. Save rejects a direct change to the displayed Table location input. Authentication belongs to the Databricks connection profile.

Configuration

Configuration keys below are relative to the node configuration. Use the field name in the panel when following instructions in the builder; required markers such as an asterisk are not part of the name.

Destination panel

Choose Asset Type and Asset Name first. Type-specific inputs appear in Configure Pipeline Settings after an asset is selected; an asset error can prevent that section from appearing. A missing or unavailable asset is a reference problem, not proof that the connector does not exist.

The fields listed here are the complete set of editable inputs for this Destination type, subject to the stated visibility conditions.

Field in the panel	Configuration key	Where to edit / when shown
Asset Type	UI only; no separate config key	Choose a Data Asset; UI selection determines the sink type on Save
Asset Name	dataAssetId	Choose a Data Asset; reference to the selected data asset
Batch Size	batchSize	Configure Pipeline Settings; required to save Destination — Number of records to accumulate before triggering a "Stage and Load" operation. The builder starts at 1,000; runtime validation requires a value from 1 to 100,000. Required: Yes.

Destination has no Table Schema control. In Data Assets → Asset Information → Edit, the asset's existing properties appear as Catalog Name, Schema Name, Table Name, Warehouse Id, and Volume Name. Edit those properties and Save → Save Changes; the service resolves the table and schema from Databricks. Save rejects a direct change to the displayed Table location input. Authentication belongs to the Databricks connection profile.

Legacy panel

The legacy Table Schema is read-only when an asset schema is present.

Use this section only when the workflow opens the legacy editor. Asset-managed connection values can be read-only; see Managing Data Assets for available edits and the replacement-asset flow.

Field in the panel	Configuration key	Where to edit / when shown
Change	dataAssetId	Legacy asset picker when an asset is selected. With no selection, Select followed by the catalog type label appears when assets exist; No followed by that label and items available appears when the catalog is empty (fallback: Data Asset).
Batch Size	batchSize	Legacy panel — Number of records to accumulate before triggering a "Stage and Load" operation. The builder starts at 1,000; runtime validation requires a value from 1 to 100,000.

Overview

The Databricks Sink Node allows you to ingest processed data directly into Databricks Unity Catalog tables. Unlike direct file writers, this node leverages a Databricks SQL Warehouse to ensure ACID compliance and governance within the Databricks ecosystem.

How It Works

This node operates in a three-step "Stage and Load" process to maximize reliability and throughput:

1. Stage: Data is converted to Parquet format locally.
2. Upload: Parquet files are uploaded securely to a Databricks Volume (Unity Catalog storage).
3. Load: A COPY INTO SQL command is executed on your designated SQL Warehouse. This command loads the data from the Volume into your target table transactionally.

Prerequisites

Before using the Databricks Sink, configure a Databricks connection profile. It supplies the host and OAuth credentials used to connect to the workspace.

Creating a New Databricks Table Data Asset

When creating a new data asset, you will need to provide:

• Select a Connection Profile — choose the Databricks profile when creating the data asset
• Catalog — the Unity Catalog catalog name
• Schema — the schema (database) within the catalog
• SQL Warehouse — the warehouse used to execute the COPY INTO command
• Staging Volume — the volume where Parquet files are staged before loading
• Table — the target table to write data into

Batch settings

Use Batch Size in Configure Pipeline Settings in Destination, or use Batch Size directly in the legacy panel.

These settings control the performance and behavior of the ingestion process but do not affect the destination topology.

Field Name	Description	Default
Batch Size	Number of records to accumulate before triggering a "Stage and Load" operation. Runtime validation requires a value from 1 to 100,000.	1,000 in the builder

The flush interval — the maximum time the node waits before flushing a partial batch — is not configurable from the builder and is fixed at 30 seconds.

Run Behavior

When Data Is Written

Records are not written one at a time. They accumulate in a buffer, and a "Stage and Load" runs when any of these happens:

1. The buffer reaches Batch Size records (default 1,000)
2. The flush interval elapses (30 seconds)
3. The node shuts down, which flushes whatever partial batch remains

The flush interval is what guarantees a low-volume workflow still lands data promptly, instead of waiting indefinitely for a batch to fill.

Test Runs Write to the Real Table

warning

A test run flushes every batch immediately, ignoring Batch Size and the 30-second flush interval. It performs a real Stage and Load: Parquet is uploaded to your volume and COPY INTO runs against the actual table. Point test runs at a table you are willing to write to.

In a deployed workflow, buffering and the flush interval behave as described above.

What Happens During a Flush

Each flush stages its batch under its own subdirectory of the volume, then:

1. Converts the records to Parquet. A record that cannot be converted is reported as an error; the rest of the batch continues.
2. Uploads the Parquet files, retrying up to 3 times per file. If a file still fails, the whole batch is aborted rather than partially loaded, so you never get a silent data gap.
3. Runs COPY INTO on the SQL Warehouse and waits for it to finish, up to 30 minutes.
4. Cleans up the local temp files and the volume subdirectory.

If a flush fails, it is retried up to 3 times with a growing delay (1s, then 2s). If it still fails, each record is validated individually: the bad ones are reported as errors with a reason, and the good ones are written. One malformed record does not cost you the batch.

Errors from a flush triggered by the timer or by shutdown are written to the Dead Letter Queue when one is configured. Without a DLQ those records are lost, so configure a DLQ for any workflow where losing records matters.

Input Format

For each column in the selected table's schema, the node looks up that exact field name in the record. Fields in the record that are not in the schema are ignored; columns not present in the record are treated as null.

Writing a null — or omitting the field entirely — for a non-nullable column fails that record. The error names the column and lists the field names that were actually present, which makes a field-name typo easy to spot.

Value Conversion

Values do not have to arrive already in the column's type. Each value is converted as follows, and a value that cannot be converted fails that record:

Column type	Accepted input
String	Any value, converted to text
Integer, Long, Float, Double, Short, Byte	Any number, or text that parses as that number
Date	A whole number, or text that parses as one (days since epoch)
Boolean	Boolean values retain their value; for other non-null values, text "true" (case-insensitive) becomes true and every other textual value becomes false. Null remains null.
Timestamp	A timestamp, an instant, or a whole number of epoch milliseconds (including as text)
Decimal	A decimal, any other number, or text in decimal form
Binary	Raw bytes, or the text form of the value
Array	A list. Each element is converted by these same rules.
Struct	A nested object.

warning

Boolean values retain their value. For other non-null values, the textual value "true" (case-insensitive) becomes true; every other textual value becomes false instead of raising a parsing error. For example, "TRUE" → true, while "yes" → false. Null remains null, subject to the column's nullable constraint described above. Equality with Boolean true is a type-sensitive predicate, not a general Boolean cast or a conversion recipe. Text "true" compared with Boolean true yields false; null compared with Boolean true also yields false. That predicate does not preserve the string or null behavior described above. Do not present an equality expression as the implementation of this sink conversion or invent an equality-based recipe for an unspecified input policy. For a different conversion, first define the accepted input values and the intended true/false/null or error policy; if no such policy is defined, describe the sink behavior without proposing a custom expression.

Schema Validation

The Databricks Sink validates the schema of your selected table against the actual Databricks table schema at startup. If the schemas do not match, the pipeline will not start. This prevents data quality issues caused by schema drift between your pipeline and the target table.

When to use the Databricks Sink

• Choose Databricks Sink if you are building business-critical tables that analysts query immediately, and you want the safety and ease of Databricks Unity Catalog governance.
• Choose Delta Lake Sink (not generally available; offered only to accounts where it has been enabled) if you are building a raw data landing zone, are sensitive to compute costs, or need to write data to storage that isn't strictly coupled to a Databricks workspace.