Delta Lake Sink Node
The Delta Lake Sink node is not generally available in Fleak Data Workflow. The workflow builder offers it only to accounts where it has been enabled. For the connectors that are generally available, see Data Assets.
This page is retained for accounts where it is enabled.
When using S3 as the cloud storage backend, Delta Lake does not support distributed writing. Deploying a data pipeline job with multiple replicas writing data in Delta Lake format to S3 will likely cause data corruption due to the lack of a distributed lock implementation. For S3 deployments, ensure your pipeline runs with a single replica.
Quick Reference
In Destination, choose Asset Type and Asset Name, then open Configure Pipeline Settings. The fields listed here are the complete set of editable inputs for this Destination type, subject to the stated visibility conditions.
- Asset Type: Choose a Data Asset; choose Databricks Delta Table for the Delta Lake sink; UI selection determines the sink type on Save.
- Asset Name: Choose a Data Asset; reference to the selected data asset.
- Partition Columns: Configure Pipeline Settings; Add Column adds a row.
- Additional Properties: Configure Pipeline Settings; key-value rows; expand a row to edit Key and Value.
Destination has no Table Schema control.
| Resource | Settings owned there |
|---|---|
| Node (Destination) | Partition Columns and Additional Properties; Asset Type and Asset Name select the referenced asset. |
| Data asset | Table location and Avro schema. |
| Connection profile (AWS) | Storage authentication: Region, AWS Access Key ID, AWS Secret Access Key. The asset links to the profile; credentials remain on the profile. |
In Data Assets → Asset Information → Edit, Table Path can be changed to an existing Delta table; saving imports that table's schema. This edit form has no Schema handling or Avro Schema control, so it cannot create a table at an empty path. Use the separate create-asset flow below for a new table.
Configuration
Configuration keys below are relative to the node configuration. Use the field name in the panel when following instructions in the builder; required markers such as an asterisk are not part of the name.
Destination panel
Choose Asset Type and Asset Name first. Type-specific inputs appear in Configure Pipeline Settings after an asset is selected; an asset error can prevent that section from appearing. A missing or unavailable asset is a reference problem, not proof that the connector does not exist.
The fields listed here are the complete set of editable inputs for this Destination type, subject to the stated visibility conditions.
| Field in the panel | Configuration key | Where to edit / when shown |
|---|---|---|
| Asset Type | UI only; no separate config key | Choose a Data Asset; choose Databricks Delta Table for the Delta Lake sink; UI selection determines the sink type on Save |
| Asset Name | dataAssetId | Choose a Data Asset; reference to the selected data asset |
| Partition Columns | partitionColumns[] | Configure Pipeline Settings; Add Column adds a row — A list of column names used to partition the data physically on storage. These columns must match the partition columns already defined on the target Delta table. Default: None. |
| Additional Properties | hadoopConfiguration.* | Configure Pipeline Settings; key-value rows; expand a row to edit Key and Value |
Destination has no Table Schema control.
| Resource | Settings owned there |
|---|---|
| Node (Destination) | Partition Columns and Additional Properties; Asset Type and Asset Name select the referenced asset. |
| Data asset | Table location and Avro schema. |
| Connection profile (AWS) | Storage authentication: Region, AWS Access Key ID, AWS Secret Access Key. The asset links to the profile; credentials remain on the profile. |
In Data Assets → Asset Information → Edit, Table Path can be changed to an existing Delta table; saving imports that table's schema. This edit form has no Schema handling or Avro Schema control, so it cannot create a table at an empty path. Use the separate create-asset flow below for a new table.
Legacy panel
The legacy Table Schema is a read-only view of the selected asset Avro schema.
Use this section only when the workflow opens the legacy editor. Asset-managed connection values can be read-only; see Managing Data Assets for available edits and the replacement-asset flow.
| Field in the panel | Configuration key | Where to edit / when shown |
|---|---|---|
| Change | dataAssetId | Legacy asset picker when an asset is selected. With no selection, Select followed by the catalog type label appears when assets exist; Add followed by that label appears when the catalog is empty (fallback: Data Asset). |
| Partition Columns | partitionColumns[] | Additional Settings — A list of column names used to partition the data physically on storage. These columns must match the partition columns already defined on the target Delta table. Default: None. |
| Hadoop Configuration | hadoopConfiguration.* | Additional Settings; key-value rows; expand a row to edit Key and Value — Advanced Key-Value pairs to override underlying Hadoop file system settings. Do not place cloud storage credentials (access keys, secret keys, etc.) here — use credentials from the asset's AWS connection profile instead. Default: None. |
Overview
The Delta Lake Sink Node enables you to write processed workflow data directly into a physical storage location in the Delta Lake table format.
Important Prerequisites
Pre-existing Tables Only: This node does not create new Delta tables automatically. The table must already exist at the path configured on the selected data asset, with a defined schema. If the table is not found during initialization, the workflow will fail.
The main Data Assets → Add New Data Asset flow does not offer Delta table creation. In Destination, when the current catalog offers the Delta type and Create New Asset, that action opens the Delta asset dialog. Select Storage Integration for the compatible AWS storage connection and Mode: Create New takes Location and Avro Schema; Import Existing reads the schema from the table at Location. Choose the resulting asset in Asset Name and Save. Existing Delta assets remain selectable. To repoint an owned existing asset to another existing Delta table, use Data Assets → Asset Information → Edit → Table Path, then Save → Save Changes. The edit form has no Schema handling or Avro Schema input, so it cannot create a table at an empty path; the node itself does not create a table during pipeline execution.
Schema Validation: Incoming data is strictly validated against the existing Delta table's schema.
Write Behavior
- Append-only: All writes are append operations. Overwrite and merge modes are not supported.
- Buffered writes: Data is buffered and flushed periodically. The batch size and flush interval are system-managed and not user-configurable.
Storage & Authentication
A Delta table asset uses an AWS connection profile for S3 storage. The reachable Delta asset dialog labels its selector Storage Integration. The AWS profile supplies Region, AWS Access Key ID, and AWS Secret Access Key; a generic Username/Password credential is not the asset profile type. Use an s3a:// storage location. The current Destination panel has no inline storage credential controls.
When to use the Delta Lake Sink
- Choose Databricks Sink if you are building business-critical tables that analysts query immediately, and you want the safety and ease of Databricks Unity Catalog governance.
- Choose Delta Lake Sink if you are building a raw data landing zone, are sensitive to compute costs, or need to write data to storage that isn't strictly coupled to a Databricks workspace.