Data Assets
Data Assets name a single location on an external platform — a queue, topic, table, folder, or endpoint — so workflows reference it by name instead of repeating paths and connection settings on every node.
Overview
A data asset is a named pointer to one location on a platform Fleak already knows how to reach. The credentials and endpoint for that platform come from a Connection Profile; the asset adds the location on top of it.
| Layer | Answers | Example |
|---|---|---|
| Connection Profile | Which platform, and what credentials | An AWS access key pair plus us-east-1 |
| Data Asset | Which location on that platform | The queue https://sqs.us-east-1.amazonaws.com/123456789012/orders |
| Node | How the workflow reads or writes it | Batch size, flush interval, encoding |
In Destination, Choose a Data Asset contains Asset Type and Asset Name. Configure Pipeline Settings contains the selected connector's node settings, such as batch size or encoding. Those examples describe connector settings, not the complete panel. The data asset supplies its location; its connection profile supplies authentication. Reusing an asset avoids re-entering that location and profile in every workflow.
Every data asset needs a Connection Profile for its platform. You can create one in advance under Settings → Connection Profiles, or inline while creating the asset. A Splunk HEC Endpoint is the exception: its API Key profile can only be created inline, as described under Splunk HEC Endpoint.
Supported Data Asset Types
The types below back the connectors that are generally available. Accounts with additional connectors enabled see further types in the asset catalog; those are not generally available, and the ones documented here are under Asset Types Not Generally Available.
| Type | Location field | Connection Profile | Used by |
|---|---|---|---|
| AWS S3 Folder | Bucket and Folder / Prefix | AWS | S3 Sink, File System Source |
| AWS SQS Queue | Queue URL | AWS | SQS Sink, SQS Source |
| AWS SQS Event Notifications | Queue URL | AWS | S3 Event Notifications Source |
| Kafka Topic | Topic | Apache Kafka | Kafka Sink, Kafka Source |
| Databricks Table | Table | Databricks | Databricks Sink |
| Databricks Zerobus Table | Table | Databricks | Databricks Zerobus Sink |
| Splunk HEC Endpoint | HEC URL | API Key | Splunk HEC Sink |
| Datadog Logs | Site (derived) | Datadog | Datadog Logs Sink |
| Microsoft Sentinel Table | Data Collection Endpoint | Microsoft Sentinel Workspace | Microsoft Sentinel Sink |
| Snowflake Table | Table | Snowflake | Snowflake Sink |
AWS S3 Folder
A folder in an S3 bucket, using an AWS connection profile. In the current create form, enter Asset Name, choose Bucket, optionally set Folder / Prefix, and choose Encoding Type: CSV, JSON Array, or JSON Lines (the default). An empty Folder / Prefix selects the whole bucket. Description is optional; Region and Storage Class are read-only when available. The final action is Import.
The current S3 create and edit forms have no Avro Schema input and do not offer Parquet as a selectable Encoding Type. An existing asset with another encoding, such as Parquet, shows Encoding Type read-only. These forms cannot add an Avro schema.
In the current Destination editor, Message Format is an editable node setting: CSV, JSON Array, JSON Lines, and Parquet only when the selected asset already has an Avro schema. You can use an existing asset with a schema for Parquet. If its metadata is loading or unavailable, schema availability is unknown; do not infer that the schema is absent. The asset encoding and the node Message Format are separate settings.
AWS SQS Queue
An SQS queue addressed by its full URL. The region and queue type are derived from the URL and stored on the asset — you do not enter them separately.
AWS SQS Event Notifications
The SQS queue that an S3 bucket publishes its event notifications to. Stored the same way as an SQS Queue, but kept as a distinct type because it is consumed differently: the S3 Event Notifications Source polls the queue and then downloads the objects the events name.
A single AWS connection profile's credential both polls the queue and reads the objects. Cross-account bucket access is configured on the node, not the asset.
Kafka Topic
A topic on the cluster its Kafka connection profile points at. Because the profile owns the security configuration, the asset carries only the topic name.
Databricks Table
A table in Databricks Unity Catalog. Tables are imported from your catalog rather than typed, so the asset always names a table that exists.
Databricks Zerobus Table
A table written through the Databricks Zerobus ingest endpoint. It uses the same Databricks connection profile as other Databricks assets. Set Zerobus Ingest Endpoint under + Show advanced options when creating that profile; it is required for Zerobus writes, not a separate profile type.
Splunk HEC Endpoint
An HTTP Event Collector endpoint. Takes the HEC URL, an API Key connection profile holding the HEC token, and a Verify SSL Certificates setting. SSL verification is on unless explicitly turned off.
The HEC token is entered while creating the asset, not on the settings page. In Connect to Splunk, Select a Connection Profile lists existing API Key profiles, marked (API key). To enter a new token, choose + Add new connection profile, enter a Connection Name, paste the token into API Key, and click Connect & Continue. The next step takes Data Asset Name, HEC URL, an optional Index, Verify SSL Certificates, and an optional Description; click Import to create the asset. A new profile is saved together with the asset on Import. A Splunk connection profile (Splunk URL, username, and password) is a different profile type and cannot back a Splunk HEC Endpoint.
The HEC URL must be the full event endpoint, including the path — for example
https://splunk.example.com:8088/services/collector/event. Records are posted to this URL
exactly as entered, so host and port alone (https://splunk.example.com:8088) would reach
Splunk's root path and fail with a 404; the create form rejects a URL without a path.
Datadog Logs
A named tagging target for Datadog Logs. Unusually, this asset has no location you type: its location is the site from the Datadog connection profile, resolved automatically.
What the asset does carry is the labelling stamped on every event sent through it — any
non-empty combination of Source (ddsource), Service (service), and Tags
(ddtags). This is how one Datadog profile serves several differently-tagged streams.
Managing Data Assets
Accessing Data Assets
Open Data Assets directly in the main left sidebar to view and manage assets (/data-assets). The current page heading is Data Asset Management. Data Assets is not inside Settings. Settings → Connection Profiles is the separate location for profile credentials and platform configuration.
Creating Data Assets
Click Add New Data Asset. In Select Asset Type, select the offered type and continue. For an S3 folder, the next step is Connect to AWS S3: choose Select a Connection Profile, or create a compatible AWS profile inline, then click Connect & Continue. In Select Asset, fill in the S3 fields described under AWS S3 Folder and click Import. Other types have their own asset fields.
In the workflow builder, the Destination Asset Name picker also offers Create New Asset. After creation, select the resulting asset and Save the node.
Viewing Usage
Each asset lists the workflows that reference it, and whether each of those workflows is currently deployed. Workflows you do not have permission to open are shown but not linked, so the count is always accurate.
Editing and Re-deploying
Open an asset you own and click Edit in Asset Information. The current form provides Name and Description; Platform, Asset Type, and Connection Profile are read-only in this card. After editing, click Save, then Save Changes in Re-deployment Required. Saving updates the asset, but active deployments keep their previous settings until you re-deploy each affected workflow.
For non-cloud-folder assets, additional property inputs appear only for keys already stored on the asset. You can edit their values, but cannot add or rename property keys here. Datadog is the exception: Source, Service, and Tags are always offered. A visible location input does not guarantee that its value can be saved: the fixed targets below reject a changed location. Leave that input unchanged when editing supported properties.
| Asset type | Supported edits in Asset Information and target limitations |
|---|---|
| S3 folder | Bucket, Folder / Prefix, and Encoding Type for CSV, JSON Array, or JSON Lines. Region is read-only when present. An existing Parquet Encoding Type is read-only. There is no Avro Schema input. |
| Databricks table | Existing Catalog Name, Schema Name, Table Name, Warehouse Id, and Volume Name properties. Save resolves the table and schema from Databricks. The displayed Table location input cannot be changed directly. |
| Databricks Zerobus table | Existing Catalog Name, Schema Name, and Table Name properties. Save resolves the table and schema from Unity Catalog. The displayed Table location input cannot be changed directly. |
| SQS queue | Region can be updated. Queue Type is derived from the saved queue URL, not independently changed. Save rejects a changed Queue URL; use a replacement asset for another queue. |
| Splunk HEC endpoint | Existing Index and Verify SSL Certificates properties. Save rejects a changed HEC URL; use a replacement asset for another HEC URL. |
| Datadog Logs | Source, Service, and Tags can be updated. Site is read-only and comes from the connection profile. |
| Microsoft Sentinel DCR stream | Existing Immutable Dcr Id and Stream Name properties. Save rejects a changed Data Collection Endpoint; use a replacement asset for another endpoint. |
| Snowflake table | Existing Database, Schema, and Warehouse properties. Save rejects a changed Table location; use a replacement asset for another table. |
| Kafka topic | Existing source-read properties can be updated; they do not change the producer connection. Save rejects a changed Topic; use a replacement asset for another topic. |
To change the linked profile, use the separate Connection Profile → Edit card, choose an available compatible Connection Profile, then Save → Save Changes. This changes the association, not the secrets inside the profile. Profile credentials and platform fields are edited separately in Settings → Connection Profiles; API Key profiles are not listed there; open one from the profile name in the asset's Connection Profile card (see API Key). The asset owner must have access to the compatible profile; a shared asset requires a shared profile, and an S3 folder requires an AWS profile.
For a fixed target, or a required property not offered in the edit form, create a replacement through the creation route available for that type: Data Assets → Add New Data Asset for types offered there, or Create New Asset in Destination when the current catalog enables it. Set the required values and profile in the available creation form. Then select the replacement in the node's Asset Name and Save. Creating another S3 asset through the current form does not provide an Avro schema editor.
Deleting
An asset that workflows still reference cannot be deleted until those references are removed. The usage list shows what is blocking the delete.
Using Data Assets in Workflows
When you configure a source or sink node, you select a data asset rather than typing a path, queue URL, or table name. The asset owns the location, and its connection profile owns authentication. In Destination, select Asset Type and Asset Name under Choose a Data Asset, then edit the connector's node settings under Configure Pipeline Settings. The asset selectors remain part of the Destination panel.
The current Destination controls depend on the selected connector; follow its Destination panel section. For example, S3 Message Format is editable on the node. Read-only connection controls or schema details described in a Legacy panel section do not imply that those controls appear in Destination.
Asset Types Not Generally Available
The asset types in this section back connectors that are not generally available. They appear in the asset catalog only for accounts where those connectors have been enabled. Each connector's own page states its availability.
| Type | Connection Profile | Used by |
|---|---|---|
| Databricks Delta Table | AWS | Delta Lake Sink |
| GCP Pub/Sub Topic | GCP | Pub/Sub Sink |
| Elasticsearch Index | Elasticsearch Index | Elasticsearch Sink |
| XSIAM Dataset | XSIAM or API Key | XSIAM Sink |
| Splunk Observability | Splunk Observability | SignalFx Sink |
Supported edits in Asset Information for these types follow the rules in Editing and Re-deploying:
| Asset type | Supported edits in Asset Information and target limitations |
|---|---|
| Delta table | Table Path can point to an existing Delta table; saving imports its schema. There is no Schema handling or Avro Schema control in this edit form. An empty target requires a schema and cannot be created from this form. |
| Pub/Sub topic | Save rejects a changed Topic and does not accept editable asset properties. Use a replacement asset for another topic. |
| Elasticsearch index | Save rejects a changed Index and does not accept editable asset properties. Use a replacement asset for another index. |
| SignalFx ingest | Existing default dimensions appear as separate property inputs named after the dimension keys. There is no add-key control; clearing a value removes that dimension on Save. Save rejects a changed Realm input. Switching the linked profile derives Realm from that profile. |
Delta table creation is not offered in the main Add New Data Asset flow; Destination retains its Delta asset dialog when the catalog offers that type.
Related pages
- Connection Profiles — the credentials these assets build on
- Workflow Lifecycle — when a re-deploy is required