S3 Output Node

The S3 Sink node writes workflow output data to AWS S3 via a configured data asset. It batches records by default for efficient, high-volume streaming.

The S3 Output Node stores workflow output data in Amazon S3. It uses a data asset to encapsulate the AWS connection details, making it ideal for data archival, backup, or integration with other AWS services.

Quick Reference

In Destination, choose Asset Type and Asset Name, then open Configure Pipeline Settings. The fields listed here are the complete set of editable inputs for this Destination type, subject to the stated visibility conditions.

• Asset Type: Choose a Data Asset; UI selection determines the sink type on Save.
• Asset Name: Choose a Data Asset; reference to the selected data asset.
• Message Format: Configure Pipeline Settings; Parquet is offered only if the selected asset has an Avro schema.
• Compression: Configure Pipeline Settings; None or GZIP; hidden when Message Format is Parquet.
• Batch Size: Configure Pipeline Settings.
• Flush Interval (ms): Configure Pipeline Settings.

Destination edits Message Format on the node; it does not show an Encoding/Schema details panel or a View details action. The S3 bucket, prefix, and any existing Avro schema belong to the data asset. In Data Assets, open an asset you own and choose Asset Information → Edit to change Bucket, Folder / Prefix, or a supported Encoding Type. AWS credentials belong to the asset's connection profile: open a profile you own in Settings → Connection Profiles, choose Configuration → Edit, and Save to change those credentials. The asset's separate Connection Profile → Edit card selects a compatible profile; it does not edit that profile's secrets. The current S3 asset create and edit forms do not offer an Avro Schema input or a Parquet encoding choice; an existing Parquet asset's Encoding Type is read-only. You can select an existing asset that already has an Avro schema to make Parquet available in Destination. These forms cannot add a schema. Changing the asset encoding does not replace the node's Message Format selection.

Configuration

Configuration keys below belong in the node's config object when configuring s3sink through MCP or the API.

Key	Type	Required	Default	Description
dataAssetId	string (UUID)	Yes	None	ID of an accessible S3 data asset with an AWS connection profile.
encodingType	string	If the asset has no encoding	Asset encoding	Output format: CSV, JSON_ARRAY, JSON_OBJECT_LINE, JSON_OBJECT, or PARQUET. A value on the node takes precedence over the asset's encoding.
batching	boolean	No	true	Buffer records before writing. Set to false for JSON_OBJECT; PARQUET requires true.
compressionType	string or null	No	No compression	Omit this key or use null for no compression. Use "GZIP" to enable compression. "NONE" is not a valid value. For PARQUET, omit the key or use null; Parquet is compressed internally.
batchSize	integer	No	10000	Maximum records per batch. Must be greater than 0 when batching is enabled.
flushIntervalMillis	integer	No	10000	Maximum time in milliseconds before a buffered batch is flushed. Applies when batching is enabled.

The selected data asset supplies the S3 bucket, prefix, region, connection-profile credentials, and any configured Avro schema. These asset-managed settings do not need to be copied into the node configuration. If encodingType is omitted, the asset must supply it.

For JSON Lines with default batching and no compression, use the following config object. Replace <s3-data-asset-id> with the ID of your S3 data asset:

{
  "dataAssetId": "<s3-data-asset-id>",
  "encodingType": "JSON_OBJECT_LINE"
}

To configure batching explicitly and enable GZIP compression:

{
  "dataAssetId": "<s3-data-asset-id>",
  "encodingType": "JSON_OBJECT_LINE",
  "batching": true,
  "compressionType": "GZIP",
  "batchSize": 10000,
  "flushIntervalMillis": 10000
}

When using the builder, follow the field names in the panel below; required markers such as an asterisk are not part of the name. The builder enables batching automatically. Its None compression option corresponds to compressionType: null, and its required form fields can differ from the keys required in JSON.

Destination panel

Choose Asset Type and Asset Name first. Type-specific inputs appear in Configure Pipeline Settings after an asset is selected; an asset error can prevent that section from appearing. A missing or unavailable asset is a reference problem, not proof that the connector does not exist.

The fields listed here are the complete set of editable inputs for this Destination type, subject to the stated visibility conditions.

Field in the panel	Configuration key	Where to edit / when shown
Asset Type	UI only; no separate config key	Choose a Data Asset; UI selection determines the sink type on Save
Asset Name	dataAssetId	Choose a Data Asset; reference to the selected data asset
Message Format	encodingType	Configure Pipeline Settings; Parquet is offered only if the selected asset has an Avro schema
Compression	compressionType	Configure Pipeline Settings; hidden when Message Format is Parquet, which is compressed internally — None or GZIP. Default: None. Required: No.
Batch Size	batchSize	Configure Pipeline Settings; required to save Destination — Number of records to accumulate before writing. Minimum: 1. Required: Yes. Placeholder: 10,000.
Flush Interval (ms)	flushIntervalMillis	Configure Pipeline Settings — Maximum time (ms) to wait before forcing a write. Required: No. Placeholder: 10,000.

Destination edits Message Format on the node; it does not show an Encoding/Schema details panel or a View details action. The S3 bucket, prefix, and any existing Avro schema belong to the data asset. In Data Assets, open an asset you own and choose Asset Information → Edit to change Bucket, Folder / Prefix, or a supported Encoding Type. AWS credentials belong to the asset's connection profile: open a profile you own in Settings → Connection Profiles, choose Configuration → Edit, and Save to change those credentials. The asset's separate Connection Profile → Edit card selects a compatible profile; it does not edit that profile's secrets. The current S3 asset create and edit forms do not offer an Avro Schema input or a Parquet encoding choice; an existing Parquet asset's Encoding Type is read-only. You can select an existing asset that already has an Avro schema to make Parquet available in Destination. These forms cannot add a schema. Changing the asset encoding does not replace the node's Message Format selection.

Legacy panel

The legacy panel shows the asset Encoding and optional Schema as read-only details, with View details. There is no Message Format selector or Batching switch in this legacy panel; it enables batching.

Use this section only when the workflow opens the legacy editor. Asset-managed connection values can be read-only; see Managing Data Assets for available edits and the replacement-asset flow.

Field in the panel	Configuration key	Where to edit / when shown
Change	dataAssetId	Legacy asset picker when an asset is selected. With no selection, Select followed by the catalog type label appears when assets exist; No followed by that label and items available appears when the catalog is empty (fallback: Data Asset).
Batch Size (Records buffered)	batchSize	Batch Settings
Flush Interval (ms)	flushIntervalMillis	Batch Settings — Maximum time (ms) to wait before forcing a write. Required: No. Placeholder: 10,000.

Batching

Batching is enabled by default. When batching is true, records accumulate in memory and are flushed to S3 when either condition is met first:

• The number of accumulated records reaches the Batch Size (default: 10,000)
• The Flush Interval (ms) timer expires (default: 10,000 ms)

This reduces the number of write operations and keeps the node efficient for high-volume streams.

Set batching to false in the node configuration to write records individually. JSON_OBJECT requires this setting. PARQUET requires batching to remain enabled.

File Formats

In Destination, select Message Format on the node. The available choices are CSV, JSON Array, JSON Lines, and Parquet when the selected asset has an Avro schema. The current S3 asset creation and editing forms offer CSV, JSON Array, and JSON Lines as selectable Encoding Type values, with no Avro Schema input. Use an existing asset that already has a schema for Parquet; these forms cannot add one. In the legacy editor, the selected asset supplies the encoding and it is shown read-only.

The encoding formats and their behavior are:

• CSV — encodingType: "CSV" (.csv)
• JSON Object — encodingType: "JSON_OBJECT" (.json); requires batching: false and is not a Message Format choice in Destination
• JSON Array — encodingType: "JSON_ARRAY" (.json)
• JSON Lines — encodingType: "JSON_OBJECT_LINE" (.jsonl)
• Parquet — encodingType: "PARQUET" (.parquet); requires an Avro schema configured on the data asset and batching enabled. Omit compressionType or set it to null.

Output Path Structure

Batches are written using the following S3 key path:

{key}/year=YYYY/month=MM/day=DD/{uuid}.{ext}

Files are organized into date-partitioned folders using UTC timestamps. Each batch is written with a unique UUID filename.

The UUID is random and says nothing about the records inside the file, so file names cannot be used to find or remove duplicate records.

Write Failures

• Every upload error is retried. A batch gets up to 3 upload attempts, waiting 1 second and then 2 seconds between them.
• A batch that still fails gets one more round. Records that do not fit the file format are set aside as errors, and the rest are uploaded again under a new file name, with another 3 attempts.
• Records that fail both rounds are dropped. They are counted as errors and not uploaded again. Files uploaded before the failure, such as earlier parts of a multi-file Parquet batch, are not removed, so some of these records can still be in S3.

Because failed records are dropped, S3 delivery is neither exactly-once nor at-least-once. See Delivery Guarantees and Write Failures.

Usage Tips

• Ensure the AWS credentials in the asset's connection profile have appropriate permissions to write to the target S3 bucket
• Tune Batch Size and Flush Interval (ms) to balance write frequency against latency for your stream volume
• Parquet format requires an Avro schema on the data asset
• JSON Object format is incompatible with batching