Skip to main content

SQS Sink Node

Quick Reference​

In Destination, choose Asset Type and Asset Name, then open Configure Pipeline Settings. The fields listed here are the complete set of editable inputs for this Destination type, subject to the stated visibility conditions.

  • Asset Type: Choose a Data Asset; UI selection determines the sink type on Save.
  • Asset Name: Choose a Data Asset; reference to the selected data asset.
  • Batch Size: Configure Pipeline Settings.
  • Message Group ID Expression: Only when the selected queue asset is FIFO; required then.
  • Deduplication ID Expression: Only when the selected queue asset is FIFO.

Destination has no Message Encoding selector; its encoding value is fixed by the panel. The FIFO fields appear only when the selected asset location ends in .fifo. The asset owns Queue URL and its derived region/type; the AWS profile owns authentication. Asset Information → Edit shows Queue URL, but Save rejects a changed URL. Region can be edited; Queue Type remains derived from the saved URL. Create a replacement asset for another queue, select it in Asset Name, then Save.

Message Group ID Expression and Deduplication ID Expression accept record path expressions starting with $, not general FEEL expressions. When supplied, each path must select a string value. For a composed ID, compute it in Fleak Eval Query on an upstream path connected to this SQS sink, for every record type reaching it. A node whose outputs cannot reach this sink cannot supply its ID field. Retain the derived string field through the intervening nodes, then reference it here with a path such as $.dedup_id; the field must exist in the record that reaches the sink. A function call such as concat(...) belongs in the upstream Eval query, not directly in either SQS field.

The selected string values must also satisfy AWS limits: MessageGroupId and MessageDeduplicationId allow at most 128 characters, using alphanumeric characters and the punctuation listed in the AWS request reference. Spaces and other whitespace are not allowed. These limits apply to the resulting ID, not to the path expression. Fleak forwards the selected string unchanged; when composing an ID upstream, preserve the intended event identity and check every contributing value against those limits.

Destination Save allows Deduplication ID Expression to be blank. For a FIFO queue, omitting the deduplication ID when content-based deduplication is disabled causes an AWS send error. Reusing an ID instead lets SQS accept the message but suppress repeat delivery within its five-minute deduplication window. A successful send does not prove another copy was delivered. A deduplication ID represents the identity of a logical event: retries of that event should keep the same ID, while distinct events should receive different IDs unless they are intentionally treated as duplicates. Concatenating grouping or classification fields does not guarantee uniqueness; different events can share all those values. Prefer an existing stable event identifier. If none is available, define the event identity before deriving an ID, or use content-based deduplication when identical message bodies should count as duplicates. Do not use now(), current timestamps, random values, counters, or other generated-at-send values as a deduplication ID: they can change on retry and do not identify the logical event. Do not present a time-based expression as a stable unique ID. See AWS FIFO deduplication.

Configuration​

Configuration keys below are relative to the node configuration. Use the field name in the panel when following instructions in the builder; required markers such as an asterisk are not part of the name.

Destination panel​

Choose Asset Type and Asset Name first. Type-specific inputs appear in Configure Pipeline Settings after an asset is selected; an asset error can prevent that section from appearing. A missing or unavailable asset is a reference problem, not proof that the connector does not exist.

The fields listed here are the complete set of editable inputs for this Destination type, subject to the stated visibility conditions.

Field in the panelConfiguration keyWhere to edit / when shown
Asset TypeUI only; no separate config keyChoose a Data Asset; UI selection determines the sink type on Save
Asset NamedataAssetIdChoose a Data Asset; reference to the selected data asset
Batch SizebatchSizeConfigure Pipeline Settings; required to save Destination — How many records are buffered before a send. Required: Yes.
Message Group ID ExpressionmessageGroupIdExpressionOnly when the selected queue asset is FIFO; required to save Destination then — For FIFO queues only. A path expression selecting the string used as the MessageGroupId. Records with the same group ID are processed in order. Required: Yes when the selected queue asset is FIFO.
Deduplication ID ExpressiondeduplicationIdExpressionOnly when the selected queue asset is FIFO — For FIFO queues only. A path expression selecting the string deduplication ID. Optional in Destination Save; AWS requires an ID unless the queue has content-based deduplication. See FIFO send behavior below. Required: No.

Destination has no Message Encoding selector; its encoding value is fixed by the panel. The FIFO fields appear only when the selected asset location ends in .fifo. The asset owns Queue URL and its derived region/type; the AWS profile owns authentication. Asset Information → Edit shows Queue URL, but Save rejects a changed URL. Region can be edited; Queue Type remains derived from the saved URL. Create a replacement asset for another queue, select it in Asset Name, then Save.

Message Group ID Expression and Deduplication ID Expression accept record path expressions starting with $, not general FEEL expressions. When supplied, each path must select a string value. For a composed ID, compute it in Fleak Eval Query on an upstream path connected to this SQS sink, for every record type reaching it. A node whose outputs cannot reach this sink cannot supply its ID field. Retain the derived string field through the intervening nodes, then reference it here with a path such as $.dedup_id; the field must exist in the record that reaches the sink. A function call such as concat(...) belongs in the upstream Eval query, not directly in either SQS field.

The selected string values must also satisfy AWS limits: MessageGroupId and MessageDeduplicationId allow at most 128 characters, using alphanumeric characters and the punctuation listed in the AWS request reference. Spaces and other whitespace are not allowed. These limits apply to the resulting ID, not to the path expression. Fleak forwards the selected string unchanged; when composing an ID upstream, preserve the intended event identity and check every contributing value against those limits.

Destination Save allows Deduplication ID Expression to be blank. For a FIFO queue, omitting the deduplication ID when content-based deduplication is disabled causes an AWS send error. Reusing an ID instead lets SQS accept the message but suppress repeat delivery within its five-minute deduplication window. A successful send does not prove another copy was delivered. A deduplication ID represents the identity of a logical event: retries of that event should keep the same ID, while distinct events should receive different IDs unless they are intentionally treated as duplicates. Concatenating grouping or classification fields does not guarantee uniqueness; different events can share all those values. Prefer an existing stable event identifier. If none is available, define the event identity before deriving an ID, or use content-based deduplication when identical message bodies should count as duplicates. Do not use now(), current timestamps, random values, counters, or other generated-at-send values as a deduplication ID: they can change on retry and do not identify the logical event. Do not present a time-based expression as a stable unique ID. See AWS FIFO deduplication.

Legacy panel​

In this legacy panel, a Queue URL ending in .fifo determines the FIFO label; both expression inputs remain present for a standard queue.

Use this section only when the workflow opens the legacy editor. Asset-managed connection values can be read-only; see Managing Data Assets for available edits and the replacement-asset flow.

Field in the panelConfiguration keyWhere to edit / when shown
AWS SQS IntegrationintegrationIdLegacy panel; read-only when dataAssetId is set, because the asset/profile supplies this value
AWS RegionregionStrLegacy panel; read-only when dataAssetId is set, because the asset/profile supplies this value
Queue URLqueueUrlLegacy panel; read-only when dataAssetId is set, because the asset/profile supplies this value
Message EncodingencodingTypeLegacy panel — Defines how each record is serialized into the message body. Options: CSV, JSON Object, JSON Array, JSON Lines. More info here. Required without a data asset; an asset-managed configuration can supply this value.
Message Group ID ExpressionmessageGroupIdExpressionLegacy FIFO queue; for other queue URLs the label is Message Group ID Expression (FIFO only) — For FIFO queues only. An expression returning the string used as the MessageGroupId. Records with the same group ID are processed in order. Required by runtime validation for FIFO queues, even though legacy Save does not enforce this condition.
Deduplication ID ExpressiondeduplicationIdExpressionLegacy panel; relevant to FIFO queues — For FIFO queues only. An expression returning a unique string per message to prevent duplicates. Only needed if content-based deduplication is not enabled on the queue. Required: No.
Batch SizebatchSizeLegacy panel — How many records are buffered before a send. Required: No.

Overview​

The SQS Sink node sends records from your workflow as messages to an Amazon SQS queue. Each record in the pipeline becomes one SQS message. Messages are sent in batches for efficiency.

Both standard and FIFO queues are supported.

The queue itself is defined by an AWS SQS Queue data asset, which supplies the queue URL and region, and links the AWS SQS connection profile that holds the credentials. The node is left with how to format and batch the messages.

Connection Settings Are Read-Only​

In the legacy editor, when the node is managed by a data asset, Fleak shows:

This node is managed by a data asset. Connection settings are read-only here; change them on the data asset, or edit the node in the Destination panel.

The integration, AWS region, and queue URL are displayed but not editable in this legacy panel. Destination uses Asset Type and Asset Name instead; it has no inline region or queue URL input. The asset stores the queue URL and derived region; its AWS profile holds authentication. Asset Information → Edit shows Queue URL, but Save rejects a changed URL. Region can be edited; Queue Type remains derived from the saved URL. Create a replacement asset for another queue, select it in Asset Name, and Save; edit credentials on the profile in Settings → Connection Profiles. Re-deploy affected workflows for the change to take effect.

FIFO Queues​

If your queue name ends in .fifo, it is a FIFO queue. You must set a Message Group ID Expression to control message ordering. Without content-based deduplication, supply a Deduplication ID Expression that resolves to a valid string ID for each record.

Message Group ID Expression and Deduplication ID Expression accept record path expressions starting with $, not general FEEL expressions. When supplied, each path must select a string value. For a composed ID, compute it in Fleak Eval Query on an upstream path connected to this SQS sink, for every record type reaching it. A node whose outputs cannot reach this sink cannot supply its ID field. Retain the derived string field through the intervening nodes, then reference it here with a path such as $.dedup_id; the field must exist in the record that reaches the sink. A function call such as concat(...) belongs in the upstream Eval query, not directly in either SQS field.

The selected string values must also satisfy AWS limits: MessageGroupId and MessageDeduplicationId allow at most 128 characters, using alphanumeric characters and the punctuation listed in the AWS request reference. Spaces and other whitespace are not allowed. These limits apply to the resulting ID, not to the path expression. Fleak forwards the selected string unchanged; when composing an ID upstream, preserve the intended event identity and check every contributing value against those limits.

Destination Save allows Deduplication ID Expression to be blank. For a FIFO queue, omitting the deduplication ID when content-based deduplication is disabled causes an AWS send error. Reusing an ID instead lets SQS accept the message but suppress repeat delivery within its five-minute deduplication window. A successful send does not prove another copy was delivered. A deduplication ID represents the identity of a logical event: retries of that event should keep the same ID, while distinct events should receive different IDs unless they are intentionally treated as duplicates. Concatenating grouping or classification fields does not guarantee uniqueness; different events can share all those values. Prefer an existing stable event identifier. If none is available, define the event identity before deriving an ID, or use content-based deduplication when identical message bodies should count as duplicates. Do not use now(), current timestamps, random values, counters, or other generated-at-send values as a deduplication ID: they can change on retry and do not identify the logical event. Do not present a time-based expression as a stable unique ID. See AWS FIFO deduplication.

For standard queues, leave both FIFO fields empty.