S3 Event Source Node
Quick Reference
- Data Asset: Select an SQS event-notification asset.
- S3 Connection Profile (Optional): Optional separate profile for reading S3 objects.
- S3 Endpoint Override (Optional): Optional. A non-default S3 endpoint, e.g.
https://s3.us-east-1.amazonaws.com. - Object Encoding: How each S3 object's content should be interpreted. Options: CSV, JSON Object, JSON Array, JSON Lines, String Line, Text, XML. More info here.
- Compression: Auto-detect (gzip) or GZIP (force).
- Max Messages Per Poll: Advanced Settings.
- Wait Time (seconds): Advanced Settings.
- Visibility Timeout (seconds): Advanced Settings.
- Max Object Size (bytes): Advanced Settings.
- Max Retries: Advanced Settings.
- Add S3 Metadata: Advanced Settings.
Configuration
Configuration keys below are relative to the node configuration. Use the field name in the panel when following instructions in the builder; required markers such as an asterisk are not part of the name.
| Field in the panel | Configuration key | Where to edit / when shown |
|---|---|---|
| Data Asset | dataAssetId | Select an SQS event-notification asset |
| S3 Connection Profile (Optional) | s3ConnectionProfileId | Optional separate profile for reading S3 objects — Optional. A separate AWS S3 connection profile for cross-account object reads. Leave blank to reuse the queue's credential. Required: No. |
| S3 Endpoint Override (Optional) | s3EndpointOverride | Node panel — Optional. A non-default S3 endpoint, e.g. https://s3.us-east-1.amazonaws.com. Required: No. |
| Object Encoding | encodingType | Node panel — How each S3 object's content should be interpreted. Options: CSV, JSON Object, JSON Array, JSON Lines, String Line, Text, XML. More info here. Required: Yes. |
| Compression | compressionType | Auto-detect (gzip) or GZIP (force) — Object compression. Auto-detect handles both plain and gzipped objects; Gzip forces decompression. Default: Auto-detect. Required: No. |
| Max Messages Per Poll | maxNumberOfMessages | Advanced Settings — How many SQS messages to fetch in each request. Maximum allowed by SQS is 10. Default: 10. Required: No. |
| Wait Time (seconds) | waitTimeSeconds | Advanced Settings — Seconds to wait for new messages before returning (long polling). Maximum is 20 seconds. Default: 20. Required: No. |
| Visibility Timeout (seconds) | visibilityTimeoutSeconds | Advanced Settings — Seconds a received message stays hidden while its object is downloaded, decompressed, and parsed. Set it above the worst-case processing time to avoid duplicate emits. Default: 300. Required: No. |
| Max Object Size (bytes) | maxObjectSizeBytes | Advanced Settings — Objects larger than this are dead-lettered without being downloaded. Default: 268435456 (256 MiB). Required: No. |
| Max Retries | maxRetries | Advanced Settings — Messages whose SQS receive count exceeds this are dead-lettered instead of retried. Default: 5. Required: No. |
| Add S3 Metadata | addS3Metadata | Advanced Settings — When enabled, enriches each record with its source __s3_bucket and __s3_key. Default: Off. Required: No. |
Overview
The S3 Event Source node streams objects from Amazon S3 into your workflow as they arrive. Rather than listing a bucket, it long-polls an Amazon SQS queue that receives S3 event notifications, downloads each newly created object, and passes its contents downstream. Messages are removed from the queue after the object is successfully processed.
Only object-creation events are processed. To use this node you must first configure S3 to publish ObjectCreated notifications to an SQS queue (optionally through an SNS topic).
Related Nodes
- S3 Sink: Write processed records to Amazon S3
- SQS Source: Read raw messages directly from an SQS queue
- Data Assets: Defining the notification queue
- Connection Profiles: AWS SQS and AWS S3 credentials