S3 Event Source Node
Quick Reference
Use Credentials (Optional)
AWS credentials used for both SQS and S3. Leave blank to use the IAM role or the AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables.
AWS Region
The AWS region where your SQS queue is located.
ex: us-east-1
Queue URL
The full URL of the SQS queue that receives the S3 event notifications.
ex: https://sqs.us-east-1.amazonaws.com/123456789012/my-queue
Object Encoding The format of the S3 objects being read.
Compression
Compression of the referenced objects. Auto-detect handles both plain and gzipped objects; choose Gzip to force decompression.
Advanced settings — Max Messages Per Poll, Wait Time, Visibility Timeout, Max Object Size, Max Retries, and Add S3 Metadata.
Overview
The S3 Event Source node streams objects from Amazon S3 into your workflow as they arrive. Rather than listing a bucket, it long-polls an Amazon SQS queue that receives S3 event notifications, downloads each newly created object, and passes its contents downstream. Messages are removed from the queue after the object is successfully processed.
Only object-creation events are processed. To use this node you must first configure S3 to publish ObjectCreated notifications to an SQS queue (optionally through an SNS topic).
Configuration
| Field | Description | Required | Default |
|---|---|---|---|
| Use Credentials | Select or create an AWS Access Key credential used for both SQS and S3. If left blank, the default AWS credential chain is used (IAM role, environment variables, etc.). | No | — |
| AWS Region | The AWS region where your SQS queue is hosted (e.g. us-east-1). | Yes | — |
| Queue URL | The full URL of the SQS queue that receives the S3 event notifications. | Yes | — |
| Object Encoding | How each S3 object's content should be interpreted. Options: CSV, JSON Object, JSON Array, JSON Lines, String Line, Text, XML. More info here. | Yes | — |
| Compression | Object compression. Auto-detect handles both plain and gzipped objects; Gzip forces decompression. | No | Auto-detect |
| Max Messages Per Poll | How many SQS messages to fetch in each request. Maximum allowed by SQS is 10. | No | 10 |
| Wait Time (seconds) | Seconds to wait for new messages before returning (long polling). Maximum is 20 seconds. | No | 20 |
| Visibility Timeout (seconds) | Seconds a received message stays hidden while its object is downloaded, decompressed, and parsed. Set it above the worst-case processing time to avoid duplicate emits. | No | 300 |
| Max Object Size (bytes) | Objects larger than this are dead-lettered without being downloaded. | No | 268435456 (256 MiB) |
| Max Retries | Messages whose SQS receive count exceeds this are dead-lettered instead of retried. | No | 5 |
| Add S3 Metadata | When enabled, enriches each record with its source __s3_bucket and __s3_key. | No | Off |
Related Nodes
- S3 Sink: Write processed records to Amazon S3
- SQS Source: Read raw messages directly from an SQS queue