Skip to main content

S3 Output Node

The S3 Output Node enables users to store workflow output data in Amazon Simple Storage Service (S3) buckets. This node provides a straightforward way to write data from your Fleak workflow directly to S3, making it ideal for data archival, backup purposes, or integration with other AWS services.

S3 Sink Demo Image

Configurations​

  • AWS Connection: Select or create a new AWS connection by
    • Giving a name to the connection
    • Inputting an AWS key
    • Inputting an AWS secret
    • Click "Create"
  • AWS Region: Select the AWS region where your S3 bucket is located. The region selection ensures your data is stored in the appropriate geographical location, which can affect latency, compliance, and costs. Available regions include:
    • us-east-1
    • us-east-2
    • us-west-1
    • us-west-2
  • Bucket Name: Specify the name of the S3 bucket where you want to store your data. The bucket name must be
    • Globally unique across all AWS accounts
    • Between 3 and 63 characters long
    • Containing only lowercase letters, numbers, periods, and hyphens
  • Key Name: Enter the object key (path) where you want to save your data within the bucket. This can include:
    • Folder structures using forward slashes (/)
    • File naming patterns
    • Dynamic paths using variables
    • Example: my_dir/my_subdir
  • File Format: Choose the format in which you want to save your data. The file format determines how your data will be structured and stored in S3. The options included are:
    • JSON Object
    • JSON Object Line
    • CSV

Usage Tips​

  • Ensure your AWS credentials have appropriate permissions to write to the specified S3 bucket
  • Use meaningful key names that reflect your data organization strategy
  • Consider implementing a consistent naming convention for your S3 objects
  • Be mindful of S3 storage costs and data transfer fees
  • Files are created with default S3 permissions based on bucket settings
  • Existing files with the same key will be overwritten
  • The node supports automatic file format conversion based on your selection
  • Large datasets are automatically split into manageable chunks if necessary