File System Source Node

note

Reading from Amazon S3 is generally available. The Google Cloud Storage and Azure Blob Storage backends are not; the workflow builder offers them only to accounts where they have been enabled.

Quick Reference

• Backend: Amazon S3, Google Cloud Storage, or Azure Blob Storage.
• Data Asset: The cloud folder to read, e.g. an AWS S3 Folder asset for Amazon S3. Its connection profile supplies the credentials.
• File Name Pattern (Optional): Regular expression matched against each file's name, not its full path. ex: .*\.log$
• Encoding Type: How each file's content is parsed. Options: CSV, JSON Object, JSON Array, JSON Lines, String Line, Text, XML. More info here.
• Max File Bytes (Optional): Size cap in decompressed bytes. Default 268435456 (256 MiB).
• Chunk Size Bytes (Optional): Read chunk size in decompressed bytes. Default 16777216 (16 MiB).
• Max Files Per Run (Optional): Files processed per run, oldest first. Default 10000.

Overview

The File System Source node reads files from a selected cloud folder in Amazon S3, Google Cloud Storage, or Azure Blob Storage, and passes each file's parsed records downstream. It is a batch source: each run lists the folder, reads the files it has not already processed, and then stops. Deploy it on a schedule to pick up new files as they land.

Workflow Builder fields

Configuration keys below are relative to the node configuration. Use the field name in the panel when following instructions in the builder; required markers such as an asterisk are not part of the name.

Field in the panel	Configuration key	Where to edit / when shown
Backend	backend	Amazon S3 (s3), Google Cloud Storage (gs), or Azure Blob Storage (azblob). Required
Data Asset	dataAssetId	After choosing a backend; select a cloud folder for that platform. Amazon S3 uses an AWS S3 Folder asset with an AWS connection profile. Required
Exact object	exactObjectKey	Read-only and shown only for a workflow pinned to a specific S3 object
File Name Pattern (Optional)	fileNameRegex	Node panel. Regular expression that must match the whole file name, e.g. .*\.log$
Encoding Type	encodingType	Node panel. CSV, JSON Object, JSON Array, JSON Lines, String Line, Text, or XML. Required
Max File Bytes (Optional)	maxFileBytes	Node panel. Default 268435456 (256 MiB)
Chunk Size Bytes (Optional)	chunkSizeBytes	Node panel. Default 16777216 (16 MiB)
Max Files Per Run (Optional)	maxFilesPerRun	Node panel. Default 10000

Changing Backend clears the selected asset and object pin. Choosing another Data Asset also clears the pin. Storage location and credentials belong to the asset and its connection profile. The DAG config fields root and backendConfig are not controls in this panel; they are filled in from the asset.

Reading from Amazon S3

1. Create an AWS connection profile in Settings → Connection Profiles, with an access key for an IAM identity that can read the bucket. See AWS S3 connection profiles.
2. Create an AWS S3 Folder data asset that points at the bucket and, optionally, a Folder / Prefix. An empty Folder / Prefix reads the whole bucket. See AWS S3 Folder.
3. Add a File System Source node, set Backend to Amazon S3, and select the asset in Data Asset.
4. Set Encoding Type to match the file contents, and optionally a File Name Pattern, such as .*\.jsonl(\.gz)? to skip other files in the folder.
5. Deploy on a schedule. In the deployment wizard, choose a Frequency such as Hourly, or a custom cron expression. Run once reads the folder a single time. See Deploy the Workflow.

The IAM identity behind the profile needs these permissions:

• s3:ListBucket on the bucket ARN, e.g. arn:aws:s3:::my-bucket, to list the folder.
• s3:GetObject on the objects, e.g. arn:aws:s3:::my-bucket/*, to read them.
• kms:Decrypt on the KMS key when the bucket uses SSE-KMS encryption.

The node only reads. It never writes, moves, or deletes objects in the bucket.

How runs pick up files

• Every file below the folder is a candidate, including files in nested folders. File Name Pattern is matched against the file name only.
• Each scheduled run reads only new files. Progress is saved per workflow, so files already read by an earlier run are not read again, and their records are not emitted twice.
• Files are read oldest first, by last-modified time. When more files are waiting than Max Files Per Run, the oldest are read and the rest wait for the next run.
• Late files with old timestamps are skipped. A file whose last-modified time is older than files an earlier run has already processed is treated as already passed and is not read.
• Gzipped files are handled automatically, whether or not their name ends in .gz.
• Replicas share the work. With more than one replica, each file is read by exactly one replica.

Choosing files and limits

Use File Name Pattern (Optional) to filter file names with a regular expression and Encoding Type to choose how file content is decoded. These inputs are in the main panel, not an Advanced Settings section.

• Max File Bytes (Optional) caps how much must be held in memory at once, in decompressed bytes. For JSON Object, Text, and XML it applies to the whole file. For JSON Lines and String Line it applies to a single line, and for CSV and JSON Array to a single record. A file that exceeds it is dropped and is not retried on later runs.
• Chunk Size Bytes (Optional) sets how much of a file is read and passed downstream at a time for line- and record-based formats.
• Max Files Per Run (Optional) limits the number of files one run reads.

A file that cannot be read, for example because of a permissions error, is retried on the next run. Records that fail to parse do not stop the file: the rest of it is still processed. When the deployment has a dead letter queue (with self-healing enabled), the failed records are written there with the file they came from; otherwise they are logged and dropped.

See the ZephFlow File System Source reference for runtime behavior and options that are only available in a DAG definition. The DAG examples there do not add controls to the builder panel.

Related Nodes

• S3 Event Source: Stream new S3 objects as they arrive, instead of on a schedule
• S3 Sink: Write processed records to Amazon S3