File Source Node
The filesource node reads one local file, deserializes its whole content with the configured encoding, and emits the
resulting records. It is a batch source: it reads the file once and then ends. Internal ZephFlow engine command for
local and CLI runs. Not offered in Fleak Data Workflow and not supported in Fleak Data Workflow workflows; do not add it
to a workflow.
Internal ZephFlow engine command for local and CLI runs. Not offered in Fleak Data Workflow and not supported in Fleak Data Workflow workflows; do not add it to a workflow.
Key Features
- Single file: Reads exactly one regular file from the local filesystem
- Whole-file parsing: The entire file is deserialized as one payload with the configured
encodingType - Batch source: Reads the file once, emits its records, and ends
Config Object
The full config object for the file source node (FileSourceDto.Config):
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
filePath | String | Yes | — | Path of the file to read. Must exist and be a regular file; directories and glob patterns are rejected |
encodingType | EncodingType | Yes | — | Format of the file. One of CSV, JSON_OBJECT, JSON_ARRAY, JSON_OBJECT_LINE, STRING_LINE, TEXT, XML. PARQUET is rejected |
The path is checked when the DAG is compiled: a missing file fails with file doesn't exist: <path>, and a directory
with file is not a regular file: <path>. Enum values are case-sensitive: json_object_line fails DAG compilation.
See Encoding Types for how each encoding maps the file content to records.
How the File Is Read
Paths and Docker
A relative filePath resolves against the working directory of the ZephFlow process. In the
fleak/zephflow-clistarter Docker image the working directory is /app, so either mount the file into /app and use a
relative path, or mount it elsewhere and use an absolute path. A file on the host is not visible to the container
unless it is mounted.
One Payload per File
The whole file is read into memory and deserialized as a single payload. If any part of it fails to parse, for example
one malformed line in a JSON_OBJECT_LINE file, the whole file fails and none of its records are emitted. The failure
is logged at ERROR level with the file content in base64. When jobContext.dlqConfig is set, the file content is
written to the DLQ with the node's ID in nodeId; otherwise it is dropped.
After the single read, the source ends and the pipeline finishes once the records have been processed.
To read many files, files in object storage, or files that should be resumed after a restart, use
fssource (the file backend reads from the local filesystem).
Job Context
The DAG definition must contain a jobContext block, which may be empty (jobContext: {}). Without it, the CLI starter
fails with a NullPointerException before the DAG runs.
Examples
DAG Definition
Create events.jsonl:
{"user": "alice", "status": 200}
{"user": "bob", "status": 500}
Save as filesource_example.yml:
jobContext: {}
dag:
- id: "file_source"
commandName: "filesource"
config:
filePath: "events.jsonl"
encodingType: "JSON_OBJECT_LINE"
outputs:
- "errors_only"
- id: "errors_only"
commandName: "filter"
config:
expression: $.status >= 500
outputs:
- "stdout_sink"
- id: "stdout_sink"
commandName: "stdout"
config:
encodingType: "JSON_OBJECT"
outputs: []
Run it with the CLI starter, mounting both files into /app:
docker run \
-v $(pwd)/filesource_example.yml:/app/filesource_example.yml \
-v $(pwd)/events.jsonl:/app/events.jsonl \
fleak/zephflow-clistarter:latest -f /app/filesource_example.yml
Between the log lines, the output contains:
{"user":"bob","status":500}