Create, Test, and Deploy a Workflow

End-to-end guide to creating a workflow in Fleak's DAG builder, testing it with sample data, publishing a version, and deploying it to a compute cluster.

Overview

A workflow on Fleak goes through a series of stages before it processes production data:

1. Create a new workflow and add sample test data
2. Build a DAG by adding and connecting processing and sink nodes
3. Test the workflow by running nodes against the sample data
4. Publish a version to snapshot the current DAG configuration
5. Deploy the workflow to a compute cluster; deploying creates a version from the current draft

This tutorial walks through each stage.

Prerequisites

Before you begin, make sure you have:

• A Fleak account with access to the Workflow Builder
• At least one compute cluster available (you will see the list of available clusters when deploying a workflow)
• Any required connection profiles and data assets already configured, if your workflow reads from or writes to external systems

Step 1: Create a Workflow and Add Sample Data

Create the Workflow

1. Navigate to the Workflows page from the sidebar.
2. Click New Workflow to open the Create workflow dialog.
3. Enter a Name for the workflow and click Create. The name must be unique.

Add Sample Test Data

A new workflow starts with a Test Data node on the canvas that shows No test data. Import sample data through this node. The data is used to validate the workflow as you build and helps you create transformation logic.

1. Click the Test Data node to open the Test data panel.
2. Under Select a Data Source, select Plain text and paste or type your sample data into the editor. To sample data directly from Kafka instead, select a different option under Select a Data Source: Import from Apache Kafka Topic, or Import from Data Asset and choose a Kafka data asset, depending on which your workspace shows.
3. Select an Encoding Type that matches your data format (e.g., String Line, JSON Array, JSON Object, CSV, XML, Text).
4. Click Next to preview the parsed records before importing
5. Review the parsed output records on the Preview tab and click Import as Test Data

To import a raw log line, such as a firewall log line, click the Test Data node (open the Configuration tab if the panel opens on Preview), select Plain text under Select a Data Source, paste the log line into the editor, select String Line (STRING_LINE) as the Encoding Type, click Next, and then click Import as Test Data. Each line becomes one record, with the full line stored in the __raw__ field, ready for a Parser node.

After importing, the Test Data node shows the encoding type and record count. This node serves as the starting point for your workflow. Test Data is a canvas input separate from the persisted processing nodes: a builder sample run does not require adding a standalone source node when Test Data is connected. Production source configuration is handled separately during deployment.

Step 2: Build the DAG

The DAG builder is a visual canvas where you assemble your data processing flow, starting from the Test Data node.

Add Nodes

Click the node selector to browse available node types:

• Processing nodes — transform and enrich data (e.g., Parser, Filter, Fleak Eval, SQL, Data Enrichment, PII Masking)
• Destination — where processed data is written; its Asset Type selects the platform (e.g., S3, Kafka, Splunk HEC, Datadog Logs, Databricks). See Destination

See the node reference docs for details on each node type.

Configure Nodes

Select a node on the canvas to open its configuration panel on the right sidebar. Each node type has its own settings — for example, a Parser node requires a parsing format and extraction rules, while a Filter node requires a filter expression.

Where Rename node is available, select the node and click the pencil button Rename node next to its title in the panel header. In Edit Node Name, change Name and click Update. Clicking the title text itself does not open the rename dialog.

Connect Nodes

Nodes are connected in sequence to form the DAG. Each upstream node's output becomes the input of its downstream node. The workflow executes nodes in topological order from the Test Data node through to the sink.

tip

Your work is auto-saved as a draft. You can close the browser and return later to pick up where you left off.

You can edit the workflow's name and description at any time from the editor.

Step 3: Test the Workflow

With sample data already configured, you can test your workflow at any point during the build process.

Run Tests

You can test in two ways:

• Run a single node — click the run button on a specific node to execute the DAG up to and including that node. This is useful for debugging intermediate steps.
• Run the entire workflow — click the Run button to open its menu: Run all with test data executes the full DAG end to end, Run all with one-time test data runs it with sample data used for that run only, and View results shows the results of the last run. The menu also shows View last inspection when a previous inspection exists, inspection is enabled for your account, and the workflow is open for editing.

Each test run accepts up to 100 input events. If you paste more, only the first 100 are kept and a warning shows how many were dropped.

View Results

Results appear in the Run Results panel with two view modes:

• JSON — raw JSON output
• Table — tabular view of output records

Any errors encountered during execution are shown per node.

You can inspect output at each step of the DAG to trace how data is transformed through the workflow.

Step 4: Publish a Version

Once you are satisfied with the test results, publish a version to snapshot the current DAG configuration.

1. Click the arrow on the Deploy button to open the dropdown menu, then select Version history.

2. The Version history panel opens on the right, showing your current edit and any previously published versions.

3. Click the + button at the top of the panel to add a new version.
4. In the Add to version history dialog, enter a Description summarizing what changed, then click Add.

The version is saved with a unique commit hash. You can view the version history, switch between versions, or clone an older version to create a new edit. You don't have to publish a version before deploying: deploying creates a version from the current draft.

Step 5: Deploy the Workflow

Click the Deploy button to open the deployment wizard. The wizard has two steps, followed by a confirmation:

5.1 Choose Production Data Source

Choose the production data source for the deployment. Pick an Asset Type, then the Asset Name of an existing data asset, or click Create New Asset, and click Select & Continue. The generally available source asset types are:

• Apache Kafka Topic — consume from a Kafka topic
• AWS MSK Kafka Topic — consume from an Amazon MSK topic
• AWS SQS Queue — read messages from an Amazon SQS queue
• S3 Event Notifications (SQS) — read S3 objects as their event notifications arrive
• AWS S3 Folder — read files from an S3 folder on a schedule

Accounts with additional sources enabled also see those asset types here.

The broker, topic and other connection details come from the selected data asset and its connection profile. For some asset types, a Configure Pipeline Settings section in this step holds the settings for this workflow only. For example, for a Kafka topic you set Read from (End (latest) or Beginning (earliest)), Message Format and Consumer Group; for an SQS queue, Message Format and Batch Size.

5.2 Configure Deployment

• Available Clusters — select the compute cluster to run the workflow on
• Replicas — set the number of parallel replicas (default 1, maximum 10)
• Frequency and Cron Expression (batch sources only, such as AWS S3 Folder) — choose when the workflow runs: Run once, Hourly, Daily at 12am, Sundays at 12am, Monthly on the 1st, 12am, or Custom (Cron), or enter a cron expression. Run once hides Time Zone and Cron Expression. Time Zone is fixed to UTC; schedules run in UTC. Streaming workflows run continuously as soon as deployed and do not require a schedule.

Click Deploy on Fleak to submit.

5.3 Deployment Confirmation

After a successful deployment, the Workflow Deployed! dialog shows the Cluster, Replicas, Frequency and First Run (for a cron schedule, the next scheduled run, or No upcoming run if the schedule never fires; otherwise the time of deployment), with the deployed version in the footer.

Any previously active deployment for this workflow is automatically stopped and replaced.