Elasticsearch Sink Node
The Elasticsearch Sink node is not generally available in Fleak Data Workflow. The workflow builder offers it only to accounts where it has been enabled. For the connectors that are generally available, see Data Assets.
This page is retained for accounts where it is enabled.
The elasticsearchsink node writes workflow records to an Elasticsearch index using the Bulk API, batching records to reduce request overhead.
Quick Reference
In Destination, choose Asset Type and Asset Name, then open Configure Pipeline Settings. The fields listed here are the complete set of editable inputs for this Destination type, subject to the stated visibility conditions.
- Asset Type: Choose a Data Asset; UI selection determines the sink type on Save.
- Asset Name: Choose a Data Asset; reference to the selected data asset.
- Batch Size: Configure Pipeline Settings.
Destination has no inline Host, Index, or credential input. The Elasticsearch connection profile owns host/authentication; the data asset owns the target Index. Asset Information → Edit shows Index, but Save rejects a changed index: create a replacement asset for another index, select it in Asset Name, then Save. Edit host/authentication on the profile in Settings → Connection Profiles.
Configuration
Configuration keys below are relative to the node configuration. Use the field name in the panel when following instructions in the builder; required markers such as an asterisk are not part of the name.
Destination panel
Choose Asset Type and Asset Name first. Type-specific inputs appear in Configure Pipeline Settings after an asset is selected; an asset error can prevent that section from appearing. A missing or unavailable asset is a reference problem, not proof that the connector does not exist.
The fields listed here are the complete set of editable inputs for this Destination type, subject to the stated visibility conditions.
| Field in the panel | Configuration key | Where to edit / when shown |
|---|---|---|
| Asset Type | UI only; no separate config key | Choose a Data Asset; UI selection determines the sink type on Save |
| Asset Name | dataAssetId | Choose a Data Asset; reference to the selected data asset |
| Batch Size | batchSize | Configure Pipeline Settings; required to save Destination — Number of records buffered before the node sends a bulk request to Elasticsearch. Minimum: 1. Required: Yes. Placeholder: 500. |
Destination has no inline Host, Index, or credential input. The Elasticsearch connection profile owns host/authentication; the data asset owns the target Index. Asset Information → Edit shows Index, but Save rejects a changed index: create a replacement asset for another index, select it in Asset Name, then Save. Edit host/authentication on the profile in Settings → Connection Profiles.
Legacy panel
Use this section only when the workflow opens the legacy editor. Asset-managed connection values can be read-only; see Managing Data Assets for available edits and the replacement-asset flow.
| Field in the panel | Configuration key | Where to edit / when shown |
|---|---|---|
| Use Credentials | credentialId | Legacy panel; read-only when dataAssetId is set, because the asset/profile supplies this value — Select or create a Username/Password credential. The username and password are sent as HTTP Basic authentication. Leave empty for clusters that do not require authentication. API-key auth is not supported. Required: No. Placeholder: Select or create new (optional). |
| Host | host | Legacy panel; read-only when dataAssetId is set, because the asset/profile supplies this value — Base URL of the Elasticsearch cluster. Include the protocol and port. Required without a data asset; an asset-managed configuration can supply this value. Placeholder: https://elasticsearch.example.com:9200. |
| Index | index | Legacy panel; read-only when dataAssetId is set, because the asset/profile supplies this value — Name of the index where the records are written. The index must already exist or the cluster must be configured to create it on demand. Required without a data asset; an asset-managed configuration can supply this value. Placeholder: my-index. |
| Batch Size | batchSize | Legacy panel — Number of records buffered before the node sends a bulk request to Elasticsearch. Minimum: 1. Required: No. Placeholder: 500. |
Setting behavior
In Destination, choose an asset created for the target Index. Asset Information → Edit shows Index, but Save rejects a changed index: create a replacement asset, select it in Asset Name, and Save. Edit Host and authentication on its Elasticsearch profile in Settings → Connection Profiles. The inline Use Credentials, Host, and Index inputs below belong to the legacy panel and are read-only for an asset-managed node.
Use Credentials
Select an existing Username/Password credential from the dropdown or create a new one. The connector authenticates against Elasticsearch using HTTP Basic auth — API-key authentication is not supported. The user must have permission to write to the target index (create and index privileges).
Host
The base URL of your Elasticsearch cluster. Format: scheme://host[:port]. Include the protocol (http or https). Trailing slashes are stripped automatically.
Index
The destination Elasticsearch index. Each record in a batch is sent with an index action targeting this index, so the cluster assigns a document ID automatically.
Batch Size
Records accumulate in memory until the batch reaches this size, at which point the node serializes them as NDJSON and POSTs the batch to the /_bulk endpoint. Larger batches reduce request overhead; smaller batches lower memory usage and keep latency tighter.
Examples
Example: Stream Workflow Events into Elasticsearch
Use this node when downstream consumers query Elasticsearch and you want every record produced by the workflow indexed there.
- In Destination, choose the Elasticsearch Asset Type and the Asset Name for the target index, for example
workflow-events. - Set the index when creating the asset; create a replacement asset to target another index. Configure the cluster URL and write credential on its connection profile. In an unmanaged legacy node, use Use Credentials, Host, and Index instead.
- Increase Batch Size for higher throughput, or lower it if you want fresher data in Elasticsearch.
Example: Per-Environment Indexes
Configure separate Elasticsearch Sink nodes for different environments by choosing assets with different index names (for example events-staging and events-prod). In an unmanaged legacy node, the equivalent input is Index. The host, credential and batch size can stay the same.
Error Handling
The node uses the Elasticsearch Bulk API. When the cluster returns a partial-failure response, successfully indexed records are counted as success and only the failed records are routed to the configured error handling. A non-200 HTTP response from /_bulk causes the entire batch to be reported as failed.
Related Nodes
- Elasticsearch Source: Read documents from an Elasticsearch index
- Kafka Sink: Publish workflow records to a Kafka topic
- S3 Sink: Write workflow records to AWS S3