Skip to main content
Unlisted page
This page is unlisted. Search engines will not index it, and only users having a direct link can access it.

Elasticsearch Source Node

Not generally available

The Elasticsearch Source node is not generally available in Fleak Data Workflow. The workflow builder offers it only to accounts where it has been enabled. For the connectors that are generally available, see Data Assets.

This page is retained for accounts where it is enabled.

The elasticsearchsource node queries an Elasticsearch index using the scroll API and emits each matching document as a workflow record.

Quick Reference​

  • Connection Profile: Select a connection profile; endpoint and authentication belong to the profile.
  • Index: Name of the index or index pattern to read from.
  • Query: Elasticsearch query DSL fragment that goes under the query key of the search request. When blank the node performs a match_all query.
  • Scroll Timeout: Duration the scroll context is kept alive between page fetches. Use Elasticsearch duration format.
  • Batch Size: Number of documents requested per scroll page (size parameter on the search request). Minimum: 1.
  • Encoding Type: Format used to deserialize each document's _source JSON into a workflow record. Typically JSON_OBJECT.

Configuration​

Configuration keys below are relative to the node configuration. Use the field name in the panel when following instructions in the builder; required markers such as an asterisk are not part of the name.

Field in the panelConfiguration keyWhere to edit / when shown
Connection ProfileconnectionProfileIdSelect a connection profile; endpoint and authentication belong to the profile
IndexindexNode panel — Name of the index or index pattern to read from. Required: Yes. Placeholder: logs-2026-04-*.
QueryqueryNode panel — Elasticsearch query DSL fragment that goes under the query key of the search request. When blank the node performs a match_all query. Required: No. Placeholder: {"match": {"level": "ERROR"}}.
Scroll TimeoutscrollTimeoutNode panel — Duration the scroll context is kept alive between page fetches. Use Elasticsearch duration format. Required: No. Placeholder: 5m.
Batch SizebatchSizeNode panel — Number of documents requested per scroll page (size parameter on the search request). Minimum: 1. Required: No. Placeholder: 500.
Encoding TypeencodingTypeNode panel — Format used to deserialize each document's _source JSON into a workflow record. Typically JSON_OBJECT. Required: Yes. Placeholder: JSON_OBJECT.

Overview​

The Elasticsearch Source connector lets you pull documents stored in Elasticsearch into your workflow. It functions by submitting a search request against the configured index — using the optional query you provide or match_all — and paging through the results using the Elasticsearch scroll API.

This source is designed for batch ingestion. When the workflow runs, the connector authenticates with the cluster, opens a scroll context, and emits each document's _source payload as a workflow record until the result set is exhausted.

Prerequisites​

Before configuring the source, ensure you have:

  • Network access from the workflow runtime to the Elasticsearch HTTP endpoint.
  • Read permission on the target index for the credential. The user must be allowed to perform _search and _search/scroll requests against the index.

Connection profile authentication​

Select Connection Profile in the node. Configure the Username/Password credential on that profile. The connector authenticates against Elasticsearch using HTTP Basic auth — API-key authentication is not supported. Make sure the user has permission to issue _search and _search/scroll requests on the target index.

Connection profile host​

The host is configured on the connection profile, not in the node panel. It is the base URL of your Elasticsearch cluster. Format: scheme://host[:port]. Include the protocol (http or https). Trailing slashes are stripped automatically.

Index​

The Elasticsearch index name. Wildcards (logs-2026-*) and comma-separated lists (orders,customers) are passed through to Elasticsearch as written.

Query​

Provide an Elasticsearch query DSL fragment as a JSON string — the node embeds it under the query key of the search request. Examples:

  • {"match_all": {}} — explicit match-all (same as leaving the field blank).
  • {"range": {"@timestamp": {"gte": "now-1d"}}} — last 24 hours.
  • {"bool": {"must": [{"term": {"status": "open"}}]}} — boolean filter.

Scroll Timeout​

Controls how long Elasticsearch keeps the scroll context alive between fetches. Use Elasticsearch duration syntax such as 30s, 1m, 5m. Larger values are safer for slow downstream processing but consume more cluster memory.

Batch Size​

Number of hits returned per scroll page. The connector keeps fetching pages until no more hits are returned. Smaller batches lower per-request memory pressure on Elasticsearch; larger batches reduce round trips.

Encoding Type​

Each document's _source field is serialized to JSON and then deserialized using the selected encoding type. In almost all cases this should be JSON_OBJECT.

Examples​

Example: Reprocess Recent Error Logs​

Use this node when you want to feed a subset of Elasticsearch documents back into the workflow — for example, to re-evaluate error logs against a new enrichment rule.

  • Choose a Connection Profile with the cluster URL and a credential authorized to read the logs index.
  • Set Index to the index pattern that contains the logs (for example logs-2026-04-*).
  • In Query, restrict to errors: {"match": {"level": "ERROR"}}.
  • Leave Scroll Timeout and Batch Size at their defaults unless the cluster is under heavy load.

Example: Full Index Snapshot​

Leave Query blank to read every document in the index. The connector defaults to match_all and emits each document until the index is fully drained.

Document Metadata​

Each emitted record carries the original Elasticsearch document's _id and _index as event metadata, so downstream nodes can route or trace records back to their source document.