Skip to main content

Elasticsearch Source Node

Quick Reference​

Use Credentials Select a Username/Password credential for Elasticsearch. Optional — leave empty for an unauthenticated cluster.

Host Base URL of the Elasticsearch cluster. ex: https://es.example.com:9200

Index The Elasticsearch index (or index pattern) to read from. ex: logs-2026-04-*

Query Optional Elasticsearch query DSL fragment. When blank the node performs a match_all query. ex: {"match": {"level": "ERROR"}}

Scroll Timeout How long Elasticsearch keeps the scroll context alive between fetches. Defaults to 5m.

Batch Size Number of documents requested per scroll page. Defaults to 500.

Encoding Type Format used to decode each document's _source payload into a workflow record. Typically JSON_OBJECT.

The elasticsearchsource node queries an Elasticsearch index using the scroll API and emits each matching document as a workflow record.

Overview​

The Elasticsearch Source connector lets you pull documents stored in Elasticsearch into your workflow. It functions by submitting a search request against the configured index — using the optional query you provide or match_all — and paging through the results using the Elasticsearch scroll API.

This source is designed for batch ingestion. When the workflow runs, the connector authenticates with the cluster, opens a scroll context, and emits each document's _source payload as a workflow record until the result set is exhausted.

Prerequisites​

Before configuring the source, ensure you have:

  • Network access from the workflow runtime to the Elasticsearch HTTP endpoint.
  • Read permission on the target index for the credential. The user must be allowed to perform _search and _search/scroll requests against the index.

Configuration​

FieldDescriptionRequiredPlaceholder
Use CredentialsSelect or create a Username/Password credential. The username and password are sent as HTTP Basic authentication. Leave empty for clusters that do not require authentication.Noes-reader
HostBase URL of the Elasticsearch cluster. Include the protocol and port.Yeshttps://es.example.com:9200
IndexName of the index or index pattern to read from.Yeslogs-2026-04-*
QueryElasticsearch query DSL fragment that goes under the query key of the search request. When blank the node performs a match_all query.No{"match": {"level": "ERROR"}}
Scroll TimeoutDuration the scroll context is kept alive between page fetches. Use Elasticsearch duration format.No5m
Batch SizeNumber of documents requested per scroll page (size parameter on the search request). Minimum: 1.No500
Encoding TypeFormat used to deserialize each document's _source JSON into a workflow record. Typically JSON_OBJECT.YesJSON_OBJECT

Use Credentials​

Select an existing Username/Password credential from the dropdown or create a new one. The connector authenticates against Elasticsearch using HTTP Basic auth — API-key authentication is not supported. Make sure the user has permission to issue _search and _search/scroll requests on the target index.

Host​

The base URL of your Elasticsearch cluster. Format: scheme://host[:port]. Include the protocol (http or https). Trailing slashes are stripped automatically.

Index​

The Elasticsearch index name. Wildcards (logs-2026-*) and comma-separated lists (orders,customers) are passed through to Elasticsearch as written.

Query​

Provide an Elasticsearch query DSL fragment as a JSON string — the node embeds it under the query key of the search request. Examples:

  • {"match_all": {}} — explicit match-all (same as leaving the field blank).
  • {"range": {"@timestamp": {"gte": "now-1d"}}} — last 24 hours.
  • {"bool": {"must": [{"term": {"status": "open"}}]}} — boolean filter.

Scroll Timeout​

Controls how long Elasticsearch keeps the scroll context alive between fetches. Use Elasticsearch duration syntax such as 30s, 1m, 5m. Larger values are safer for slow downstream processing but consume more cluster memory.

Batch Size​

Number of hits returned per scroll page. The connector keeps fetching pages until no more hits are returned. Smaller batches lower per-request memory pressure on Elasticsearch; larger batches reduce round trips.

Encoding Type​

Each document's _source field is serialized to JSON and then deserialized using the selected encoding type. In almost all cases this should be JSON_OBJECT.

Examples​

Example: Reprocess Recent Error Logs​

Use this node when you want to feed a subset of Elasticsearch documents back into the workflow — for example, to re-evaluate error logs against a new enrichment rule.

  • Select the credential authorized to read the logs index.
  • Enter the cluster URL in Host.
  • Set Index to the index pattern that contains the logs (for example logs-2026-04-*).
  • In Query, restrict to errors: {"match": {"level": "ERROR"}}.
  • Leave Scroll Timeout and Batch Size at their defaults unless the cluster is under heavy load.

Example: Full Index Snapshot​

Leave Query blank to read every document in the index. The connector defaults to match_all and emits each document until the index is fully drained.

Document Metadata​

Each emitted record carries the original Elasticsearch document's _id and _index as event metadata, so downstream nodes can route or trace records back to their source document.