Splunk Indexer Node
The splunk_indexer node applies a Splunk props.conf and transforms.conf to records that are already whole events.
For each record it picks the matching stanzas from the event's host, source and sourcetype, then runs SEDCMD and
TRANSFORMS the way a Splunk indexer does at index time. Records sent to the nullQueue are dropped. The node does not
break events: event boundaries come from the source's decoder framing, for example text multiline framing with a
startPattern.
Key Features
- Your Splunk files: paste the
props.confandtransforms.conffrom your indexer or Heavy Forwarder - Splunk stanza matching:
[default],[sourcetype],[host::…]and[source::…]stanzas, matched as Splunk does - SEDCMD and TRANSFORMS: masking,
_rawrewrites,nullQueuedrops, and index, sourcetype and host rewrites - Configurable fields: read the event text and its metadata from any field, with fallback values
- Built-in sourcetypes: common sourcetypes such as
syslogwork without an uploaded stanza
Configuration
Every field is optional. An empty string is treated as absent.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
propsConf | String | No | — | The text of a Splunk props.conf. At most 256 KiB. |
transformsConf | String | No | — | The text of a Splunk transforms.conf. At most 256 KiB. |
rawField | String | No | __raw__ | Field holding the event text. The node reads it and writes the result back to it. |
hostField | String | No | host | Field holding the event's host. |
sourceField | String | No | source | Field holding the event's source. |
sourcetypeField | String | No | sourcetype | Field holding the event's sourcetype. |
indexField | String | No | index | Field holding the event's index. |
hostFallback | String | No | — | Host used when hostField is missing, empty or not a string. |
sourceFallback | String | No | — | Source used when sourceField is missing, empty or not a string. |
sourcetypeFallback | String | No | — | Sourcetype used when sourcetypeField is missing, empty or not a string. |
indexFallback | String | No | — | Index used when indexField is missing, empty or not a string. |
With no files, the node still applies the built-in sourcetype settings.
Data from a Splunk S2S endpoint
Records from a Splunk S2S push stream carry their metadata in these fields. Point the field settings at them:
| Setting | Field |
|---|---|
hostField | __md_kafka_header_splunk_host |
sourceField | __md_kafka_header_splunk_source |
sourcetypeField | __md_kafka_header_splunk_sourcetype |
indexField | __md_kafka_header_splunk_index |
Behavior
Per record
- The node reads the event text from
rawField, and host, source, sourcetype and index from their fields. A metadata field that is missing, empty or not a string takes its fallback value. - It picks the settings from the matching stanzas. A later stanza in this list overrides an earlier one for the same
key:
[default],[sourcetype],[host::…],[source::…]. Patterns use Splunk's...and*wildcards, and host patterns are case-insensitive. - It applies every
SEDCMD-*, then everyTRANSFORMS-*in class order. - If a transform sends the event to the
nullQueue, the record is dropped. - Otherwise the text is written back to
rawField, and host, source, sourcetype and index are written to their configured fields, including any rewrites. A metadata field that holds a non-string value is kept unless a transform rewrites it.
A record whose rawField is missing or not a string goes to the error output with the message
field '<rawField>' is missing or not a string. See Error Handling.
Supported transforms
A TRANSFORMS-<class> = <name> entry in props.conf names a stanza in transforms.conf. The node supports:
DEST_KEY | Effect |
|---|---|
queue | FORMAT = nullQueue drops the event. FORMAT = indexQueue keeps it. |
_MetaData:Index | Sets the index to FORMAT. |
MetaData:Sourcetype | Sets the sourcetype. Write FORMAT = sourcetype::<value>. |
MetaData:Host | Sets the host. Write FORMAT = host::<value>. |
_raw | Replaces the event text with FORMAT. |
FORMAT can use $1, $2 and so on for the groups captured by REGEX.
Built-in sourcetypes
Fleak has built-in settings for these sourcetypes. They apply only where no uploaded stanza sets the same key.
| Sourcetype | Built-in settings |
|---|---|
syslog | One line per event. Host taken from the RFC 3164 header. |
access_combined | One line per event. |
access_common | One line per event. |
apache_error | One line per event. |
_json | Each object that starts a line is an event. |
csv | One row per event. |
log4j | A dated line starts an event; stack traces join it. |
This node receives whole events, so only the per-event parts take effect here. For example, a syslog event gets
its host from the syslog header. Line and event breaking comes from the source's decoder.
Validation errors
The files are checked when the node starts. An invalid file stops the node with an error that names the file and line, for example:
props.conf line 2: expected [stanza] or key = valueprops.conf must be at most 256 KiB
Settings not applied
Settings the node does not apply are reported as warnings, and the node still runs. For example:
props.conf [app_log] KV_MODE is not appliedprops.conf [app_log] LINE_BREAKER is not applied to whole events
Event-breaking settings (LINE_BREAKER, SHOULD_LINEMERGE, BREAK_ONLY_BEFORE, BREAK_ONLY_BEFORE_DATE,
MUST_BREAK_AFTER, MAX_EVENTS), TRUNCATE, TIME_PREFIX, TIME_FORMAT and MAX_TIMESTAMP_LOOKAHEAD are not
applied, because each record is already one event. Set the event boundaries in the source's decoding instead.
A SEDCMD whose replacement refers to a group its regex does not define is also listed and not applied.
These are not supported:
- Search-time settings, such as
EXTRACT,REPORTandFIELDALIAS - Timestamp extraction into a time field
CLONE_SOURCETYPE,INGEST_EVALandWRITE_META- Index-time field extraction
Examples
Drop DEBUG events from S3 logs
The source reads text objects from S3. Its multiline framing starts a new event at each line that begins with a
date, so a stack trace stays with its log line. The records have no sourcetype, so sourcetypeFallback sets it to
app_log. The [app_log] stanza sends every event that contains DEBUG to the nullQueue.
jobContext:
metricTags: {}
otherProperties: {}
dag:
- id: source
commandName: fssource
config:
backend: s3
root: s3://my-bucket/app-logs/
decoding:
format:
type: text
framing:
type: multiline
startPattern: '^\d{4}-\d{2}-\d{2}'
backendConfig:
region: us-east-1
outputs:
- indexer
- id: indexer
commandName: splunk_indexer
config:
sourcetypeFallback: app_log
propsConf: |
[app_log]
TRANSFORMS-drop = drop_debug
transformsConf: |
[drop_debug]
REGEX = DEBUG
DEST_KEY = queue
FORMAT = nullQueue
outputs:
- sink
- id: sink
commandName: stdout
config:
encodingType: JSON_OBJECT
If an object holds 10 events and 3 of them contain DEBUG, 7 records leave the node. Each has the event text in
__raw__ and sourcetype set to app_log.