Skip to main content

Splunk Indexer Node

The splunk_indexer node applies a Splunk props.conf and transforms.conf to records that are already whole events. For each record it picks the matching stanzas from the event's host, source and sourcetype, then runs SEDCMD and TRANSFORMS the way a Splunk indexer does at index time. Records sent to the nullQueue are dropped. The node does not break events: event boundaries come from the source's decoder framing, for example text multiline framing with a startPattern.

Key Features​

  • Your Splunk files: paste the props.conf and transforms.conf from your indexer or Heavy Forwarder
  • Splunk stanza matching: [default], [sourcetype], [host::…] and [source::…] stanzas, matched as Splunk does
  • SEDCMD and TRANSFORMS: masking, _raw rewrites, nullQueue drops, and index, sourcetype and host rewrites
  • Configurable fields: read the event text and its metadata from any field, with fallback values
  • Built-in sourcetypes: common sourcetypes such as syslog work without an uploaded stanza

Configuration​

Every field is optional. An empty string is treated as absent.

FieldTypeRequiredDefaultDescription
propsConfStringNo—The text of a Splunk props.conf. At most 256 KiB.
transformsConfStringNo—The text of a Splunk transforms.conf. At most 256 KiB.
rawFieldStringNo__raw__Field holding the event text. The node reads it and writes the result back to it.
hostFieldStringNohostField holding the event's host.
sourceFieldStringNosourceField holding the event's source.
sourcetypeFieldStringNosourcetypeField holding the event's sourcetype.
indexFieldStringNoindexField holding the event's index.
hostFallbackStringNo—Host used when hostField is missing, empty or not a string.
sourceFallbackStringNo—Source used when sourceField is missing, empty or not a string.
sourcetypeFallbackStringNo—Sourcetype used when sourcetypeField is missing, empty or not a string.
indexFallbackStringNo—Index used when indexField is missing, empty or not a string.

With no files, the node still applies the built-in sourcetype settings.

Data from a Splunk S2S endpoint​

Records from a Splunk S2S push stream carry their metadata in these fields. Point the field settings at them:

SettingField
hostField__md_kafka_header_splunk_host
sourceField__md_kafka_header_splunk_source
sourcetypeField__md_kafka_header_splunk_sourcetype
indexField__md_kafka_header_splunk_index

Behavior​

Per record​

  1. The node reads the event text from rawField, and host, source, sourcetype and index from their fields. A metadata field that is missing, empty or not a string takes its fallback value.
  2. It picks the settings from the matching stanzas. A later stanza in this list overrides an earlier one for the same key: [default], [sourcetype], [host::…], [source::…]. Patterns use Splunk's ... and * wildcards, and host patterns are case-insensitive.
  3. It applies every SEDCMD-*, then every TRANSFORMS-* in class order.
  4. If a transform sends the event to the nullQueue, the record is dropped.
  5. Otherwise the text is written back to rawField, and host, source, sourcetype and index are written to their configured fields, including any rewrites. A metadata field that holds a non-string value is kept unless a transform rewrites it.

A record whose rawField is missing or not a string goes to the error output with the message field '<rawField>' is missing or not a string. See Error Handling.

Supported transforms​

A TRANSFORMS-<class> = <name> entry in props.conf names a stanza in transforms.conf. The node supports:

DEST_KEYEffect
queueFORMAT = nullQueue drops the event. FORMAT = indexQueue keeps it.
_MetaData:IndexSets the index to FORMAT.
MetaData:SourcetypeSets the sourcetype. Write FORMAT = sourcetype::<value>.
MetaData:HostSets the host. Write FORMAT = host::<value>.
_rawReplaces the event text with FORMAT.

FORMAT can use $1, $2 and so on for the groups captured by REGEX.

Built-in sourcetypes​

Fleak has built-in settings for these sourcetypes. They apply only where no uploaded stanza sets the same key.

SourcetypeBuilt-in settings
syslogOne line per event. Host taken from the RFC 3164 header.
access_combinedOne line per event.
access_commonOne line per event.
apache_errorOne line per event.
_jsonEach object that starts a line is an event.
csvOne row per event.
log4jA dated line starts an event; stack traces join it.

This node receives whole events, so only the per-event parts take effect here. For example, a syslog event gets its host from the syslog header. Line and event breaking comes from the source's decoder.

Validation errors​

The files are checked when the node starts. An invalid file stops the node with an error that names the file and line, for example:

  • props.conf line 2: expected [stanza] or key = value
  • props.conf must be at most 256 KiB

Settings not applied​

Settings the node does not apply are reported as warnings, and the node still runs. For example:

  • props.conf [app_log] KV_MODE is not applied
  • props.conf [app_log] LINE_BREAKER is not applied to whole events

Event-breaking settings (LINE_BREAKER, SHOULD_LINEMERGE, BREAK_ONLY_BEFORE, BREAK_ONLY_BEFORE_DATE, MUST_BREAK_AFTER, MAX_EVENTS), TRUNCATE, TIME_PREFIX, TIME_FORMAT and MAX_TIMESTAMP_LOOKAHEAD are not applied, because each record is already one event. Set the event boundaries in the source's decoding instead.

A SEDCMD whose replacement refers to a group its regex does not define is also listed and not applied.

These are not supported:

  • Search-time settings, such as EXTRACT, REPORT and FIELDALIAS
  • Timestamp extraction into a time field
  • CLONE_SOURCETYPE, INGEST_EVAL and WRITE_META
  • Index-time field extraction

Examples​

Drop DEBUG events from S3 logs​

The source reads text objects from S3. Its multiline framing starts a new event at each line that begins with a date, so a stack trace stays with its log line. The records have no sourcetype, so sourcetypeFallback sets it to app_log. The [app_log] stanza sends every event that contains DEBUG to the nullQueue.

jobContext:
metricTags: {}
otherProperties: {}

dag:
- id: source
commandName: fssource
config:
backend: s3
root: s3://my-bucket/app-logs/
decoding:
format:
type: text
framing:
type: multiline
startPattern: '^\d{4}-\d{2}-\d{2}'
backendConfig:
region: us-east-1
outputs:
- indexer

- id: indexer
commandName: splunk_indexer
config:
sourcetypeFallback: app_log
propsConf: |
[app_log]
TRANSFORMS-drop = drop_debug
transformsConf: |
[drop_debug]
REGEX = DEBUG
DEST_KEY = queue
FORMAT = nullQueue
outputs:
- sink

- id: sink
commandName: stdout
config:
encodingType: JSON_OBJECT

If an object holds 10 events and 3 of them contain DEBUG, 7 records leave the node. Each has the event text in __raw__ and sourcetype set to app_log.

  • fssource: Read files and set event boundaries with multiline framing
  • filter: Drop records with a FEEL condition instead of a Splunk transform