Fleak Eval Expression Language (FEEL) Reference

FEEL is a flexible expression language for ZephFlow data transformations, supporting rich data manipulation, path selection, and extensive functions for filtering, mapping, and validation.

Introduction

Fleak Eval Expression Language is a flexible expression language designed for data transformation and evaluation. It provides various operations for manipulating dictionaries, arrays, strings, and timestamps, with a focus on data processing efficiency.

Basic Syntax and Types

Data Types

Numbers

Integers: 0 or any sequence of digits not starting with 0 (e.g., 42, 123)
Floating-point numbers: Numbers with decimal points (e.g., -1.5, .5, 10.0)
Negative numbers are supported for both integer and floating-point values

Strings

Can be enclosed in either single quotes ('hello') or double quotes ("world")
Support escape sequences:
- \' - Single quote in single-quoted strings
- \" - Double quote in double-quoted strings
- \\ - Backslash

Examples:

"This is a 'string'"
'This is a "string"'
"String with \"quotes\""
'String with \'quotes\''
"Path with \\backslashes\\"

Booleans

Two possible values: true or false
Used in conditional expressions and comparisons

Null

Represented by the keyword: null
Used to represent missing or undefined values

Path Selection

The language provides flexible path selection syntax for accessing nested data structures:

$: References the root record/document
.fieldName: Accesses a field in a dictionary using dot notation (field name must be a valid identifier)
[expression]: Accesses an array element by index, where the expression evaluates to an integer (zero-based)
["fieldName"]: Alternative syntax for accessing dictionary fields, required when:
- Field names contain special characters or spaces
- Field names are reserved words

Step Operations: Path selection steps can be applied to any expression result, not just the root $. This means you can chain operations on function results, variables, or any expression that returns an array or object.

If using dot notation ($.user.name), a field name must be a valid identifier. Valid identifiers:

Start with letter or underscore
Contain only letters, numbers, and underscores
Case-sensitive

Examples:

// Basic field access
$.user.name              // Access 'name' field in user object
$["user"]["name"]        // Same access using bracket notation

// Static array access
$.items[0]              // Access first element of items array
$.users[1].address      // Combine array and field access

// Dynamic array access
$.items[idx]            // Access element at variable index
$.data[i + 1]           // Access element using arithmetic
$.array[parse_int($.index)]  // Use function result as index
$.matrix[row][col]      // Multi-dimensional array access with variables

// Special character handling
$["special.field"]      // Access field containing a dot
$["user name"]          // Access field containing a space

// Step operations on expressions
(array($.a, $.b, $.c))[0]     // Array access on function result
(dict(x=$.value))["x"]       // Field access on dict expression
(case($.type == "A" => $.dataA, _ => $.dataB)).field  // Field access on case result

// Nested combinations
$.users[0]["address.line1"]  // Mix of array access and special fields
$.data[$.currentIndex].value // Use field value as array index
(grok($.raw, "%{PATTERN}")).extractedField  // Field access on grok result

Operators and Precedence

Operators

Listed in order of precedence (highest to lowest):

Unary Operations (highest)

Unary minus (-)
Logical NOT (not)

Multiplicative Operations

Multiplication (*)
Division (/)
Modulo (%)

Additive Operations

Addition / String concatenation (+)
Subtraction (-)

Relational Operations

Less than (<)
Greater than (>)
Less than or equal (<=)
Greater than or equal (>=)

Equality Operations

Equal (==)
Not equal (!=)

Logical AND (and)
Logical OR (or) (lowest)

Examples:

// Unary has highest precedence
-3 * 4      // Evaluates as (-3) * 4 = -12
not true and false  // Evaluates as (not true) and false = false

// Multiplication before addition
2 + 3 * 4   // Evaluates as 2 + (3 * 4) = 14

// Comparison before logical operations
x > 5 and y < 10  // Evaluates as (x > 5) and (y < 10)

// AND before OR
a or b and c  // Evaluates as a or (b and c)

Use parentheses to override default precedence:

(2 + 3) * 4  // Evaluates as 5 * 4 = 20
a and (b or c)  // Evaluates AND after OR

Control Flow

Case Expressions

Provides conditional logic with pattern matching syntax. The first matching condition's expression is evaluated and returned.

case(
    condition_1 => expression_1,
    condition_2 => expression_2,
    ...
    _ => default_expression
)

Examples:

// Simple status mapping
case(
    $.status == "active" => "Current",
    $.status == "pending" => "In Progress",
    _ => "Unknown"
)

// Data transformation
case(
    size_of($.items) == 0 => dict(status="empty", count=0),
    size_of($.items) > 10 => dict(
        status="full",
        count=size_of($.items),
        firstItem=$.items[0]
    ),
    _ => dict(status="partial", items=$.items)
)

// Type-based handling
case(
    $.value == null => "missing",
    str_contains(to_str($.value), "error") => "error",
    $.value > 100 => "high",
    _ => "normal"
)

Built-in Functions

Utility Functions

size_of()

Returns the size of the argument based on its type.

Syntax: size_of(expression)

Input Types:

Array: Returns number of elements
Dictionary/Object: Returns number of key-value pairs
String: Returns string length

Returns: Integer

Examples:

// Array size
size_of($.items)              // Returns number of elements
size_of(array(1, 2, 3))      // Returns 3

// Dictionary size
size_of(dict(a=1, b=2))      // Returns 2
size_of($.user.properties)    // Returns number of properties

// String size
size_of("hello")             // Returns 5
size_of($.message)           // Returns string length

exists()

Checks whether a path is present in the event. Unlike $.field == null, it tells a missing key apart from a key whose value is null.

Syntax: exists(pathExpression)

Input: A path expression (for example $.a.b or $.list[1]) Returns: Boolean — true if the last key or index on the path is present, even when its value is null Throws: Error if the argument is not a path expression

Examples:

// Input: {"a": 1, "b": null, "list": [10, 20]}
exists($.a)          // Returns true
exists($.b)          // Returns true (the key is present, its value is null)
exists($.c)          // Returns false (the key is not present)
exists($.a.b)        // Returns false ($.a is not a dictionary)
exists($.list[1])    // Returns true
exists($.list[5])    // Returns false

// Only set a field when the source key was sent
case(
    exists($.user.email) => dict(email=$.user.email),
    _ => dict()
)

coalesce()

Returns the first argument that is not null. The returned value keeps its original type.

Syntax: coalesce(value1, value2, ...)

Input: One or more expressions of any type Returns: The first non-null argument, or null if every argument is null

Note: All arguments are evaluated, even after a non-null one is found, so an error in a later argument still fails the expression.

Examples:

coalesce(null, "b", "c")                               // Returns "b"
coalesce(null, null)                                   // Returns null

// Read a value that may live under one of several paths
coalesce($.process.file.path, $.actor.process.file.path)

// Provide a default value
coalesce($.user.name, "unknown")

deep_contains()

Searches a value recursively for a substring, case-insensitively. Every primitive value inside dictionaries and arrays (strings, numbers, booleans) is checked by its string form. Dictionary keys are not searched.

Syntax: deep_contains(haystack, needle)

Input:

haystack: Any value, typically the whole event ($)
needle: The substring to look for (string)

Returns: Boolean — true if the needle appears in any value inside the haystack, false otherwise. Returns false if the haystack is null. Throws: Error if the needle is not a string (unless the haystack is null)

Examples:

// Input: {"process": {"cmd": "PowerShell.exe -enc ..."}, "tags": ["prod", "edge"]}
deep_contains($, "powershell")   // Returns true (case-insensitive)
deep_contains($.tags, "edge")    // Returns true
deep_contains($, "process")      // Returns false (keys are not searched)

cidr_match()

Checks whether an IP address falls inside a CIDR range. Supports IPv4 and IPv6.

Syntax: cidr_match(ip, cidr)

Input:

ip: The IP address to test (string), or an array of IP addresses
cidr: The CIDR range in address/prefixLength form (string)

Returns: Boolean — true if the IP is in the range. For an array, true if any element is in the range. Returns false if the IP is null, either value cannot be parsed, or the IP and the range are different families (IPv4 vs IPv6). IPv4-mapped IPv6 addresses such as ::ffff:10.1.2.3 are treated as IPv4. Throws: Error if the CIDR range is not a string

Examples:

cidr_match("10.1.2.3", "10.0.0.0/8")         // Returns true
cidr_match("192.168.1.10", "10.0.0.0/8")     // Returns false
cidr_match("2001:db8::1", "2001:db8::/32")   // Returns true
cidr_match("10.1.2.3", "2001:db8::/32")      // Returns false (different families)
cidr_match("not-an-ip", "10.0.0.0/8")        // Returns false

// Match any address in a multi-valued field
cidr_match($.observables.ips, "172.16.0.0/12")

// Flag traffic from private networks
case(
    cidr_match($.src_ip, "10.0.0.0/8") or cidr_match($.src_ip, "192.168.0.0/16") => "internal",
    _ => "external"
)

Dictionary Operations

dict()

Creates a dictionary with specified key-value pairs.

Syntax: dict(key1=expression1, key2=expression2, ...)

Key formats supported:

Simple identifiers: Start with letter or underscore, contain only letters, numbers, and underscores
Namespaced identifiers: Use :: separator for namespace qualification (e.g., namespace::field)
Quoted identifiers: Use single or double quotes for keys with special characters

Examples:

// Simple keys
dict(
  city=$.address.city,
  state=$.address.state
)

// Namespaced keys
dict(
  user::name=$.user.name,
  user::id=$.user.id,
  system::timestamp=$.timestamp
)

// Quoted keys for special characters
dict(
  "field.with.dots"=$.special_field,
  'field with spaces'=$.another_field,
  regular_field=$.normal_field
)

// Mixed key types
dict(
  namespace::field=$.data,
  "special.key"=$.value,
  normalKey=$.other
)

dict_merge()

Merges multiple dictionaries into one. Later dictionaries override earlier ones for duplicate keys.

Syntax: dict_merge(dict1, dict2, ...)

Input: One or more dictionaries Returns: Dictionary

Example:

dict_merge(
  $,
  dict(newField="value"),
  grok($.__raw__, "%{PATTERN}")
)

dict_remove()

Removes specified keys from a dictionary and returns a new dictionary without those keys.

Syntax: dict_remove(dictionary, key1, key2, ...)

Input:

Dictionary to remove keys from
One or more string keys to remove

Returns: New dictionary with specified keys removed Throws: Error if first argument is not a dictionary or if any key argument is not a string

Note: dict_remove only operates on top-level keys. It does not support nested keys: a key like "a.b" is treated as a single literal field name, not a path into a nested dictionary. To remove a nested field, operate on the nested dictionary directly (for example with dict_set to rebuild the parent), or use dict_remove_values to remove by value recursively.

Examples:

// Remove single key
dict_remove($.user, "password")
// Returns user object without password field

// Remove multiple keys
dict_remove($.data, "temp", "cache", "debug")
// Returns data object without temp, cache, and debug fields

// Chain with other operations
dict_merge(
  dict_remove($, "internal", "private"),
  dict(processed=true, timestamp=now())
)
// Remove internal fields and add new ones

dict_set()

Sets one or more key-value pairs on a dictionary and returns a new dictionary. Keys are evaluated at runtime, so dynamically computed key names are supported (unlike the dict() literal, whose keys are static). Existing keys are overwritten.

Syntax: dict_set(dictionary, key1, value1, key2, value2, ...)

Input:

Dictionary to set fields on
One or more key-value pairs; keys must be strings

Returns: New dictionary with the key-value pairs set Throws: Error if first argument is not a dictionary, if a key is not a string, or if key-value pairs are unbalanced

Behavior:

A null dictionary returns null
A null key skips that pair
A null value sets the field to null
Setting an existing key overwrites its value; for duplicate keys in one call, the last wins
The input is not mutated

Examples:

// Add a new field
dict_set({"a": 1}, "b", 2)
// Returns {"a": 1, "b": 2}

// Overwrite an existing key and add another
dict_set({"a": 1}, "a", 2, "b", 3)
// Returns {"a": 2, "b": 3}

// Set a dynamically named field
dict_set($, $.field_name, $.value)

dict_remove_values()

Recursively removes all entries whose value equals any of the given values. Walks nested dictionaries and arrays; array elements matching a value are removed from the array. Containers that become empty are kept.

Syntax: dict_remove_values(dictionary, value1, value2, ...)

Input:

Dictionary to clean
One or more values; any entry or array element equal to one of them is removed

Returns: New dictionary with matching values removed Throws: Error if first argument is not a dictionary

Behavior:

A null dictionary returns null
A null value argument removes entries/elements whose value is null
Equality is type-sensitive ("0" does not match 0); numbers compare by value (0 matches 0.0)
The input is not mutated

Examples:

// Remove a value from a flat dictionary
dict_remove_values({"a": "0x0", "b": 1}, "0x0")
// Returns {"b": 1}

// Recurse into nested dictionaries
dict_remove_values({"a": {"x": "-", "y": 2}}, "-")
// Returns {"a": {"y": 2}}

// Remove matching elements from arrays
dict_remove_values({"a": [1, "-", 2]}, "-")
// Returns {"a": [1, 2]}

dict_flatten()

Flattens a nested dictionary into a single-level dictionary. Nested keys are joined with a delimiter, and array elements get their zero-based index as a key segment. Useful before serializing to a flat format such as key-value pairs.

Syntax:

dict_flatten(dictionary) - Flatten with the default delimiter and depth
dict_flatten(dictionary, delimiter) - Flatten with a custom delimiter
dict_flatten(dictionary, delimiter, depth) - Flatten with a custom delimiter and depth

Input:

Dictionary to flatten
Optional delimiter placed between key segments; must be a non-empty string (default "_")
Optional depth, the maximum number of levels descended below each top-level key; must be an integer >= 1 (default 5)

Returns: New flat dictionary Throws: Error if first argument is not a dictionary, if the delimiter is empty or not a string, if depth is not an integer >= 1, or if the flattened keys would total more than 10,000,000 characters

Behavior:

A null dictionary returns null
Top-level scalar fields pass through unchanged
Descent stops after depth levels, so a key has at most depth + 1 segments; the value at the cutoff is kept unchanged
Scalars, nulls, empty dictionaries, and empty arrays are kept as values, never dropped
When two flattened keys collide, the one with fewer levels wins, so a literal "a_b" key beats a nested {"a": {"b": ...}}; a tie goes to the path that sorts first. No error is raised
The input is not mutated

Examples:

// Nested dictionaries and arrays
dict_flatten({"accounting": [{"firstName": "John"}, {"firstName": "Mary"}], "host": "h1"})
// Returns {"accounting_0_firstName": "John", "accounting_1_firstName": "Mary", "host": "h1"}

// Custom delimiter
dict_flatten({"a": {"b": {"c": 1}}}, ".")
// Returns {"a.b.c": 1}

// Limit the depth
dict_flatten({"a": {"b": {"c": 1}}}, "_", 1)
// Returns {"a_b": {"c": 1}}

// Flatten a sub-object and prefix its keys with "res_"
dict_flatten(dict(res=$.resource))

// Keep the original fields and add the flattened keys beside them
dict_merge($, dict_flatten($.resource))

Note: dict_flatten works on dictionary keys. To remove one level of nesting from an array of arrays, use arr_flatten.

Array Operations

array()

Creates an array from the provided arguments. Arguments can be of any type and can be mixed.

Syntax: array([expression1, expression2, ...])

Returns: Array

Example:

array($.field1, $.field2, "constant", 42)

range()

Generates an array of integer numbers based on the provided arguments. The 'end' parameter is exclusive (the range goes up to, but does not include, 'end').

Syntax variations:

range(count) - Generates integers from 0 up to (but not including) 'count'
range(start, end) - Generates integers from 'start' up to (but not including) 'end'
range(start, end, step) - Generates integers from 'start' up to (but not including) 'end', incrementing by 'step'

Input:

count: Integer (for single-argument form)
start, end: Integers
step: Integer (can be positive or negative, but not zero)

Returns: Array of integers

Examples:

// Single argument
range(5)                    // Returns [0, 1, 2, 3, 4]
range(0)                    // Returns []
range(-2)                   // Returns []

// Two arguments
range(2, 5)                 // Returns [2, 3, 4]
range(5, 2)                 // Returns [] (start >= end)

// Three arguments
range(0, 10, 2)            // Returns [0, 2, 4, 6, 8]
range(10, 0, -2)           // Returns [10, 8, 6, 4, 2]
range(0, 5, -1)            // Returns [] (wrong step direction)

// Practical use with dynamic array access
arr_foreach(
    range(size_of($.k1)),   // Generate indices [0, 1, 2, ...]
    idx,
    dict(
        k1=$.k1[idx],       // Access arrays using the generated index
        k2=$.k2[idx]
    )
)

arr_foreach()

Applies an expression to each element of an array.

Syntax: arr_foreach(arrayExpression, elementVariable, transformExpression)

Parameters:

arrayExpression: Expression that evaluates to an array. If it points to an object, treat that object as a single element array.
elementVariable: Variable name to represent each array element
transformExpression: Expression to apply to each element

Returns: Array of transformed elements

Example:

arr_foreach(
    $.users,
    user,
    dict(
        name=user.name,
        age=user.age,
        type=$.userType
    )
)

With dynamic array access, you can now use arr_foreach for indexed operations:

// Zip two arrays together using range and dynamic indexing
arr_foreach(
    range(size_of($.arr1)),
    idx,
    dict(
        k1=$.arr1[idx],
        k2=$.arr2[idx],
        position=idx
    )
)

arr_find()

Finds and returns the first element in an array that satisfies a condition. Returns null if no element matches.

Syntax: arr_find(arrayExpression, elementVariable, conditionExpression)

Parameters:

arrayExpression: Expression that evaluates to an array. If it points to an object, treats that object as a single-element array.
elementVariable: Variable name to represent each array element
conditionExpression: Boolean expression evaluated for each element

Returns: The first matching element, or null if no match found

Examples:

// Find user by ID
arr_find($.users, user, user.id == "100")
// Returns: { "name": "Alice", "id": "100" }

// Find first item above threshold
arr_find($.items, item, item.price > 100)

// With null safety
case(
    arr_find($.users, u, u.id == $.targetId) != null =>
        arr_find($.users, u, u.id == $.targetId).name,
    _ => "not found"
)

// Find in nested structure
arr_find($.orders, order, order.status == "pending" and order.total > 50)

arr_filter()

Filters and returns all elements in an array that satisfy a condition. Returns an empty array if no elements match.

Syntax: arr_filter(arrayExpression, elementVariable, conditionExpression)

Parameters:

arrayExpression: Expression that evaluates to an array. If it points to an object, treats that object as a single-element array.
elementVariable: Variable name to represent each array element
conditionExpression: Boolean expression evaluated for each element

Returns: Array of all matching elements (empty array if none match)

Examples:

// Filter by category
arr_filter($.items, item, item.category == "A")
// Returns: [{ "price": 100, "category": "A" }, { "price": 150, "category": "A" }]

// Filter with multiple conditions
arr_filter($.orders, order, order.status == "active" and order.amount > 100)

// Get first filtered element
arr_filter($.items, item, item.price > 50)[0]

// Combine with other functions
size_of(arr_filter($.logs, log, str_contains(log.level, "ERROR")))

// Chained operations
arr_foreach(
    arr_filter($.users, u, u.active == true),
    user,
    dict(name=user.name, email=user.email)
)

arr_to_dict()

Converts an array into a dictionary by evaluating key and value expressions for each element. If the input is a single object, it is treated as a single-element array. Duplicate keys are overwritten (last value wins).

Syntax: arr_to_dict(arrayExpression, elementVariable, keyExpression, valueExpression)

Parameters:

arrayExpression: Expression that evaluates to an array or object
elementVariable: Variable name representing each element
keyExpression: Expression that must evaluate to a string (used as dictionary key)
valueExpression: Expression evaluated for each element (used as dictionary value)

Returns: Dictionary, or null if input is null

Throws: Error if key expression evaluates to a non-string value

Examples:

// Convert key-value pairs array to dictionary
// $.columns = [{"key": "cpu", "value": "0.85"}, {"key": "mem", "value": "0.62"}]
arr_to_dict($.columns, elem, elem.key, parse_float(elem.value))
// Returns: {"cpu": 0.85, "mem": 0.62}

// With array values
arr_to_dict($.columns, elem, elem.key, array(parse_float(elem.value)))
// Returns: {"cpu": [0.85], "mem": [0.62]}

// Duplicate keys (last wins)
// $.items = [{"k": "a", "v": 1}, {"k": "a", "v": 2}]
arr_to_dict($.items, e, e.k, e.v)
// Returns: {"a": 2}

// Single object input (treated as single-element array)
// $.item = {"k": "key1", "v": 42}
arr_to_dict($.item, e, e.k, e.v)
// Returns: {"key1": 42}

in()

Checks whether a value exists in an array.

Syntax: in(value, arrayExpression)

Parameters:

value: The value to search for
arrayExpression: An expression that evaluates to an array

Returns: Boolean — true if the value is found in the array, false otherwise

Examples:

// Check membership
in(2, array(1, 2, 3))              // Returns true
in("x", array("a", "b"))           // Returns false

// With field references
in($.status, array("active", "pending", "approved"))

// Combined with other expressions
case(
    in($.country, array("US", "CA", "MX")) => "North America",
    in($.country, array("UK", "FR", "DE")) => "Europe",
    _ => "Other"
)

arr_flatten()

Flattens an array of arrays by one level, extracting all elements from nested arrays.

Syntax: arr_flatten(arrayOfArrays)

Input: An array that may contain nested arrays and/or simple elements Returns: A flattened array with one level of nesting removed

Behavior:

Extracts elements from first-level nested arrays
Non-array elements are preserved as-is in the result
Only performs shallow flattening (one level deep)
Preserves original order of elements

Examples:

// Array of arrays with simple elements
// $.nested = [[1, 2], [3, 4], [5, 6]]
arr_flatten($.nested)
// Results: [1, 2, 3, 4, 5, 6]

// Array of arrays with mixed element types
// $.data = [["a", "b"], [1, 2], [true, false]]
arr_flatten($.data)
// Results: ["a", "b", 1, 2, true, false]

// Array of arrays with deeper nesting (nesting preserved)
// $.deep = [[1, [2, 3]], [4, 5], [[6, 7]]]
arr_flatten($.deep)
// Results: [1, [2, 3], 4, 5, [6, 7]]

// Mixed array and non-array elements
// $.mixed = [[1, 2], 3, [4, 5], 6]
arr_flatten($.mixed)
// Results: [1, 2, 3, 4, 5, 6]

Note: This performs a shallow flatten operation, not a deep recursive flatten. Deeper nested arrays remain intact. Non-array elements at the top level are preserved as-is in the flattened result. To turn nested dictionary keys into flat, delimited keys, use dict_flatten.

String Operations

concat()

Joins one or more values into a single string.

Syntax: concat(value1, value2, ...)

Input:

values: One or more values of any type. Each value is converted to a string the same way as to_str(). A null value is treated as an empty string.

Returns: String, or null if every value is null.

Examples:

concat("a", "-", "b")              // Returns "a-b"
concat("id-", 42)                  // Returns "id-42"
concat("v", 1.5, true)             // Returns "v1.5true"
concat($.host, "|", $.missing)     // Returns "web-1|" when $.host is "web-1" and $.missing is null
concat($.missing)                  // Returns null

Note: The + operator fails when joining a string with a number and returns null when either side is null. concat accepts values of any type and skips null values, so it is the safer way to build a string from several fields.

str_split()

Splits a string into an array of substrings based on a delimiter.

Syntax: str_split(string, delimiter)

Input:

string: The string to split.
delimiter: The literal string delimiter to split by. Special regex characters are automatically escaped and treated as literal characters.

Returns: An array of strings.

Examples:

str_split("a,b,c", ",")     // Returns ["a", "b", "c"]
str_split("a+b+c", "+")     // Returns ["a", "b", "c"] (+ treated as literal)
str_split("a.b.c", ".")     // Returns ["a", "b", "c"] (. treated as literal)
str_split("a*b*c", "*")     // Returns ["a", "b", "c"] (* treated as literal)
str_split("hello world", " ") // Returns ["hello", "world"]

Note: Unlike regex-based split functions, all characters in the delimiter are treated literally. You don't need to escape special regex characters like +, ., *, ?, etc.

str_replace()

Replaces all occurrences of a target substring with a replacement string. Uses literal matching (not regex).

Syntax: str_replace(string, target, replacement)

Input:

string: The string to process (returns null if null)
target: The substring to find (must be a string)
replacement: The string to replace with (must be a string)

Returns: String

Throws: Error if target or replacement is null or not a string

Examples:

str_replace("2024 3Q", " ", "-")     // Returns "2024-3Q"
str_replace("a.b.c", ".", "/")       // Returns "a/b/c"
str_replace("hello", "x", "y")      // Returns "hello" (no match, unchanged)
str_replace("abc", "b", "")         // Returns "ac" (remove characters)
str_replace(null, "a", "b")         // Returns null

regex_replace()

Replaces every match of a regular expression in a string. Uses Java regex syntax. Use it instead of str_replace() when the text to replace follows a pattern rather than being a fixed substring, or when the replacement reuses parts of the match.

Syntax: regex_replace(string, pattern, replacement)

Input:

string: The string to process (returns null if null; must be a string otherwise)
pattern: Regex pattern (Java regex syntax, string)
replacement: The replacement string. $1, $2, ... insert the matching capture group, and ${name} inserts a named group (?<name>...).

Returns: String with all matches replaced. If nothing matches, the string is returned unchanged.

Throws: Error if:

string is not a string (for example a number; wrap it with to_str() first)
pattern or replacement is null or not a string
pattern is not a valid regex
replacement refers to a group the pattern does not have, or ends with a lone $

Examples:

// Collapse whitespace
regex_replace("a   b    c", "\\s+", " ")                       // Returns "a b c"

// Reorder parts with capture groups
regex_replace("2024-03-15", "(\\d+)-(\\d+)-(\\d+)", "$3/$2/$1")  // Returns "15/03/2024"

// Named groups
regex_replace("john smith", "(?<first>\\w+) (?<last>\\w+)", "${last}, ${first}")  // Returns "smith, john"

// Rewrite Windows event XML to key=value
regex_replace("<Data Name='User'>bob</Data><Data Name='Port'>443</Data>",
              "<Data Name='(\\w+)'>([^<]*)</Data>", "$1=$2 ")  // Returns "User=bob Port=443 "

// Strip a block that spans lines ((?s) lets . match newlines)
regex_replace($.raw, "(?s)<System>.*?</System>", "")

// Mask all but the last 4 digits
regex_replace("123-45-6789", "\\d{3}-\\d{2}-(\\d{4})", "***-**-$1")  // Returns "***-**-6789"

// No match, or null input
regex_replace("hello", "\\d+", "#")                             // Returns "hello" (no match, unchanged)
regex_replace(null, "a", "b")                                   // Returns null

Notes:

Every match is replaced, not just the first.
Use inline flags at the start of the pattern for matching modes: (?i) ignores case, (?s) lets . match newlines, (?m) makes ^ and $ match at each line.
Use double backslash (\\) to represent regex special characters like \d, \s, \w.
To put a literal $ or \ in the replacement, escape it as \\$ or \\\\.
A pattern that can match an empty string (such as x*) matches between every character: regex_replace("abc", "x*", "-") returns "-a-b-c-".

substr()

Extracts a substring from a string using SQL or Python style syntax.

Syntax:

substr(string, start) - Extract from start position to end of string
substr(string, start, length) - Extract specified number of characters

Parameters:

string: The string to extract from
start: Starting position (0-based index)
- Positive values: Count from beginning (0 is first character)
- Negative values: Count from end (-1 is last character)
length (optional): Number of characters to extract
- If omitted, extracts to end of string
- Must be non-negative

Returns: String

Throws: Error if:

Input string is not a string
Start or length are not numeric
Length is negative

Note: Returns null if input string is null.

Edge cases:

Start beyond string length: Returns empty string ""
Start before beginning (e.g., -100 on 5-char string): Treated as 0
Length exceeds remaining characters: Returns substring to end of string

Examples:

// Basic extraction
substr("hello world", 6)        // Returns "world"
substr("hello world", 0, 5)     // Returns "hello"
// Negative indices (Python-style)
substr("hello", -2)             // Returns "lo" (last 2 chars)
substr("hello", -3, 2)          // Returns "ll" (2 chars starting 3 from end)
// Edge cases
substr("hello", 10)             // Returns "" (start beyond length)
substr("hello", -10)            // Returns "hello" (clamped to start)
substr("hello", 2, 100)         // Returns "llo" (length truncated)

upper()

Converts a string to uppercase.

Syntax: upper(expression)

Input: String Returns: String Throws: Error if input is not a string

Example:

upper($.name)  // "john" -> "JOHN"

lower()

Converts a string to lowercase.

Syntax: lower(expression)

Input: String Returns: String Throws: Error if input is not a string

Example:

lower($.name)  // "JOHN" -> "john"

str_contains()

Tests if a string contains a substring.

Syntax: str_contains(string, substring)

Input: Two strings (string to search, substring to find) Returns: Boolean Throws: Error if either input is not a string

Examples:

// Basic usage
str_contains($.message, "error")  // Returns true if found

// In case expressions
case(
    str_contains($.level, "ERROR") => "critical",
    str_contains($.level, "WARN") => "warning",
    _ => "info"
)

// With other functions
str_contains(lower($.text), $.searchTerm)

regex_match()

Tests if a string matches a regular expression pattern. Uses Java regex syntax.

Syntax: regex_match(string, pattern)

Input:

String to test
Regex pattern (Java regex syntax)

Returns: Boolean (true if string matches the entire pattern, false otherwise)

Null handling: Returns false if input string is null

Examples:

// Match digits
regex_match($.num, "\\d+")                     // matches "12345"

// Match version format
regex_match($.ver, "\\d+\\.\\d+\\.\\d+")       // matches "1.2.3"

// Match with whitespace
regex_match($.text, "hello\\s+world")          // matches "hello world" with any whitespace

// Match email format
regex_match($.email, "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$")

// In case expressions
case(
    regex_match($.input, "^\\d{4}-\\d{2}-\\d{2}$") => "date",
    regex_match($.input, "^\\d+$") => "number",
    _ => "text"
)

Note: Use double backslash (\\) to represent regex special characters like \d, \s, \w, etc.

regex_match_any()

Tests whether a value, or any element of an array, fully matches a regular expression. Uses Java regex syntax.

Syntax: regex_match_any(value, pattern)

Input:

value: A scalar or an array. Numbers and booleans are matched by their string form.
pattern: Regex pattern (Java regex syntax, string)

Returns: Boolean — true if the value (or any array element) matches the entire pattern. null values and dictionaries never match. A match that runs too long (catastrophic backtracking) is treated as no match. Throws: Error if the pattern is not a string or is not a valid regex

Examples:

regex_match_any("ERROR", "ERROR|FATAL")                   // Returns true
regex_match_any(array("info", "404"), "\\d+")             // Returns true ("404" matches)
regex_match_any("error: disk full", "error")              // Returns false (must match the entire string)
regex_match_any($.dns.answers, ".*\\.example\\.com")

regex_search_any()

Like regex_match_any(), but the pattern only needs to be found somewhere in the value instead of matching all of it.

Syntax: regex_search_any(value, pattern)

Input:

value: A scalar or an array. Numbers and booleans are matched by their string form.
pattern: Regex pattern (Java regex syntax, string)

Returns: Boolean — true if the pattern is found anywhere in the value (or in any array element). null values and dictionaries never match. A match that runs too long (catastrophic backtracking) is treated as no match. Throws: Error if the pattern is not a string or is not a valid regex

Examples:

regex_search_any("error: disk full", "error")                      // Returns true
regex_search_any(array("GET /health", "POST /login"), "^POST")     // Returns true
regex_search_any($.process.cmd_line, "(?i)-enc(odedcommand)?\\s")

grok()

Applies a Grok pattern to extract structured data from a string.

Syntax: grok(string, pattern)

Input:

String to parse
Grok pattern string

Returns: Dictionary of extracted fields Throws: Error if pattern is invalid

Example:

grok($.rawLog, "%{TIMESTAMP:timestamp} %{WORD:level}: %{GREEDYDATA:message}")

Time and Date Operations

ts_str_to_epoch()

Converts a timestamp string to epoch milliseconds.

Syntax: ts_str_to_epoch(timestampString, formatPattern)

Input:

Timestamp string
Java SimpleDateFormat pattern

Returns: Integer (epoch milliseconds) Throws: Error if timestamp doesn't match pattern

Example:

ts_str_to_epoch($.timestamp, "yyyy-MM-dd HH:mm:ss")

epoch_to_ts_str()

Converts epoch milliseconds to formatted timestamp string.

Syntax: epoch_to_ts_str(epochMillis, formatPattern)

Input:

Epoch milliseconds (integer)
Java SimpleDateFormat pattern

Returns: String Throws: Error if timestamp is invalid

Example:

epoch_to_ts_str($.epochTime, "yyyy-MM-dd HH:mm:ss")

duration_str_to_mills()

Converts a duration string to milliseconds.

Syntax: duration_str_to_mills(durationString)

Input: Duration string in "HH:mm:ss" format Returns: Integer (milliseconds) Throws: Error if duration format is invalid

Example:

duration_str_to_mills("01:30:00")  // Returns 5400000 (1.5 hours in milliseconds)

now()

Returns the current time as epoch milliseconds.

Syntax: now()

Input: None Returns: Long integer (current epoch milliseconds)

Examples:

// Get current timestamp
now()  // Returns current epoch milliseconds (e.g., 1699564800000)

// Use in data processing
dict(
  processedAt=now(),
  data=$.payload
)

// Calculate time difference
dict(
  eventTime=$.timestamp,
  currentTime=now(),
  ageInMillis=now() - $.timestamp
)

// Use with epoch_to_ts_str for formatted output
epoch_to_ts_str(now(), "yyyy-MM-dd HH:mm:ss")

date_component_eq_any()

Checks whether one calendar component of a timestamp, in UTC, equals a number. The timestamp may be a scalar or an array.

Syntax: date_component_eq_any(timestamp, unit, number)

Input:

timestamp: Epoch milliseconds (number) or an ISO-8601 string with a time zone offset (for example "2024-05-01T14:30:00Z" or "2024-05-01T16:30:00+02:00"), or an array of these
unit: One of "minute", "hour", "day" (day of month), "week" (ISO week of year), "month", "year". Case-insensitive.
number: The integer to compare the component against. Any fractional part is dropped.

Returns: Boolean — true if the component equals the number. For an array, true if any element does. Returns false for null, for timestamps that cannot be parsed (including ISO strings without an offset), and for an unknown unit. Throws: Error if the unit is not a string or the number is not a number

Examples:

date_component_eq_any("2024-05-01T14:30:00Z", "hour", 14)        // Returns true
date_component_eq_any("2024-05-01T16:30:00+02:00", "hour", 14)   // Returns true (converted to UTC)
date_component_eq_any(1714573800000, "month", 5)                 // Returns true (2024-05-01T14:30:00Z)
date_component_eq_any("2024-05-01T14:30:00", "hour", 14)         // Returns false (no offset)

// True if any of the timestamps falls in ISO week 18
date_component_eq_any($.timestamps, "week", 18)

Type Conversion Functions

parse_int()

Parses a string into a long integer (equivalent to Java's Long.parseLong()).

Syntax:

parse_int(string) - Parse with base 10
parse_int(string, radix) - Parse with specified base

Input:

String containing an integer
Optional radix (base) for number parsing (2-36)

Returns: Long integer Throws: Error if string is not a valid integer or radix is invalid

Examples:

parse_int("42")        // Returns 42 (base 10)
parse_int("1010", 2)   // Returns 10 (binary)
parse_int("FF", 16)    // Returns 255 (hexadecimal)
parse_int("77", 8)     // Returns 63 (octal)

parse_float()

Parses a string into a floating-point number (equivalent to Java's Double.parseDouble()).

Syntax: parse_float(string)

Input: String that represents a number Returns: Double Throws: Error if string is not a valid number

Example:

parse_float("3.14")  // Returns 3.14
parse_float("-1.5")  // Returns -1.5
parse_float("42")    // Returns 42.0

to_str()

Converts a value to string.

Syntax: to_str(expression)

Input: Any value Returns: String, or null if input is null

Example:

to_str($.numericField)  // Converts number to string
to_str(null)            // Returns null

Math Operations

floor()

Rounds down a floating-point number to the nearest integer.

Syntax: floor(number)

Input: Number (integer or floating-point) Returns: Long integer (rounded down) Throws: Error if input is not a number

Examples:

floor(123.45)   // Returns 123
floor(-123.45)  // Returns -124
floor(42)       // Returns 42
floor(0.9)      // Returns 0

ceil()

Rounds up a floating-point number to the nearest integer.

Syntax: ceil(number)

Input: Number (integer or floating-point) Returns: Long integer (rounded up) Throws: Error if input is not a number

Examples:

ceil(123.45)   // Returns 124
ceil(-123.45)  // Returns -123
ceil(42)       // Returns 42
ceil(0.1)      // Returns 1

num_compare_any()

Compares a value, or each element of an array, against a number. Numeric strings are converted to numbers before comparing.

Syntax: num_compare_any(value, operator, number)

Input:

value: A number, a numeric string, or an array of these
operator: One of "lt" (less than), "lte" (less than or equal), "gt" (greater than), "gte" (greater than or equal)
number: The number to compare against

Returns: Boolean — true if the value (or any array element) satisfies the comparison. null and non-numeric values never satisfy it. Throws: Error if the operator is not a string or the number is not a number. Operators are lowercase and case-sensitive. An unknown operator only raises an error when a numeric value is compared; for null or non-numeric input the function returns false.

Examples:

num_compare_any(80, "lt", 1024)                     // Returns true
num_compare_any("8080", "gte", 1024)                // Returns true (numeric string)
num_compare_any(array(22, 443, 8080), "gt", 1024)   // Returns true (8080 > 1024)
num_compare_any("n/a", "lt", 1024)                  // Returns false

// Flag events that used any privileged port
num_compare_any($.dst_ports, "lt", 1024)

Random Number Generation

random_long()

Generates a cryptographically secure random long integer.

Syntax: random_long()

Input: None Returns: Long integer (cryptographically secure random value)

Examples:

// Generate random ID
dict(
  id=random_long(),
  data=$.payload
)

// Create random seed
random_long()  // Returns random long value (e.g., -3458764513820540928)

// Use for distributed tracing
dict(
  traceId=random_long(),
  spanId=random_long(),
  timestamp=now()
)

// Generate multiple random values
array(random_long(), random_long(), random_long())

Note: This function uses Java's SecureRandom class to generate cryptographically strong random numbers, suitable for security-sensitive applications.

Python Integration

Experimental — Open-Source ZephFlow Only

The python() function is experimental and only available in open-source ZephFlow (zephflow-core). It is not available in the Mapper App or the Workflow Builder App.

python()

Executes a Python function defined within a script string. Requires GraalVM with Python language support.

Syntax: python(scriptString, arg1, arg2, ...)

Input:

scriptString: String literal containing Python code with exactly one function definition
arguments: Zero or more FEEL expressions to pass to the Python function

Returns: Value returned by the Python function, converted to FEEL data types Throws: Error if script contains zero or multiple functions, or if Python execution fails

Requirements:

Script must define exactly one top-level function
GraalVM with Python language support must be configured
Python functions are pre-compiled for execution

Examples:

// Simple calculation
python("
def calculate(x, y):
    return x * y + 10
", $.value1, $.value2)

// Data processing
python("
def process_items(items):
    return [item.upper() for item in items if len(item) > 2]
", $.stringArray)

// Complex transformation
python("
def transform_data(data, multiplier):
    result = {}
    for key, value in data.items():
        if isinstance(value, (int, float)):
            result[key] = value * multiplier
        else:
            result[key] = str(value).upper()
    return result
", $.inputData, 1.5)

Note: Python function execution is sandboxed and cached for performance. The same script string will reuse the compiled function.

Advanced Patterns with Dynamic Array Access

The addition of dynamic array access enables powerful new patterns:

Parallel Array Processing

Process multiple arrays in parallel by iterating over indices:

// Transform parallel arrays into structured data
dict(
  results=arr_foreach(
    range(size_of($.names)),
    i,
    dict(
      name=$.names[i],
      score=$.scores[i],
      grade=case(
        $.scores[i] >= 90 => "A",
        $.scores[i] >= 80 => "B",
        $.scores[i] >= 70 => "C",
        _ => "F"
      )
    )
  )
)

Matrix Operations

Access multi-dimensional arrays:

// Process a 2D matrix
arr_foreach(
  range(size_of($.matrix)),
  row,
  arr_foreach(
    range(size_of($.matrix[row])),
    col,
    $.matrix[row][col] * 2
  )
)

Conditional Array Access

Use computed indices based on conditions:

// Access array element based on status
dict(
  selectedItem=$.items[
    case(
      $.priority == "high" => 0,
      $.priority == "medium" => 1,
      _ => 2
    )
  ]
)

Sliding Windows

Process arrays with sliding windows:

// Create pairs of consecutive elements
arr_foreach(
  range(size_of($.data) - 1),
  i,
  dict(
    current=$.data[i],
    next=$.data[i + 1],
    diff=$.data[i + 1] - $.data[i]
  )
)

Error Handling

Null Safety

Operations on null values generally propagate null
Function-specific null handling is documented in each function

Use case expressions for explicit null handling:

case(
  $.field == null => "missing",
  _ => to_str($.field)
)

Type Safety

Type mismatches in operations throw errors
Explicit type conversion functions available:
- to_str() for string conversion
- parse_int() for integer conversion
- parse_float() for floating-point conversion

Common Error Conditions

Type Mismatches

// These will throw errors:
"string" * 5        // Can't multiply string
upper(42)           // upper() requires string
size_of(true)       // size_of() not valid for boolean
floor("text")       // floor() requires number
ceil(null)          // ceil() doesn't accept null

Field Access

// These return null:
$.nonexistent.field  // Missing field
$.array[999]         // Index out of bounds
$.string[0]          // Array access on a non-array
$[true]              // Field/index access on a non-matching type

Function Arguments

// These throw errors:
size_of()            // Missing argument
str_contains("a")    // Missing second argument
upper(42)            // Wrong argument type
parse_int("abc")     // Invalid number format
parse_int("10", 1)   // Invalid radix (must be 2-36)
python("not a function") // Script must contain one function

Best Practices

Check for null values:

case(
  $.amount != null => $.amount * 100,
  _ => 0
)

Provide default values:

dict(
  name=case($.name != null => $.name, _ => "unknown"),
  value=case($.value != null => $.value, _ => 0)
)

Validate data structure:

case(
  $.data == null => dict(error="data is missing"),
  _ => arr_foreach($.data, item, upper(item.name))
)

Bounds checking for dynamic array access:

case(
  idx >= 0 and idx < size_of($.array) => $.array[idx],
  _ => null
)

Safe type conversion:

// With error handling
case(
  str_contains($.input, "0x") => parse_int(substr($.input, 2), 16),
  str_contains($.input, ".") => floor(parse_float($.input)),
  _ => parse_int($.input)
)

// Safe string conversion
case(
  $.value != null => to_str($.value),
  _ => "N/A"
)

Math operations with validation:

// Ensure positive result for floor/ceil
case(
  $.value >= 0 => floor($.value),
  _ => ceil($.value)  // Different rounding for negative numbers
)

Introduction​

Basic Syntax and Types​

Data Types​

Numbers​

Strings​

Booleans​

Null​

Path Selection​

Operators and Precedence​

Operators​

Control Flow​

Case Expressions​

Built-in Functions​

Utility Functions​

size_of()​

exists()​

coalesce()​

deep_contains()​

cidr_match()​

Dictionary Operations​

dict()​

dict_merge()​

dict_remove()​

dict_set()​

dict_remove_values()​

dict_flatten()​

Array Operations​

array()​

range()​

arr_foreach()​

arr_find()​

arr_filter()​

arr_to_dict()​

in()​

arr_flatten()​

String Operations​

concat()​

str_split()​

str_replace()​

regex_replace()​

substr()​

upper()​

lower()​

str_contains()​

regex_match()​

regex_match_any()​

regex_search_any()​

grok()​

Time and Date Operations​

ts_str_to_epoch()​

epoch_to_ts_str()​

duration_str_to_mills()​

now()​

date_component_eq_any()​

Type Conversion Functions​

parse_int()​

parse_float()​

to_str()​

Math Operations​

floor()​

ceil()​

num_compare_any()​

Random Number Generation​

random_long()​

Python Integration​

python()​

Advanced Patterns with Dynamic Array Access​

Parallel Array Processing​

Matrix Operations​

Conditional Array Access​

Sliding Windows​

Error Handling​

Null Safety​

Type Safety​

Common Error Conditions​

Best Practices​

Introduction

Basic Syntax and Types

Data Types

Numbers

Strings

Booleans

Null

Path Selection

Operators and Precedence

Operators

Control Flow

Case Expressions

Built-in Functions

Utility Functions

size_of()

exists()

coalesce()

deep_contains()

cidr_match()

Dictionary Operations

dict()

dict_merge()

dict_remove()

dict_set()

dict_remove_values()

dict_flatten()

Array Operations

array()

range()

arr_foreach()

arr_find()

arr_filter()

arr_to_dict()

in()

arr_flatten()

String Operations

concat()

str_split()

str_replace()

regex_replace()

substr()

upper()

lower()

str_contains()

regex_match()

regex_match_any()

regex_search_any()

grok()

Time and Date Operations

ts_str_to_epoch()

epoch_to_ts_str()

duration_str_to_mills()

now()

date_component_eq_any()

Type Conversion Functions

parse_int()

parse_float()

to_str()

Math Operations

floor()

ceil()

num_compare_any()

Random Number Generation

random_long()

Python Integration

python()

Advanced Patterns with Dynamic Array Access

Parallel Array Processing

Matrix Operations

Conditional Array Access

Sliding Windows

Error Handling

Null Safety

Type Safety

Common Error Conditions

Best Practices