Skip to main content

Enrichment Node

What is data enrichment​

Data enrichment is the process of improving existing data by adding relevant information from various sources, making the data more accurate, complete, and useful for decision-making. This involves merging raw data with additional context, such as demographics or behavioral insights, to create a fuller picture of customers or operations. By enriching data, businesses can gain deeper insights that lead to better strategic decisions, allowing them to tailor their approaches more effectively.

How is data enrichment executed in Fleak?​

Fleak’s Enrichment Node performs data enrichment. Every incoming event triggers a query against an external data store. It takes the query result and attaches it to the original input event as the output event.

An Example​

Assuming every input event contains an order_id and the node is configured to query order_detail from an external data store.

{
  "order_id": 123
}

Every input triggers it to fetch that order’s details. Let’s say for the data store returns this as the query result:

{
  "product_id": 11,
  "quantity": 5,
  "price": 100,
  "currency": "USD"
}

The final output will look like this:

{
  "order_id": 123,
  "enrichment_output": {
    "product_id": 11,
    "quantity": 5,
    "price": 100,
    "currency": "USD"
  }
}

Configuration​

Provider​

The enrichment node is designed to work with various data storage systems. We're starting with Google BigQuery, and Postgres. To configure it, the user needs to first choose the Provider (the data store system) and the connection credentials.

Google BigQuery Creating Provider Connection​

  1. Give your connection a name
  2. Enter a Google Service Account Key in JSON format. Check here for more information.

Postgres Creating Provider Connection​

  1. Give your connection a name
  2. Build the structure of the JDBC URL. A URL string used to establish a connection to a database. A sample format looks like this: jdbc:postgresql://localhost:5432/my_database?. Use the separate fields to build the structure of the JDBC URL.
  3. Enter your Postgres Username and Password.

Output Field Name​

The Output Field Name is where the query result will be stored. In the example above, it's named enrichment_output, but you're free to rename it to whatever you like.

Query Template​

Query Template contains the query string to be sent to the target data store system. For the above example, the query can be:

SELECT *
FROM OrderDetails
WHERE OrderId={{$.order_id}};

Keep in mind that {{$.order_id}} is just a placeholder. What's inside the double brackets is the path to a field in the input event. When you run it, Fleak will look at each input event, figure out the value for $.order_id, and replace the placeholder with that value. In this way, the final query is created for the event. For the above example, it is:

SELECT *
FROM OrderDetails
WHERE OrderId=123;