Skip to main content

Delivery Guarantees and Write Failures

What happens to records when a deployed workflow cannot write them to a destination: a sink may retry first, and records that still fail are counted as errors and dropped. Fleak does not guarantee exactly-once or at-least-once delivery to sinks.

What Happens When a Write Fails​

This applies to every sink in a deployed workflow.

  1. The sink may retry first. Retry rules differ by sink, and some sinks do not retry at all, so a single failure fails the write. Where the rules are documented, they are on the sink's node page. For example, the S3 Sink retries any failed upload, while the Splunk HEC Sink retries only rate limiting, server errors and network failures.
  2. Records that still fail are dropped. After any retries are used up, the records in that write are counted as errors and are not tried again. Some of them may still have reached the destination, for example when a request timed out after the destination had accepted it, so the error count is not an exact list of records missing at the destination. The workflow keeps running and moves on to new data; the failed records are not replayed later.

No exactly-once or at-least-once guarantee. Because failed records are dropped once any retries are used up, a destination outage that outlasts them loses data. Retries can also deliver the same records twice when an earlier attempt partly succeeded. Do not assume a sink is at-least-once unless its node page says so for the case you care about.

Store and Forward​

Some sinks offer Enable Store and Forward, for example the Databricks Zerobus Sink and the legacy panel of the Kafka Sink. With it on, a connectivity failure buffers records to a local queue and replays them when the destination is reachable again, instead of dropping them. Records are still dropped once the local queue is full, and failures that are not connectivity problems follow the rules above.