Skip to main content

Azure Event Hub Sink Node

The azureeventhubsink node publishes records to an Azure Event Hub. Records are serialized, packed into EventDataBatches that respect the service's 1 MB batch limit, and sent with the native Event Hub producer client. An optional partition key expression controls how events are distributed across partitions.

Authentication mirrors the Event Hub source: either a SAS connection string or Microsoft Entra ID.

Key Features​

  • Automatic batch packing: events fill a batch until the service's 1 MB limit is reached, then a new batch is started — no manual batch sizing
  • Partition key routing: a path expression resolves each record's partition key, so related events (per device, per user, per tenant) land on the same partition and preserve their relative order
  • Per-record error isolation: a record that fails to serialize, or that is too large for an empty batch, is reported as an error output instead of aborting the flush
  • Dual authentication: SAS connection string, or Entra ID via an explicit service principal or the ambient DefaultAzureCredential

Configuration​

FieldTypeRequiredDefaultDescription
eventHubNameStringYes—Name of the Event Hub (topic) to publish to
connectionStringStringConditional—SAS connection string for the namespace or entity. Mutually exclusive with fullyQualifiedNamespace
fullyQualifiedNamespaceStringConditional—Namespace host for Entra ID auth, e.g. my-namespace.servicebus.windows.net. Mutually exclusive with connectionString
tenantIdStringNo—Entra ID service principal tenant. Requires clientId, clientSecret and fullyQualifiedNamespace
clientIdStringNo—Entra ID service principal client id
clientSecretStringNo—Entra ID service principal secret
partitionKeyFieldExpressionStrStringNo—Path expression whose value becomes the event's partition key, e.g. $.deviceId. Omit to let Event Hub distribute events round-robin
encodingTypeStringYes—Output encoding, e.g. JSON_OBJECT. See Encoding Types

Authentication​

Exactly one authentication mode must be configured — the same rules as the Event Hub source.

Set connectionString for SAS auth, or fullyQualifiedNamespace for Entra ID. With Entra ID, supplying tenantId, clientId and clientSecret uses that service principal (all three are required together); omitting them falls back to DefaultAzureCredential.

The identity needs send rights on the hub — Azure Event Hubs Data Sender for Entra ID, or a Send SAS policy.

How It Works​

Each flush serializes its records, then sends them in batches:

  1. Every record is serialized using encodingType. A record that fails to serialize is recorded as an error output and skipped.
  2. Records are grouped by resolved partition key. A batch carries a single partition key, so each key becomes its own group. Records with no key (or no expression configured) form one group.
  3. Within a group, events are added to a batch until the service reports it full, at which point the batch is sent and a fresh one started. Grouping preserves the original record order within each key.
  4. Any leftover partial batch is sent at the end of the flush.

Failure handling​

  • Serialization failure: reported as an error output; the rest of the batch still sends.
  • Single event larger than an empty batch: reported as an error output and dropped — it can never be sent.
  • Send failure: propagates, so the enclosing sink treats the whole batch as failed. Store-and-forward, if enrolled, can buffer and retry it.

Partition keys​

Without partitionKeyFieldExpressionStr, Event Hub distributes events across partitions on its own — good for throughput, but events arrive downstream in no particular order.

With a partition key expression, all events resolving to the same key are sent to the same partition and consumed in order. Choose a key with enough distinct values to spread load; a low-cardinality key concentrates traffic onto a few partitions. Records whose expression resolves to nothing are treated as having no key.

DAG Example​

jobContext:
  otherProperties: {}
  metricTags: {}
  dlqConfig:

dag:
  - id: "source"
    commandName: "stdin"
    config:
      encodingType: "JSON_OBJECT"
    outputs:
      - "sink"

  - id: "sink"
    commandName: "azureeventhubsink"
    config:
      connectionString: "Endpoint=sb://my-namespace.servicebus.windows.net/;SharedAccessKeyName=send;SharedAccessKey=..."
      eventHubName: "processed-telemetry"
      partitionKeyFieldExpressionStr: "$.deviceId"
      encodingType: "JSON_OBJECT"