Skip to main content

Azure Monitor Sink Node

The azuremonitorsink node forwards pipeline records to Azure Monitor Logs via the Azure Monitor Ingestion API (Data Collection Endpoint).

Records are batched and sent to a Log Analytics workspace using a Data Collection Endpoint (DCE) and a Data Collection Rule (DCR) that define the target table and schema. Authentication uses an Entra ID (formerly Azure AD) token obtained from a Service Principal credential.

Key Features​

  • Logs Ingestion API: uses the Azure Monitor Logs Ingestion API for reliable, schema-validated ingestion into Log Analytics workspaces
  • Entra ID token authentication: obtains a bearer token from Azure Entra ID using a Service Principal for secure API access
  • Configurable DCR stream routing: target any custom table in a Log Analytics workspace by specifying the DCR immutable ID and stream name
  • Batched ingestion: records are accumulated and sent in batches for efficiency, reducing API call overhead

Configuration​

FieldTypeRequiredDefaultDescription
tenantIdStringYes—Azure Active Directory tenant ID
dceEndpointStringYes—Data Collection Endpoint (DCE) URL (e.g. https://my-dce-xxxx.eastus-1.ingest.monitor.azure.com)
dcrImmutableIdStringYes—Immutable ID of the Data Collection Rule (DCR) that defines the target table
streamNameStringYes—Name of the stream in the DCR to ingest into (e.g. Custom-MyTable_CL)
credentialIdStringYes—ID of Azure Service Principal credentials in jobContext.otherProperties. Omit to use default credential chain.
timeGeneratedFieldStringNoTimeGeneratedRecord field used as the ingestion timestamp
batchSizeIntegerNo500Records sent per ingestion API request

Setting Up a DCE and DCR​

Before using this sink, two Azure resources must be created:

  • Data Collection Endpoint (DCE): the HTTPS ingestion URL that accepts log data. Create one in the Azure portal under Monitor > Data Collection Endpoints. The dceEndpoint field takes the endpoint's URI (e.g. https://my-dce-xxxx.eastus-1.ingest.monitor.azure.com).
  • Data Collection Rule (DCR): defines the schema transformation and the destination Log Analytics table. The dcrImmutableId is the read-only ID shown on the DCR's Overview page (format: dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx). The streamName must match a stream declared in the DCR (custom tables follow the pattern Custom-<TableName>_CL).

The Service Principal referenced by credentialId must be assigned the Monitoring Metrics Publisher role on the DCR. Refer to the Azure Monitor Logs Ingestion API documentation for full setup steps.

DAG Example​

jobContext:
  otherProperties:
    azure-cred:
      username: my-client-id
      password: my-client-secret
  metricTags: {}
  dlqConfig:

dag:
  - id: "source"
    commandName: "stdin"
    config:
      encodingType: "JSON_OBJECT"
    outputs:
      - "sink"

  - id: "sink"
    commandName: "azuremonitorsink"
    config:
      tenantId: "yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy"
      dceEndpoint: "https://my-dce-xxxx.eastus-1.ingest.monitor.azure.com"
      dcrImmutableId: "dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      streamName: "Custom-MyTable_CL"
      credentialId: "azure-cred"
      timeGeneratedField: "TimeGenerated"
      batchSize: 500
  • azuremonitorsource: Query Azure Monitor Log Analytics workspaces using KQL
  • splunkhecsink: Forward pipeline records to Splunk via the HTTP Record Collector