Skip to main content

Monitoring and Debugging

Grid provides observability for distributed data pipeline execution through an integrated monitoring stack. Access metrics, logs, and debugging tools whether running on-premises or through Fleak's managed platform.

Monitoring Stack Components​

Grid's observability infrastructure consists of four integrated components deployed by the Grid Helm chart. Their settings live under monitoring in the chart's values.

The commands below write the chart's resource-name prefix as <release>. See Resource Names for how to work it out from your Helm release name.

VictoriaLogs - Log Aggregation​

VictoriaLogs provides centralized log collection and search capabilities:

Features:

  • Collects logs from all job tasks across worker nodes
  • Flexible querying with LogsQL syntax
  • Logs are indexed by the job_id, task_id, worker_id, and source fields
  • 20Gi persistent storage and 7-day retention by default (monitoring.victorialogs.persistence.size and monitoring.victorialogs.retentionPeriod)

Query Capabilities:

# Find all logs for a specific job
job_id:"<job-id>"

# Search for errors in a task
task_id:"<task-id>" AND level:"ERROR"

# Logs for a job from the last hour
job_id:"<job-id>" AND _time:1h

Access:

  • Via Grafana (Explore view, VictoriaLogs data source)
  • REST API: GET /api/v1/logs/search on JobMaster (see Logs API)
  • On-prem UI: the Logs tab of the Task Details view tails a task's logs live

InfluxDB v2 - Metrics Storage​

InfluxDB stores time-series metrics for performance monitoring:

Collected Metrics: Workers report pipeline counters to the worker_metrics measurement in the zephflow bucket (org fleak, 30-day retention by default). Each point is tagged with job_id, node_id, and command_name. Fields include:

  • counter_input_event_count and counter_input_event_size: records and bytes read
  • counter_sink_output_count: records written by sinks
  • counter_error_event_count, counter_sink_error_count, and counter_input_deser_err_count: processing, sink, and deserialization errors

Query Language: Supports Flux for time-series analysis:

from(bucket: "zephflow")
  |> range(start: -1h)
  |> filter(fn: (r) => r._measurement == "worker_metrics")
  |> filter(fn: (r) => r.job_id == "<job-id>")
  |> filter(fn: (r) => r._field == "counter_input_event_count")
  |> aggregateWindow(every: 1m, fn: sum)

Vector - Log Pipeline​

Vector handles log ingestion and routing. The chart runs two kinds of Vector:

Central Vector service:

  1. Input: Listens for JSON log lines on a Logstash-style TCP port (default 9000)
  2. Transform: Normalizes the timestamp, message, logger, and thread fields
  3. Output: Forwards to VictoriaLogs over HTTP, using job_id, task_id, worker_id, and source as stream fields

Logs are sent in batches of up to 1MB, flushed at least every 5 seconds.

Per-node Vector DaemonSet: A DaemonSet runs Vector on every node to collect Kubernetes pod logs. It is enabled by default and can be turned off with monitoring.vectorDaemonset.enabled: false. On OpenShift it is skipped unless you set monitoring.vectorDaemonset.enabledOnOpenShift: true and grant its service account the privileged SCC, because it reads node logs through a hostPath volume.

Configuration: Customize via Helm values:

monitoring:
  vector:
    logstashPort: 9000
    replicaCount: 2

Grafana - Visualization Dashboard​

Grafana provides a single view across logs and metrics:

Pre-configured Dashboards:

  • Grid Logs Overview: Log exploration across jobs and tasks
  • Grid Metrics Overview: Pipeline metrics from InfluxDB

Data Sources:

  • InfluxDB for metrics queries
  • VictoriaLogs for log exploration

Alerting: Grafana's built-in alerting can notify you on metric thresholds, but the chart ships no alert rules or notification channels. Configure them in Grafana yourself if you need them.

Accessing Grafana Locally​

For on-premises deployments, Grafana is deployed as part of the Grid Helm chart and can be accessed locally using Kubernetes port forwarding.

Port Forward to Grafana Service:

kubectl port-forward -n <namespace> svc/<release>-grafana 3000:3000

Replace <namespace> with your Grid deployment namespace and <release> with your chart's resource-name prefix.

Access Grafana: Once port forwarding is active, open your browser to:

http://localhost:3000

Default Credentials: The Grafana admin credentials come from monitoring.grafana.auth.username and monitoring.grafana.auth.password in your Helm values (admin / admin unless you changed them; change them in production). To read the password from the cluster:

kubectl get secret -n <namespace> <release>-grafana -o jsonpath="{.data.admin-password}" | base64 --decode

Pre-configured Data Sources: Grafana comes with pre-configured connections to:

  • InfluxDB: URL http://<release>-influxdb:9000; organization and bucket are pre-configured
  • VictoriaLogs: URL http://<release>-victorialogs:9428

On-Premises Monitoring UI​

Grid's on-premises deployment includes a browser-based interface served by the JobMaster service. See Job Submission for how to open it and a full walkthrough.

Cluster Overview​

The Cluster view lists the workers registered with Grid:

Worker Status:

  • Worker ID and URL
  • Max Tasks: the worker's concurrent task limit
  • State: Operational, Unstable, Maxed, Down
  • Last Registered (most recent heartbeat) and Created At timestamps
  • Paginated, 10 workers per page

The Cluster view does not refresh on its own; reopen it to see current state.

Job Management Dashboard​

View and control all jobs in the system:

Job List Features:

  • Job ID with clickable rows that open the detail view
  • External ID mapping for system integration
  • Task counters: Ready, Scheduled, Running, Completed, Failed, Timeouts, Total
  • Status badges with color coding:
    • COMPLETED (green): All tasks finished successfully
    • RUNNING (blue): Job in progress
    • FAILED (red): One or more tasks failed
    • PARTIAL_FAILURE (yellow): Some tasks failed while others are still active
    • TIMEOUT (yellow): Tasks exceeded schedule or run time limits
    • MIXED (gray): Combination of states
    • EMPTY (gray): No tasks in job
  • Creation date, newest first
  • Pagination controls
  • Automatic refresh every 10 seconds

Job and Task Actions:

  • Stop a job from its Job Detail view
  • Kill a single task from the Job Detail task list
  • Open a task to see its details, pipeline graph, and live logs

REST API for Programmatic Access​

Grid exposes REST APIs on the JobMaster service (port 9527) for integration with external monitoring tools and custom dashboards. The examples use localhost:9527, which works while the JobMaster port-forward from Job Submission is running.

Metrics API​

Query time-series metrics from InfluxDB:

Endpoint:

GET /api/v1/metrics

Query Parameters:

  • from: Start timestamp (ISO 8601 date-time). Defaults to one minute ago if omitted.
  • to: End timestamp (ISO 8601 date-time). Defaults to now if omitted.
  • interval: Aggregation window (e.g., 1m, 5m, 1h)
  • limit: Maximum data points to return (default 1000)
  • <key>=<value>: Filter by tag (e.g., job_id=<job-id>)

Response Format:

{
  "from": "2024-01-29T09:00:00Z",
  "to": "2024-01-29T10:00:00Z",
  "interval": "1m",
  "filters": {
    "job_id": "018f9a2b-1c3d-7e4f-8a9b-0c1d2e3f4a5b"
  },
  "dataPoints": [
    {
      "timestamp": "2024-01-29T10:00:00Z",
      "counter_input_event_count": 1250
    }
  ],
  "totalDataPoints": 1,
  "metricNames": ["counter_input_event_count"]
}

Example Query:

curl -X GET "http://localhost:9527/api/v1/metrics?from=2024-01-29T09:00:00&to=2024-01-29T10:00:00&interval=1m&job_id=<job-id>"

Logs API​

Search a job's logs in VictoriaLogs:

Endpoint:

GET /api/v1/logs/search

Query Parameters:

  • jobId (required): Job ID (UUID)
  • taskId: Limit results to one task (UUID)
  • from, to: Time range (ISO 8601 date-time)
  • limit: Maximum log lines to return (default 1000)

The response is newline-delimited JSON, one log entry per line.

Job Status API​

Manage and monitor jobs:

List All Jobs:

GET /api/v1/jobs?page=0&size=20

Get Job Details:

GET /api/v1/jobs/{jobId}

{jobId} is the Grid job ID (a UUID).

Response:

{
  "id": "018f9a2b-1c3d-7e4f-8a9b-0c1d2e3f4a5b",
  "externalId": "workflow-abc",
  "createdAt": "2024-01-29T09:00:00Z",
  "replicaRunTimeoutSeconds": 0,
  "taskCounters": {
    "readyTasks": 0,
    "scheduledTasks": 2,
    "runningTasks": 5,
    "completedTasks": 100,
    "failedTasks": 3,
    "scheduleTimeout": 0,
    "runTimeoutTasks": 0
  }
}

List Job Tasks:

GET /api/v1/jobs/{jobId}/tasks?page=0&size=50

Stop Job:

POST /api/v1/jobs/stop/{jobIdOrExternalId}

Kill Job:

DELETE /api/v1/jobs/kill/{jobIdOrExternalId}

Both endpoints accept the job ID or its external ID. Both kill every task of the job on its workers and mark all unfinished tasks COMPLETED, so they are not retried. Stop is what the UI's Stop Job button calls: it returns 204 No Content, and if marking one task COMPLETED fails it logs the error and moves on to the next task, where kill stops. Kill returns 200 OK.

Task State Tracking​

Grid keeps a record of every state transition for each task, following a well-defined state machine.

State Machine​

READY ──→ SCHEDULED ──→ RUNNING ──→ COMPLETED
  │           │            │
  │           └────────────┼──→ FAILED ──→ READY (if retries remain)
  │                        └──→ RUN_TIMEOUT
  └──→ SCHEDULE_TIMEOUT

A SCHEDULED task can also move straight to COMPLETED, FAILED, or RUN_TIMEOUT when its worker reports that state before it is seen running.

State Descriptions:

  • READY: Task is queued and waiting for worker assignment
  • SCHEDULED: Task assigned to worker but not yet started
  • RUNNING: Task is actively executing on worker
  • COMPLETED: Task finished successfully
  • FAILED: Task encountered an error, or its worker stopped reporting on it in time (may retry)
  • SCHEDULE_TIMEOUT: Task was not scheduled within the allowed time (terminal)
  • RUN_TIMEOUT: Task exceeded its run time limit (terminal)

Retry Logic​

Failed tasks are automatically retried based on the task's retry count:

Retry Strategy:

  • Each task carries a retry count set at submission. Jobs submitted through the API default to 3 retries; the JobMaster UI uses the file's task.retryCount, or 5 if it is not set
  • JobMaster periodically checks FAILED tasks and resets them to READY for rescheduling
  • A task is retried while the number of retries already made is less than its retry count
  • Once retries are used up, the task stays FAILED
tip

The chart ships no Grafana alert rules. If you need to know about failures without watching the dashboards, set up alerts in Grafana on the metrics above.

info

For questions about monitoring configuration or troubleshooting, contact support@fleak.ai