Skip to main content

Job Submission

Learn how to submit and manage jobs in Grid's on-premises deployment using the JobMaster UI. This guide covers accessing the interface, viewing cluster workers, submitting DAG configurations, and monitoring job and task execution.

Accessing the JobMaster UI​

Grid's on-premises deployment includes a web-based interface for job submission and management. The JobMaster service runs on port 9527 and can be accessed locally using Kubernetes port forwarding.

Resource Names​

The Grid Helm chart prefixes every resource it creates (services, secrets, the worker StatefulSet) with a name derived from your Helm release. The commands on this page and on Monitoring and Debugging write that prefix as <release>. Work it out like this:

  • If your release name contains grid (for example grid or prod-grid), the prefix is the release name itself: grid-jobmaster, prod-grid-jobmaster.
  • Otherwise the chart appends -grid: a release named prod gives prod-grid-jobmaster.
  • If you set fullnameOverride in your Helm values, the prefix is that value.

To confirm the prefix, list the services in your namespace:

kubectl get svc -n <namespace>

Expose JobMaster Locally​

Use kubectl port-forward to access the JobMaster UI from your local machine:

kubectl port-forward -n <namespace> svc/<release>-jobmaster 9527:9527

Replace <namespace> with your Grid deployment namespace and <release> with the prefix described in Resource Names.

Example:

# Release "grid" in the "grid" namespace
kubectl port-forward -n grid svc/grid-jobmaster 9527:9527

# Release "prod" in the default namespace
kubectl port-forward svc/prod-grid-jobmaster 9527:9527

Output:

Forwarding from 127.0.0.1:9527 -> 9527
Forwarding from [::1]:9527 -> 9527
tip

Keep the terminal window open while using the UI. Closing the terminal or pressing Ctrl+C will stop the port forwarding.

Access the UI in Browser​

Once port forwarding is active, open your browser and navigate to:

http://localhost:9527

The sidebar has three views:

  • Cluster - Worker list and health
  • Jobs - Monitor and manage submitted jobs
  • Submit - Upload and submit DAG configurations

Cluster View​

The Cluster View lists all workers registered with your Grid deployment. Use it to check which workers are healthy before submitting jobs.

Grid Cluster View

Worker Information​

Each worker displays:

  • Worker ID: Unique identifier for the worker instance
  • URL: Network endpoint for the worker service
  • Max Tasks: Maximum number of concurrent tasks the worker can handle
  • State: Worker state indicator
    • OPERATIONAL: Worker is healthy and accepting tasks
    • UNSTABLE: Worker is experiencing intermittent issues
    • MAXED: Worker has reached its task capacity limit
    • DOWN: Worker is not responding to heartbeats
  • Last Registered: Timestamp of the most recent heartbeat
  • Created At: When the worker first joined the cluster

The list shows 10 workers per page. It does not refresh on its own; reopen the view to see current state.

Capacity Planning​

Use the cluster view to:

  • Verify worker availability before submitting large jobs
  • Identify unhealthy workers (UNSTABLE or DOWN) that may need attention
  • Spot saturated workers in the MAXED state
  • Plan job parallelism based on the total task capacity of your workers

Example Calculation: If you have 3 OPERATIONAL workers with Max Tasks of 10 each, your cluster can run up to 30 concurrent tasks. The view does not show how many tasks each worker is running right now; a worker that has reached its limit shows as MAXED.

The Fleak deploy dialog caps a workflow's replicas at the cluster's total task slots (workers × task max).

info

Worker capacity is configurable via Helm values. Adjust worker.task.max based on worker resource limits (CPU/memory).

Submit Job View​

The Submit Job view allows you to upload and execute DAG (Directed Acyclic Graph) configuration files. Grid processes these DAG files to create distributed data pipeline jobs.

Grid Submit Job View

Submitting a DAG File​

Step 1: Prepare Your DAG Configuration

Create a YAML file defining your data pipeline. The file must have a top-level dag field containing the list of nodes.

Step 2: Upload the DAG File

  1. Click the "Choose File" button
  2. Select your DAG YAML file (.yaml or .yml) from your local filesystem
  3. The selected filename appears below the button

Step 3: Submit the Job

  1. Click the "Submit" button
  2. If the file has no dag list, or Grid rejects the job, an error message displays with details
  3. Otherwise the job is created

Step 4: Confirmation

Upon successful submission, you'll see:

  • A success message with the Job ID and External ID
  • The job appears in the Jobs View
  • Tasks are automatically scheduled to available workers

Jobs View​

The Jobs View displays all submitted jobs with their current status and task breakdowns. This is your primary interface for monitoring job execution.

Job List​

Each job entry shows:

  • ID: Unique Grid-assigned identifier (a UUID, e.g., 018f9a2b-1c3d-7e4f-8a9b-0c1d2e3f4a5b)
  • External ID: Your custom identifier (if provided during submission)
  • Created: Job submission timestamp
  • Status: Color-coded job state, worked out from the task counters
    • RUNNING (blue): Job has active tasks (ready, scheduled, or running) and no failures
    • COMPLETED (green): All tasks finished successfully
    • FAILED (red): Tasks failed and none are still active
    • PARTIAL_FAILURE (amber): Some tasks failed while others are still active
    • TIMEOUT (amber): Tasks exceeded their schedule or run time limits
    • MIXED (gray): Task states don't fall into any other category
    • EMPTY (gray): Job has no tasks
  • Task Counters: Breakdown of task states
    • Ready: Tasks queued, waiting for worker assignment
    • Scheduled: Tasks assigned to workers but not started
    • Running: Tasks currently executing
    • Completed: Successfully finished tasks
    • Failed: Tasks that encountered errors
    • Timeouts: Tasks that timed out waiting for a worker or exceeded their maximum runtime
    • Total: All tasks in the job

Pagination and Refresh​

  • Jobs are listed most recent first, 10 per page
  • Use First, Previous, Next, and Last to move between pages
  • The list refreshes automatically every 10 seconds

Job Actions​

From the Job List:

  • View Details: Click on a job row to open the detailed view

Job and per-task termination actions are available in the Job Detail View.

Job Detail View​

Click on any job in the Jobs View to access detailed information about its execution.

Grid Job Detail View

Job Summary​

The top section displays aggregated job information:

Status Overview:

  • Job ID and External ID
  • Creation timestamp

Task Summary: Task counts by state:

  • Total: All tasks in the job
  • Ready: Tasks in queue
  • Running: Tasks actively executing
  • Completed: Successfully finished tasks
  • Failed: Tasks with errors

If the Replica Startup Monitor stopped the job because not all replicas reached RUNNING within the job's replica timeout, a banner says so.

Task List: Below the summary, each task shows its ID, External ID, Status, Retry Count, Max Schedule Time, Max Run Time, Created and Last Updated times, and a Kill Task button. Click a task row to open the Task Details View.

Job-Level Actions​

Stop Job: Terminates all running tasks in the job and prevents retries:

  1. Click the "Stop Job" button next to the task summary
  2. Confirm the action in the dialog
  3. All non-terminal tasks transition to COMPLETED state
  4. Job task counters update accordingly

Kill Task: Click "Kill Task" on a task row and confirm to kill that single task.

Use Cases for Stopping Jobs:

  • Job is processing incorrect data
  • Pipeline configuration needs to be changed
  • Resource constraints require stopping work
  • Development/testing iterations
warning

Stopping a job is not reversible. Tasks that were in progress will be terminated and marked as completed. Only stop jobs when necessary.

Real-Time Updates​

The Job Detail View automatically refreshes every 10 seconds to show task state changes and updated counters.

You can manually refresh by reloading the page.

Task Details View​

Click a task row in the Job Detail View to open the Task Details View. It has three tabs.

Grid Task Details View

Task Details:

  • Task ID, Worker ID, and Scheduled At
  • Status and Condition (extra detail on the current state, such as Replica startup timeout)
  • Created At, Last Updated, Max Schedule Time, and Max Runtime
  • A table of the task's pipeline metrics as reported by the worker

Task DAG: A graph of the pipeline DAG this task runs.

Logs: Streams the task's logs from its worker and keeps tailing new lines while the tab is open.

The view reloads the task every 5 seconds while the browser tab is visible.

Best Practices​

Before Submission​

Test Your DAG:

  • Start with a small sample of data
  • Verify all node configurations are correct
  • Test connections to external systems (Kafka, databases)

Right-Size Your Job:

  • Match parallelism to cluster capacity
  • Avoid over-subscribing worker capacity
  • Consider data volume and processing time

Use External IDs:

  • Always provide meaningful external IDs
  • Makes correlation with your system easier
  • Simplifies job tracking and auditing

During Execution​

Monitor Actively:

  • Watch the first few tasks complete successfully
  • Check task logs early to catch configuration issues
  • Monitor resource usage in Kubernetes

Don't Over-Kill:

  • Let automatic retries handle transient errors
  • Only stop jobs for unrecoverable issues
  • Consider stopping and resubmitting for config errors

After Completion​

Review Metrics:

  • Check task execution times
  • Identify slow nodes in the pipeline
  • Look for optimization opportunities

Troubleshooting​

Cannot Access JobMaster UI​

Symptom: Browser shows "Connection refused" at http://localhost:9527

Solutions:

  1. Verify port forwarding is active (check terminal window)
  2. Confirm JobMaster pod is running: 
    kubectl get pods -n <namespace> | grep jobmaster
  3. Check JobMaster service exists: 
    kubectl get svc -n <namespace> | grep jobmaster
  4. Verify the namespace and the <release> prefix in the port-forward command (see Resource Names)

Job Submission Fails​

Symptom: Error message after clicking "Submit"

Solutions:

  1. Validate DAG syntax
  2. Check error message for specific validation failures
  3. Verify all node types are supported by your Grid version
  4. Ensure source and sink nodes are present
  5. Check for circular dependencies in edges

Tasks Stuck in READY State​

Symptom: Tasks never transition to SCHEDULED or RUNNING

Solutions:

  1. Check Cluster View for available workers
  2. Verify workers are in OPERATIONAL state
  3. Scale up worker replicas if capacity is full: 
    kubectl scale statefulset <release>-worker -n <namespace> --replicas=5
  4. Check the JobMaster logs for TaskScheduleMonitor errors

High Task Failure Rate​

Symptom: Many tasks in FAILED state

Solutions:

  1. Open the job, click a failed task, and check its Condition and Logs tab
  2. Check for common error patterns
  3. Verify external service connectivity (Kafka, databases)
  4. Review node configurations for errors
  5. Check worker resource limits (CPU/memory)
info

For additional support with job submission or troubleshooting, contact support@fleak.ai