Scheduler Operations¶

Operational guide for the MeshFlows Scheduler service: architecture, monitoring, and troubleshooting.

Internal reference: internal/scheduler-operations.md.

Replace meshflows-<env> with your target namespace (e.g. meshflows-dev, meshflows-lab). Local minikube uses meshflows.

Architecture¶

Services involved¶

Scheduler (port 8089): APScheduler managing cron jobs
Orchestrator (port 8080/8083): Executes workflows triggered by scheduler
Dashboard: Shows scheduled jobs and last run status
Storage / runtime store: Loads workflow and schedule definitions

Data flow¶

YAML schedule files (/app/flows/schedules/)
         ↓
Scheduler pod (startup)
    ├─ Load YAML: heartbeat-every-minute.yaml
    ├─ Register APScheduler cron job: "* * * * *"
    └─ Thread pool waiting...
         ↓
[Each minute at :00s]
    ├─ APScheduler ThreadPool triggers
    ├─ _trigger_workflow_job() wrapper
    ├─ POST /invoke/scheduled (+ Bearer token)
         ↓
    Orchestrator validates:
    ├─ SCHEDULE_INVOCATION_TOKEN check
    ├─ Workflow access (invocation.type contains schedule)
    ├─ Payload size limits
         ↓
    Workflow execution → trace + trigger events
         ↓
Dashboard shows Last Run timestamp

How it works¶

Startup¶

Schedules load only at startup. To reload after adding YAML files:

Restart the scheduler pod, or
Call POST /admin/reload-schedules (requires reload token)

YAML schedule definition¶

File: flows/schedules/heartbeat-every-minute.yaml

name: heartbeat-every-minute
workflow: heartbeat_timestamp
cron_expression: "* * * * *"     # minute hour day month weekday
payload: "<heartbeat/>"
description: "Heartbeat schedule every minute"

Cron format: minute hour day month weekday

* = every value
0 9 * * 1-5 = 9:00 AM weekdays
*/5 * * * * = every 5 minutes

Logging¶

Scheduler logs¶

Event	Log message
Schedule loaded	`schedule loaded from yaml: name='…' workflow='…' cron='…'`
Trigger succeeded	`triggered workflow=… via schedule job_id=…`
Trigger failed	`trigger failed: workflow=… status=401 …`

export NAMESPACE=meshflows-lab

kubectl logs -n "$NAMESPACE" deployment/scheduler -f --timestamps=true
kubectl logs -n "$NAMESPACE" deployment/scheduler --tail=50 | grep "triggered"

Orchestrator logs¶

kubectl logs -n "$NAMESPACE" deployment/orchestrator -f --timestamps=true
kubectl logs -n "$NAMESPACE" deployment/orchestrator -f | grep "request_id.*scheduler"

Traces and trigger events¶

kubectl exec -n "$NAMESPACE" deployment/orchestrator -- \
  curl -s http://localhost:8083/api/traces?workflow=heartbeat_timestamp | jq .

kubectl exec -n "$NAMESPACE" deployment/orchestrator -- \
  curl -s http://localhost:8083/api/trigger-events?workflow=heartbeat_timestamp | jq .

Monitoring¶

Scripts: engine/scripts/monitor-scheduler.sh (Linux/Kubernetes), engine/scripts/monitor-scheduler.ps1 (Windows).

bash ./engine/scripts/monitor-scheduler.sh logs
bash ./engine/scripts/monitor-scheduler.sh jobs
bash ./engine/scripts/monitor-scheduler.sh diagnostics

Key checks:

# YAML schedules loaded?
kubectl exec -n "$NAMESPACE" deployment/scheduler -- \
  curl -s http://localhost:8089/schedules | jq '.[] | {name, cron, last_triggered_at, last_trigger_status}'

# Scheduler triggering?
kubectl logs -n "$NAMESPACE" deployment/scheduler -f | grep "triggered"

# Orchestrator receiving?
kubectl logs -n "$NAMESPACE" deployment/orchestrator -f | grep "invoke/scheduled"

Troubleshooting¶

401 Unauthorized on `/invoke/scheduled`¶

Cause: SCHEDULE_INVOCATION_TOKEN missing in orchestrator or token mismatch.

kubectl get deploy orchestrator -n "$NAMESPACE" \
  -o jsonpath="{.spec.template.spec.containers[0].env[?(@.name=='SCHEDULE_INVOCATION_TOKEN')]}"

kubectl get deploy scheduler -n "$NAMESPACE" \
  -o jsonpath="{.spec.template.spec.containers[0].env[?(@.name=='SCHEDULE_INVOCATION_TOKEN')]}"

kubectl get secret meshflows-schedule-token -n "$NAMESPACE" -o jsonpath="{.data.token}" | base64 -d

Fix: Ensure both deployments use the same secret (meshflows-schedule-token), then restart:

kubectl rollout restart deployment/orchestrator -n "$NAMESPACE"
kubectl rollout restart deployment/scheduler -n "$NAMESPACE"

See Secrets Management.

No scheduled jobs visible (Last Run = "Never")¶

Cause 1: YAML schedules not loaded at startup.

kubectl logs -n "$NAMESPACE" deployment/scheduler | grep "schedule loaded"

If 0 yaml-schedule(s) loaded: ensure files are in /app/flows/schedules/ with .yaml or .yml extension, then restart or reload.

Cause 2: Invalid cron expression.

kubectl logs -n "$NAMESPACE" deployment/scheduler | grep "ongeldige cron"

Dashboard missing Last Run¶

Scheduler has the job? curl …/schedules on scheduler pod
Job firing? scheduler logs for triggered
Orchestrator received? orchestrator logs for invoke/scheduled

Configuration¶

Variable	Service	Purpose
`SCHEDULES_DIR`	scheduler	Path to YAML schedule files (default `/app/schedules`)
`SCHEDULE_INVOCATION_TOKEN`	orchestrator + scheduler	Bearer token for `/invoke/scheduled`
`ORCHESTRATOR_URL`	scheduler	Orchestrator base URL
`SCHEDULER_RELOAD_TOKEN`	scheduler	Token for `POST /admin/reload-schedules`
`LOG_LEVEL`	both	Logging verbosity

Manifests: engine/deploy/k8s/scheduler-deployment.yaml, engine/deploy/k8s/orchestrator-deployment.yaml.

Example workflow: flows/workflows/heartbeat_timestamp.yaml
Example schedule: flows/schedules/heartbeat-every-minute.yaml