Skip to content

Error Handling

Workflows should explicitly model retries, failure branches, and compensation.

Source of truth: engine/services/orchestrator/app/workflow_runner.py (retry_policy, TryWorkflowStep, compensate, run_after, workflow on_error / finally). Field details: Step Contract.

Building blocks

Mechanism Role
retry_policy Per-step (or service_call) retry override for transient failures
try / catch Local recovery on a grouped step; optional rethrow
compensate Compensation steps if this step (or a later step) fails — LIFO unwind
run_after Gate a sibling on prior step status (succeeded, failed, skipped, timedout); status list = OR
on_error Workflow-level handler after the main graph when a failure remains pending
finally Always run after main (+ on_error), before return or re-raise
  1. Classify expected error types.
  2. Retry only transient dependency failures (retry_policy).
  3. Use try/catch for recoverable local failures; set rethrow: true when the run must still fail after catch.
  4. Add compensate where external state is mutated.
  5. Use run_after when a sibling should run after Succeeded, Failed, or both ({step: [succeeded, failed]}).
  6. Use workflow on_error / finally for generic run-level handling (designer lanes under the main pipeline). Default on_error_rethrow: true.
  7. Emit clear logs and trace context for failed paths.

Validation

  • Run failure-path test payloads (trigger path, dashboard, or POST /run/{name}).
  • Verify expected retry and compensation behavior in traces.