Error Handling¶
Workflows should explicitly model retries, failure branches, and compensation.
Source of truth: engine/services/orchestrator/app/workflow_runner.py (retry_policy, TryWorkflowStep, compensate, run_after, workflow on_error / finally). Field details: Step Contract.
Building blocks¶
| Mechanism | Role |
|---|---|
retry_policy |
Per-step (or service_call) retry override for transient failures |
try / catch |
Local recovery on a grouped step; optional rethrow |
compensate |
Compensation steps if this step (or a later step) fails — LIFO unwind |
run_after |
Gate a sibling on prior step status (succeeded, failed, skipped, timedout); status list = OR |
on_error |
Workflow-level handler after the main graph when a failure remains pending |
finally |
Always run after main (+ on_error), before return or re-raise |
Recommended approach¶
- Classify expected error types.
- Retry only transient dependency failures (
retry_policy). - Use
try/catchfor recoverable local failures; setrethrow: truewhen the run must still fail after catch. - Add
compensatewhere external state is mutated. - Use
run_afterwhen a sibling should run after Succeeded, Failed, or both ({step: [succeeded, failed]}). - Use workflow
on_error/finallyfor generic run-level handling (designer lanes under the main pipeline). Defaulton_error_rethrow: true. - Emit clear logs and trace context for failed paths.
Validation¶
- Run failure-path test payloads (trigger path, dashboard, or
POST /run/{name}). - Verify expected retry and compensation behavior in traces.