Skip to content

Engine Internals: How Workflows Start

This page explains the main workflow start paths:

  • HTTP sync (/run/{workflow_name})
  • HTTP async enqueue (/run/{workflow_name}/enqueue)
  • Scheduler (/invoke/scheduled)
  • AMQP / queue (/invoke/amqp, /invoke/amqp/dispatch, workers)
  • Dashboard (/invoke/dashboard)

1. Scheduler → Orchestrator Start Path

  1. Scheduler stores and evaluates cron jobs.
  2. When a job fires, Scheduler calls Orchestrator scheduled-invocation endpoint.
  3. Orchestrator validates scheduled auth (SCHEDULE_INVOCATION_TOKEN and optional OAuth mode).
  4. Orchestrator resolves workflow, trigger context, runtime config, and cursor state.
  5. run_workflow() executes step-by-step in workflow_runner.py.
  6. Trace and trigger-event records are persisted.
  7. Result is returned to Scheduler caller.

Key endpoint:

  • POST /invoke/scheduled

2. HTTP → Orchestrator Start Path

  1. Client calls Gateway route for a workflow.
  2. Gateway enforces edge auth/policy and proxies to Orchestrator.
  3. Orchestrator resolves workflow by URL path (/run/{workflow_name}).
  4. Invocation auth is validated (HTTP_INVOCATION_TOKEN, per-workflow token, or OAuth scopes).
  5. Input payload and request context are normalized.
  6. run_workflow() executes all steps and applies retry/throttle/idempotency rules.
  7. Orchestrator returns final response and X-Request-ID for correlation.

Key endpoints:

  • POST /run/{workflow_name} (sync)
  • POST /run/{workflow_name}/enqueue (async 202; requires messaging env)

Body-selected POST /run is deprecated / unsupported as the public contract — always use path-based /run/{name} (or gateway aliases).

3. AMQP / Queue Start Path

Two related patterns:

Trigger subscriber → Orchestrator

  1. rabbitmq-trigger-subscriber (egress-rabbitmq image role) consumes broker messages.
  2. Subscriber validates the event and calls POST /invoke/amqp (or the dispatch path as configured).
  3. Orchestrator authorizes the AMQP path, loads the target workflow, and runs (or enqueues) it.

Orchestrator enqueue → workers

  1. Client (or gateway) calls POST /run/{name}/enqueue.
  2. API pod builds a command envelope; large payloads offload to storage (payload_ref).
  3. Message is published to ORCH_MESSAGE_WORK_EXCHANGE (run.small / run.large).
  4. orchestrator-worker-small / orchestrator-worker-large consume and execute via the same run_workflow() path.
  5. Run ledger updates go to Postgres when ORCH_MESSAGE_DATABASE_DSN is set.

Key env: ORCH_MESSAGE_RABBITMQ_URL, ORCH_MESSAGE_*, ORCH_ASYNC_QUEUE_ENABLED.

4. Dashboard Start Path

  1. Dashboard IAM checks user scope.
  2. Dashboard calls POST /invoke/dashboard.
  3. Orchestrator validates dashboard token / OAuth and executes the workflow.

Why Step Classes Live in Orchestrator

The step classes in workflow_runner.py are orchestrator execution contracts, not microservice internals.

  • Orchestrator owns workflow schema validation and step dispatch for built-in types (context_*, if, parallel, …) plus the generic service_call.
  • Capability providers own protocol-specific I/O (HTTP, Gmail, S3, transforms, SES, …).
  • A step class defines what workflow YAML is valid and how orchestration chooses the next action.

For Google Workspace specifically:

  • There are no first-class gmail_* / calendar_* / drive_* step types.
  • Workflows use type: service_call with capability_name: egress.google_workspace.* and the provider path (e.g. /gmail/list).
  • Discovery is via the capability registry — not an orchestrator EGRESS_GOOGLE_WORKSPACE_URL code path.

This split keeps orchestration deterministic while isolating external protocol complexity.

Reload and Runtime Data Flow

  1. Flows/connections/policies are uploaded to Storage.
  2. POST /admin/reload in Orchestrator re-pulls runtime data from the configured runtime store.
  3. In-memory workflow/connection/policy snapshots are atomically swapped under reload lock.

If reload fails, check:

  • connection schema compatibility in runtime store
  • unresolved secret refs
  • storage API health and payload validity
  • capability registry reachability for service_call validation (?strict=true)

Orchestrator vs RabbitMQ Trigger Subscriber

MeshFlows splits execution from trigger ingestion:

  • orchestrator: API-facing execution runtime for HTTP, schedule, dashboard, and AMQP invoke endpoints.
  • rabbitmq-trigger-subscriber: RabbitMQ ingress adapter (egress-rabbitmq image) that consumes queue/topic messages and calls Orchestrator.

This keeps Orchestrator focused on workflow execution and isolates broker-consumer behavior.

Why the Logs Look the Same

  • Both are Python/FastAPI services and can emit similar infra logs (httpx, probes, OTEL).
  • The queue subscriber no longer hosts Orchestrator workflow runtime internals.

So seeing similar lines in both pods is expected behavior, not necessarily a duplicate execution bug.

What Actually Differs

The main difference is role configuration (environment variables), not code identity.

  • API orchestrator focuses on endpoint-driven execution (/run/{name}, /invoke/*).
  • Queue subscriber focuses on RabbitMQ consume/ack/nack and forwards valid events to Orchestrator.

Queue subscriber deployment reference:

  • engine/deploy/k8s/rabbitmq-trigger-subscriber-deployment.yaml

Practical Log Differentiation

To reduce operator confusion:

  1. Filter by pod/deployment name (orchestrator-* vs rabbitmq-trigger-subscriber-*).
  2. Filter by Kubernetes component labels.
  3. Use different log levels per role (for example INFO on queue subscriber, DEBUG only when debugging).
  4. Optionally add an explicit role field in log context (api vs queue-subscriber).