Skip to content

Services Overview

This page describes the Engine services with a technical focus on the orchestration component.

Terminology

A capability provider is an HTTP service that registers with the capability-registry. Workflows use exclusively type: service_call plus capability_name and path for integrations, transforms, and scripts. Built-in control/context steps remain orchestrator-owned. Do not add new first-class integration types.

Service Map

Service Typical port / DNS Role
Gateway 8080 External API entrypoint, edge alias routes, auth boundary
Orchestrator 8080 (orchestrator) Workflow runtime and execution control plane
Capability Registry 8080 (capability-registry; Compose host publish often 8093) Register / heartbeat / capability discovery
Transformers 8081 XSLT, XML↔JSON, CSV, Liquid via transformers.*
Script Execution Service (SES) 8080 (ses) script.execute
Variables 8094 Stateless variables.* operations
Storage 8086 Runtime artifacts, secrets, config documents
Identity 8088 Authentication, authorization, tokens
Scheduler 8089 Timed and named schedule management
Dashboard (deploy-specific) Web UI
Egress HTTP 8082 egress.http.call
Egress FTP 8084 ftp.call
Egress SSH 8085 egress.ssh.exec / egress.ssh.sftp
Standalone SFTP (deploy) sftp.list / download / upload
Egress RabbitMQ 8087 Publish + optional trigger subscriber
Egress IMAP 8091 egress.imap.call
Egress SMTP 8090 egress.smtp.call
Egress Google Workspace 8080 egress.google_workspace.*
Egress S3 (deploy) egress.s3.* (e.g. MinIO)
Egress PostgreSQL (deploy) egress.postgres.query / .execute
Egress Weatherstack (deploy) egress.weatherstack.*
Egress Positionstack (deploy) egress.positionstack.*
Egress Joke API (deploy) egress.joke-api.random

Ports above are typical k8s Service ports; compose port-forwards may differ (orchestrator often mapped as host 8083 → container 8080).

Orchestration Component (Orchestrator)

The Orchestrator is the core execution runtime for MeshFlows workflows.

Responsibilities

  • Loads workflow, connection, and policy definitions at startup and on reload.
  • Validates and executes workflow steps in a deterministic order.
  • Enforces invocation constraints (HTTP / schedule / dashboard / AMQP paths).
  • Discovers capability providers via CAPABILITY_REGISTRY_URL (no EGRESS_*_URL reads in orchestrator Python).
  • Captures run traces, run ledger, and trigger-event records.
  • Applies runtime concerns such as retry, idempotency, throttling, and OAuth token handling for connections.

Runtime Flow

  1. Receive invocation via HTTP, enqueue, scheduler, AMQP, or dashboard endpoint.
  2. Resolve workflow and invocation policy.
  3. Build trigger context from headers and request metadata.
  4. Execute workflow steps through the workflow runner (service_call → registry → provider).
  5. Persist trace / ledger / trigger-event data.
  6. Return result (sync) or 202 + queue ack path (async).

Key Interfaces

Health and readiness:

  • GET /healthz
  • GET /readyz

Workflow discovery:

  • GET /workflows
  • GET /workflows/http
  • GET /workflows/{workflow_name}
  • GET /triggers

Execution entrypoints:

  • POST /run/{workflow_name}
  • POST /run/{workflow_name}/enqueue
  • POST /invoke/scheduled
  • POST /invoke/amqp
  • POST /invoke/amqp/dispatch
  • POST /invoke/dashboard

Administration:

  • POST /admin/validate?strict=true
  • POST /admin/reload

Runs & observability:

  • GET /api/runs*
  • GET /api/traces*
  • GET /api/trigger-events*

Full detail: Orchestrator API.

Reliability Controls

  • Idempotency support for duplicate-request protection.
  • Throttling controls for rate and in-flight concurrency.
  • Retry behavior is workflow-driven.
  • Trace masking and secret-aware handling reduce sensitive-data leakage.

Security and Access

  • HTTP workflow invocation can use OAuth2/JWT scope checks and optional bearer-token fallback.
  • Scheduled / dashboard paths have dedicated auth configuration.
  • Admin endpoints are protected by reload token configuration.
  • Connection secret references resolve from storage-backed secrets.

Integration with Other Services

  • Resolves providers through the capability registry (HTTP, transformers, SES, S3, GWS, …).
  • Uses Storage for runtime config resolution and secret retrieval.
  • Uses RabbitMQ + Postgres + MinIO for async messaging and run persistence when enabled.
  • Proxies schedule management operations to the Scheduler service.
  • Executes workflows started by external trigger adapters (Gateway, Scheduler, RabbitMQ subscriber, SFTP/IMAP triggers as deployed).

Operational Notes

  • Keep WORKFLOWS_DIR, WORKFLOW_ARTIFACTS_DIR, CONNECTIONS_DIR, and POLICIES_DIR mounted and versioned.
  • Set CAPABILITY_REGISTRY_URL on orchestrator (and workers).
  • Use POST /admin/validate?strict=true before rollout.
  • Use POST /admin/reload after updating flows/connections/policies.
  • Track run behavior with /api/runs and /api/traces; correlate with X-Request-ID.