Services Overview¶
This page describes the Engine services with a technical focus on the orchestration component.
Terminology¶
A capability provider is an HTTP service that registers with the capability-registry. Workflows use exclusively type: service_call plus capability_name and path for integrations, transforms, and scripts. Built-in control/context steps remain orchestrator-owned. Do not add new first-class integration types.
Service Map¶
| Service | Typical port / DNS | Role |
|---|---|---|
| Gateway | 8080 | External API entrypoint, edge alias routes, auth boundary |
| Orchestrator | 8080 (orchestrator) |
Workflow runtime and execution control plane |
| Capability Registry | 8080 (capability-registry; Compose host publish often 8093) |
Register / heartbeat / capability discovery |
| Transformers | 8081 | XSLT, XML↔JSON, CSV, Liquid via transformers.* |
| Script Execution Service (SES) | 8080 (ses) |
script.execute |
| Variables | 8094 | Stateless variables.* operations |
| Storage | 8086 | Runtime artifacts, secrets, config documents |
| Identity | 8088 | Authentication, authorization, tokens |
| Scheduler | 8089 | Timed and named schedule management |
| Dashboard | (deploy-specific) | Web UI |
| Egress HTTP | 8082 | egress.http.call |
| Egress FTP | 8084 | ftp.call |
| Egress SSH | 8085 | egress.ssh.exec / egress.ssh.sftp |
| Standalone SFTP | (deploy) | sftp.list / download / upload |
| Egress RabbitMQ | 8087 | Publish + optional trigger subscriber |
| Egress IMAP | 8091 | egress.imap.call |
| Egress SMTP | 8090 | egress.smtp.call |
| Egress Google Workspace | 8080 | egress.google_workspace.* |
| Egress S3 | (deploy) | egress.s3.* (e.g. MinIO) |
| Egress PostgreSQL | (deploy) | egress.postgres.query / .execute |
| Egress Weatherstack | (deploy) | egress.weatherstack.* |
| Egress Positionstack | (deploy) | egress.positionstack.* |
| Egress Joke API | (deploy) | egress.joke-api.random |
Ports above are typical k8s Service ports; compose port-forwards may differ (orchestrator often mapped as host 8083 → container 8080).
Orchestration Component (Orchestrator)¶
The Orchestrator is the core execution runtime for MeshFlows workflows.
Responsibilities¶
- Loads workflow, connection, and policy definitions at startup and on reload.
- Validates and executes workflow steps in a deterministic order.
- Enforces invocation constraints (HTTP / schedule / dashboard / AMQP paths).
- Discovers capability providers via
CAPABILITY_REGISTRY_URL(noEGRESS_*_URLreads in orchestrator Python). - Captures run traces, run ledger, and trigger-event records.
- Applies runtime concerns such as retry, idempotency, throttling, and OAuth token handling for connections.
Runtime Flow¶
- Receive invocation via HTTP, enqueue, scheduler, AMQP, or dashboard endpoint.
- Resolve workflow and invocation policy.
- Build trigger context from headers and request metadata.
- Execute workflow steps through the workflow runner (
service_call→ registry → provider). - Persist trace / ledger / trigger-event data.
- Return result (sync) or
202+ queue ack path (async).
Key Interfaces¶
Health and readiness:
GET /healthzGET /readyz
Workflow discovery:
GET /workflowsGET /workflows/httpGET /workflows/{workflow_name}GET /triggers
Execution entrypoints:
POST /run/{workflow_name}POST /run/{workflow_name}/enqueuePOST /invoke/scheduledPOST /invoke/amqpPOST /invoke/amqp/dispatchPOST /invoke/dashboard
Administration:
POST /admin/validate?strict=truePOST /admin/reload
Runs & observability:
GET /api/runs*GET /api/traces*GET /api/trigger-events*
Full detail: Orchestrator API.
Reliability Controls¶
- Idempotency support for duplicate-request protection.
- Throttling controls for rate and in-flight concurrency.
- Retry behavior is workflow-driven.
- Trace masking and secret-aware handling reduce sensitive-data leakage.
Security and Access¶
- HTTP workflow invocation can use OAuth2/JWT scope checks and optional bearer-token fallback.
- Scheduled / dashboard paths have dedicated auth configuration.
- Admin endpoints are protected by reload token configuration.
- Connection secret references resolve from storage-backed secrets.
Integration with Other Services¶
- Resolves providers through the capability registry (HTTP, transformers, SES, S3, GWS, …).
- Uses Storage for runtime config resolution and secret retrieval.
- Uses RabbitMQ + Postgres + MinIO for async messaging and run persistence when enabled.
- Proxies schedule management operations to the Scheduler service.
- Executes workflows started by external trigger adapters (Gateway, Scheduler, RabbitMQ subscriber, SFTP/IMAP triggers as deployed).
Operational Notes¶
- Keep
WORKFLOWS_DIR,WORKFLOW_ARTIFACTS_DIR,CONNECTIONS_DIR, andPOLICIES_DIRmounted and versioned. - Set
CAPABILITY_REGISTRY_URLon orchestrator (and workers). - Use
POST /admin/validate?strict=truebefore rollout. - Use
POST /admin/reloadafter updating flows/connections/policies. - Track run behavior with
/api/runsand/api/traces; correlate withX-Request-ID.