Skip to content

Backend (Engine) — Microservices Infrastructure

The Engine is the core infrastructure and microservices layer of MeshFlows.

For how this differs from UI-first automation tools or embedded in-app workflow engines, see Why MeshFlows.

What is the Engine?

The Engine provides all the operational infrastructure needed to: - Execute workflows — YAML-driven workflow orchestration - Handle requests — Gateway for public API access and alias route mapping - Transform data — XSLT, XML↔JSON, CSV, Liquid templates - Store state — Runtime key-value store & document management - Authenticate users — JWT tokens, OAuth 2.0, identity management - Schedule tasks — CronJob-based workflow triggers - Manage integrations — HTTP, FTP, SSH, RabbitMQ connections - Monitor everything — Observability stack (Tempo, Loki, Prometheus)

Services at a Glance

Service Purpose
Gateway Public API entry point
Orchestrator Workflow execution (orchestrator:8080); discovers providers via registry
Capability Registry Register / heartbeat / /v1/registry/capabilities
Transformers transformers.* via service_call
SES / Variables script.execute, variables.*
Storage Artifacts, secrets, runtime documents
Identity User management & JWT
Scheduler Cron / named schedule triggers
Dashboard Web UI for workflows
Capability providers HTTP, FTP, SSH/SFTP, SMTP/IMAP, RabbitMQ, S3, Postgres, Weatherstack, Positionstack, Joke API, Google Workspace, …
RabbitMQ Message broker

Orchestrator does not use EGRESS_*_URL for discovery — only CAPABILITY_REGISTRY_URL. See Services Overview.

Start Here

New Platform Setup

  1. Prerequisites
  2. Kubernetes Deployment — or AKS Deployment
  3. Secrets Management — platform / CI; for $secret: see Runtime Secrets
  4. Operations

Local Development

  1. Local Development
  2. Develop and Test
  3. Longhorn Storage Setup
  4. Flows Overview (Forgejo mirror: Flows Overview)

API and Runtime Internals

  1. Architecture
  2. Services Overview
  3. Gateway API
  4. Edge routes (implementation)
  5. Trigger policy-chain redesign plan
  6. Orchestrator API
  7. Orchestrator Messaging Blueprint

I’m an Operator — What Do I Need to Know?

First Time Setup?

  1. Prerequisites — Get your environment ready
  2. Kubernetes Deployment — Deploy to your cluster
  3. Operations & Monitoring — Run and manage the system
  4. Forgejo Container Package Cleanup — Registry retention (keep vX.Y.Z)

Local Development?

  1. Local Development (Docker Compose)
  2. Secrets Management

Going Deeper?

Deployment Options

Docker Compose (Development/Testing)

cd engine
docker-compose up -d
All services + RabbitMQ + observability stack in one command.

Kubernetes (Production)

cd engine/deploy
kubectl apply -k overlays/linux-cluster/
Full production-ready deployment with persistence, networking, and monitoring.

Azure Kubernetes Service (AKS)

cd engine/deploy
kubectl apply -k overlays/aks/ --load-restrictor=LoadRestrictionsNone
Standalone MeshFlows on AKS (Azure Disk CSI). → AKS Deployment

Minikube (Learning)

# From repo root — preferred
./deploy-minikube.ps1
# Or: kubectl apply -k engine/deploy/overlays/minikube/
Simplified deployment for learning and testing. Legacy path engine/deploy/k8s/overlays/minikube/ is a deprecation stub — use apply-via-canonical.sh or the canonical overlay directly.

Key Features

  • YAML Workflows — Declarative workflow definitions
  • Parallel Execution — Concurrent step execution
  • Retry & Compensation — Automatic failure recovery
  • Data Transformations — transformers.* capability providers
  • Integrations — registry-driven service_call (HTTP, FTP, SSH/SFTP, RabbitMQ, S3, …)
  • Edge Route Aliases — Public endpoint-to-workflow mapping in Gateway
  • OAuth 2.0 — Inbound and outbound authentication
  • Observability — OpenTelemetry + Tempo + Loki
  • Storage ACLs — Role-based access control
  • Scheduler — CronJob or HTTP-driven triggers
  • Dashboard — Visual workflow editor

Technology Stack

  • Language: Python 3.11+ (FastAPI)
  • Container: Docker
  • Orchestration: Kubernetes (Kustomize)
  • Storage: Longhorn persistent volumes
  • Message Broker: RabbitMQ
  • Observability: OpenTelemetry + Tempo + Loki + Prometheus
  • Testing: pytest
  • CI/CD: Forgejo Actions + container image build pipeline

Common Tasks

Deploy to Production

→ Kubernetes Deployment How-to

Configure Secrets

→ Deploy Runtime Secrets — $secret: / Dashboard / Storage
→ Secrets Management (platform / CI) — Forgejo → Kubernetes

Setup Longhorn Storage

→ Longhorn Storage How-to

Monitor with Tempo/Loki

→ Observability Reference

Secure Your Deployment

→ Security & Hardening Reference

Understand Our CI/CD Pipeline

→ CI/CD Pipeline Architecture — How builds, tests, and deployments work

Clean up Forgejo container packages

→ Forgejo Container Package Cleanup — what to keep (vX.Y.Z, recent build tags) and how to delete -pr/-dev without breaking deploy

Capability providers

→ Integration end-to-end — new provider + connection + workflow + trigger
→ Add a Capability Provider — Python, Go, and C# starters
→ Capability provider contract — registration, designer metadata, UI ownership
→ Deploy integration — images, kustomize, CI checklist

Internal docs (repo only, not in wiki)

Topic Location Why not published
Edge control backlog / tickets internal/edge-control-layer-*.md (except sprint1 guide, now reference/20) Planning/backlog
Script Execution Service design internal/script-execution-service-*.md Superseded by Script Steps + engine/services/script-execution-service/README.md
Platform proposals internal/proposals/ Pre-implementation proposals
AI scratch prompts internal/meshflows-from-scratch-prompt* Not operator/developer documentation
Connector issue backlog internal/codeberg-issue-requests-connectors.md Internal tracker

Develop and Test Locally

→ Develop and Test How-to — Local setup through CI promotion

Understand OAuth

→ Authentication Reference

Next Steps

  1. Operators/DevOps: Start with Prerequisites
  2. Developers: Check out Flows (Workflows) documentation (Forgejo mirror: Flows (Workflows))
  3. Learning: Try Local Development