Skip to content

Workflows (Flows) — Workflow Definitions & Deployment

The Flows component contains all workflow definitions, connections, policies, and configurations for task automation and integration.

What are Flows?

Flows represent the application logic layer of MeshFlows: - Workflow Definitions — YAML files describing execution steps - Triggers — Start contracts (HTTP, schedule, AMQP, Azure Service Bus, S3) under flows/triggers/ - Connections — Secure credential storage (HTTP, OAuth, FTP, SSH, …) - Policies — Gateway chain policies (throttle, headers, xml_json, oauth_outbound) - Schedules — Cron expressions that start schedule-bound workflows - Settings — Runtime configuration overrides

Flows are version controlled in Git and can be: - Deployed alongside the Engine (integrated deployment) - Deployed independently (as ConfigMaps or directly) - Updated without restarting other Engine services

Workflow Types

HTTP-Triggered Workflows

Public start via a trigger’s gateway_public_path (Gateway /v1/api/...):

curl -X POST http://meshflows/v1/api/minimal
Internal: orchestrator POST /run/{name}. Gateway /v1/run is legacy/dev and may be disabled. See Triggers.

Scheduled Workflows

Schedule trigger (type: schedule) + schedule YAML:

cron_expression: "0 2 * * *"  # Daily at 2 AM
payload: "<root/>"

Event-Driven Workflows

AMQP / RabbitMQ (invocation.type: amqp), Azure Service Bus (azure_servicebus), and S3/MinIO object events (s3). IMAP/SFTP remain service_call steps, not start triggers.

Start Here

Build a New Workflow

  1. First Workflow
  2. Workflow YAML Syntax
  3. Step Types
  4. Script Steps (JS/Python/Bash)
  5. C# Custom Process — .NET logic via capability provider
  6. SOAP over HTTP + XSLT — SOAP without a WSDL adapter
  7. Business events — type: log convention for Loki (not BAM)
  8. HR employee onboarding — ERPNext → NexusID → HarborERP / Ledgerly / Taxline
  9. Integrations catalog — per-system connections and flows
  10. Step Contract (Fields & Options)
  11. Testing

Integrate External Systems

  1. Connections — credentials via $secret:
  2. Deploy Runtime Secrets — create/update secrets, deploy sync, verify
  3. Prefer an existing service_call capability (Step types); copy weatherstack_current_demo when unsure
  4. Need a new connector? Integration end-to-end (provider + connection + workflow)
  5. Transformations
  6. Policies
  7. Examples
  8. Integrations catalog

Run on Schedule or Events

  1. Triggers
  2. Schedules
  3. Deployment
  4. Error Handling

I’m a Developer — What Do I Need to Know?

Getting Started?

  1. First Workflow — Create your first workflow in 5 minutes
  2. Connections & Credentials — Store credentials securely
  3. Deploy Runtime Secrets — $secret: values out of Git, into Storage
  4. Deployment Methods — Deploy to the Engine

Writing Workflows?

Testing & Debugging?

  1. Local Testing — Test locally before deploying
  2. Viewing Traces — Debug execution with Tempo (Forgejo mirror: Observability)

Key Concepts

Terminology

A capability provider is an HTTP service that registers with the capability-registry. Workflows use exclusively type: service_call plus capability_name and path for integrations, transforms, and scripts. Built-in control/context steps remain orchestrator-owned. Do not add new first-class integration types.

Workflow Structure

name: my-workflow
# Start contracts live in flows/triggers/ — not as first-class HTTP step types

steps:
  - id: fetch
    type: service_call
    capability_name: egress.http.call
    capability_version: "^1.0"
    path: /call
    connection: my_api
    method: GET
    url: /data

  - id: store
    type: service_call
    capability_name: egress.s3.put
    capability_version: "^1.0"
    path: /put
    connection: minio
    bucket: object-demo
    key: archive/{id}.json
    payload_from: fetch

Use id: on every step (not name:).

Step Types

Type Purpose
service_call Integrations, transforms, scripts (via capability registry)
context_set / context_bind / extract Context and bound documents
json_set / xml_set_text In-document mutations
if / try / for_each / repeat_until Branching and loops
parallel Concurrent scopes (not flat steps)
collect / wait / workflow_call Merge, delay, nested workflow
log / terminate Logging and controlled stop

Full list: Step Types. Demos: weatherstack_current_demo, transform_testing_demo, joke_api_random_demo, variable_binding_demo, storage_minio_demo, soap_http_xslt_demo, business_event_demo, script steps, C# custom process.

Connections (Secure Credentials)

name: my_api
type: http
base_url: https://api.example.com
# credentials via $secret:… — see Connections howto

Policies (Gateway Chains)

name: example-throttle-ip
type: throttle
direction: inbound
config:
  max_requests: 60
  window_seconds: 60
  key_from: ip

Attach via ordered policy_chain on triggers (and global/product/user layers) — not apply_to / required_headers. See Policies.

Common Workflows

1. Data Integration

Fetch data from external API, transform, store:

service_call (egress.http.call) → service_call (transformers.xml2json) → service_call (egress.s3.put)

2. File Processing

Fetch, upload via FTP, persist:

service_call (http) → service_call (ftp.call) → service_call (egress.s3.put)

3. Scheduled Reports

Generate reports on schedule, email results:

Schedule trigger → service_call (transformers.*) → service_call (egress.smtp.call)

4. Event-Driven Processing

React to broker messages:

AMQP / RabbitMQ trigger → process → service_call (egress.rabbitmq.topic.send)

5. Multi-Step Integration

Complex workflow with branching logic:

service_call → context_extract_* → if → parallel (scopes) → compensate on error

Deployment Methods

1. ConfigMap (Development)

Quick deployment for testing:

kubectl create configmap workflows --from-file=workflows/
kubectl create configmap connections --from-file=connections/

Workflows stored in Git; CI validates and deploys without engine rebuild:

  • Push workflow YAML to Git
  • deploy.yml validates headers/triggers on flows push
  • Artifacts uploaded to storage; orchestrator reload triggered
  • Push to main → tst; other branches → dev

See Deployment Methods and CI/CD Pipeline.

3. API Upload (Advanced)

curl -X POST http://storage:8080/internal/upload/workflows/minimal \
  -H "Authorization: Bearer <runtime-admin-token>" \
  -H "Content-Type: application/yaml" \
  --data-binary @flows/workflows/minimal.yaml

Other single-item uploads:

  • POST /internal/upload/connections/{name}
  • POST /internal/upload/policies/{name} (throttle | headers | xml_json | oauth_outbound)
  • POST /internal/upload/schedules/{name}
  • POST /internal/upload/config (runtime settings YAML)

After uploading a workflow, trigger a reload in orchestrator (POST /admin/reload).

Workflow Lifecycle

Edit YAML
   ↓
Local Testing
   ↓
Push to Git
   ↓
CI/CD Validation
   ↓
Deploy to Cluster
   ↓
Monitor in Tempo/Dashboard
   ↓
Update if needed

Common Tasks

Create First Workflow

→ First Workflow How-to

Add a Trigger

→ Triggers How-to

Add Credentials

→ Connections How-to

Deploy Workflows

→ Deployment How-to

Learn All Step Types

→ Step Types Reference

Debug Execution

→ Observability Guide (Forgejo mirror: Observability)

Understand Transformations

→ Transformations Reference

Next Steps

  1. First Time?: Start with First Workflow
  2. Learning: Check out Examples & Recipes
  3. Production: Review Deployment Methods and Security
  4. Advanced: Explore Error Handling & Compensation