Workflows (Flows) — Workflow Definitions & Deployment¶
The Flows component contains all workflow definitions, connections, policies, and configurations for task automation and integration.
What are Flows?¶
Flows represent the application logic layer of MeshFlows:
- Workflow Definitions — YAML files describing execution steps
- Triggers — Start contracts (HTTP, schedule, AMQP, Azure Service Bus, S3) under flows/triggers/
- Connections — Secure credential storage (HTTP, OAuth, FTP, SSH, …)
- Policies — Gateway chain policies (throttle, headers, xml_json, oauth_outbound)
- Schedules — Cron expressions that start schedule-bound workflows
- Settings — Runtime configuration overrides
Flows are version controlled in Git and can be: - Deployed alongside the Engine (integrated deployment) - Deployed independently (as ConfigMaps or directly) - Updated without restarting other Engine services
Workflow Types¶
HTTP-Triggered Workflows¶
Public start via a trigger’s gateway_public_path (Gateway /v1/api/...):
curl -X POST http://meshflows/v1/api/minimal
POST /run/{name}. Gateway /v1/run is legacy/dev and may be disabled. See Triggers.
Scheduled Workflows¶
Schedule trigger (type: schedule) + schedule YAML:
cron_expression: "0 2 * * *" # Daily at 2 AM
payload: "<root/>"
Event-Driven Workflows¶
AMQP / RabbitMQ (invocation.type: amqp), Azure Service Bus (azure_servicebus), and
S3/MinIO object events (s3). IMAP/SFTP remain service_call steps, not start triggers.
Start Here¶
Build a New Workflow¶
- First Workflow
- Workflow YAML Syntax
- Step Types
- Script Steps (JS/Python/Bash)
- C# Custom Process — .NET logic via capability provider
- SOAP over HTTP + XSLT — SOAP without a WSDL adapter
- Business events —
type: logconvention for Loki (not BAM) - HR employee onboarding — ERPNext → NexusID → HarborERP / Ledgerly / Taxline
- Integrations catalog — per-system connections and flows
- Step Contract (Fields & Options)
- Testing
Integrate External Systems¶
- Connections — credentials via
$secret: - Deploy Runtime Secrets — create/update secrets, deploy sync, verify
- Prefer an existing
service_callcapability (Step types); copyweatherstack_current_demowhen unsure - Need a new connector? Integration end-to-end (provider + connection + workflow)
- Transformations
- Policies
- Examples
- Integrations catalog
Run on Schedule or Events¶
I’m a Developer — What Do I Need to Know?¶
Getting Started?¶
- First Workflow — Create your first workflow in 5 minutes
- Connections & Credentials — Store credentials securely
- Deploy Runtime Secrets —
$secret:values out of Git, into Storage - Deployment Methods — Deploy to the Engine
Writing Workflows?¶
- Workflow YAML Syntax — Complete reference
- Step Types — All available step types
- Step Contract — Required/optional fields and validation rules per step
- Context & Variables — Passing data between steps
- Conditions & Path Expressions — IF branches, JSON/XML paths, service output
- Transformations — XSLT, Liquid, CSV, JSON conversions
- Examples & Recipes — Real-world examples
Testing & Debugging?¶
- Local Testing — Test locally before deploying
- Viewing Traces — Debug execution with Tempo (Forgejo mirror: Observability)
Key Concepts¶
Terminology¶
A capability provider is an HTTP service that registers with the capability-registry. Workflows use exclusively type: service_call plus capability_name and path for integrations, transforms, and scripts. Built-in control/context steps remain orchestrator-owned. Do not add new first-class integration types.
Workflow Structure¶
name: my-workflow
# Start contracts live in flows/triggers/ — not as first-class HTTP step types
steps:
- id: fetch
type: service_call
capability_name: egress.http.call
capability_version: "^1.0"
path: /call
connection: my_api
method: GET
url: /data
- id: store
type: service_call
capability_name: egress.s3.put
capability_version: "^1.0"
path: /put
connection: minio
bucket: object-demo
key: archive/{id}.json
payload_from: fetch
Use id: on every step (not name:).
Step Types¶
| Type | Purpose |
|---|---|
service_call |
Integrations, transforms, scripts (via capability registry) |
context_set / context_bind / extract |
Context and bound documents |
json_set / xml_set_text |
In-document mutations |
if / try / for_each / repeat_until |
Branching and loops |
parallel |
Concurrent scopes (not flat steps) |
collect / wait / workflow_call |
Merge, delay, nested workflow |
log / terminate |
Logging and controlled stop |
Full list: Step Types. Demos: weatherstack_current_demo, transform_testing_demo, joke_api_random_demo, variable_binding_demo, storage_minio_demo, soap_http_xslt_demo, business_event_demo, script steps, C# custom process.
Connections (Secure Credentials)¶
name: my_api
type: http
base_url: https://api.example.com
# credentials via $secret:… — see Connections howto
Policies (Gateway Chains)¶
name: example-throttle-ip
type: throttle
direction: inbound
config:
max_requests: 60
window_seconds: 60
key_from: ip
Attach via ordered policy_chain on triggers (and global/product/user layers) — not apply_to / required_headers. See Policies.
Common Workflows¶
1. Data Integration¶
Fetch data from external API, transform, store:
service_call (egress.http.call) → service_call (transformers.xml2json) → service_call (egress.s3.put)
2. File Processing¶
Fetch, upload via FTP, persist:
service_call (http) → service_call (ftp.call) → service_call (egress.s3.put)
3. Scheduled Reports¶
Generate reports on schedule, email results:
Schedule trigger → service_call (transformers.*) → service_call (egress.smtp.call)
4. Event-Driven Processing¶
React to broker messages:
AMQP / RabbitMQ trigger → process → service_call (egress.rabbitmq.topic.send)
5. Multi-Step Integration¶
Complex workflow with branching logic:
service_call → context_extract_* → if → parallel (scopes) → compensate on error
Deployment Methods¶
1. ConfigMap (Development)¶
Quick deployment for testing:
kubectl create configmap workflows --from-file=workflows/
kubectl create configmap connections --from-file=connections/
2. Forgejo Actions (Recommended for Production)¶
Workflows stored in Git; CI validates and deploys without engine rebuild:
- Push workflow YAML to Git
deploy.ymlvalidates headers/triggers on flows push- Artifacts uploaded to storage; orchestrator reload triggered
- Push to main → tst; other branches → dev
See Deployment Methods and CI/CD Pipeline.
3. API Upload (Advanced)¶
curl -X POST http://storage:8080/internal/upload/workflows/minimal \
-H "Authorization: Bearer <runtime-admin-token>" \
-H "Content-Type: application/yaml" \
--data-binary @flows/workflows/minimal.yaml
Other single-item uploads:
POST /internal/upload/connections/{name}POST /internal/upload/policies/{name}(throttle|headers|xml_json|oauth_outbound)POST /internal/upload/schedules/{name}POST /internal/upload/config(runtime settings YAML)
After uploading a workflow, trigger a reload in orchestrator (POST /admin/reload).
Workflow Lifecycle¶
Edit YAML
↓
Local Testing
↓
Push to Git
↓
CI/CD Validation
↓
Deploy to Cluster
↓
Monitor in Tempo/Dashboard
↓
Update if needed
Common Tasks¶
Create First Workflow¶
Add a Trigger¶
Add Credentials¶
Deploy Workflows¶
Learn All Step Types¶
Debug Execution¶
→ Observability Guide (Forgejo mirror: Observability)
Understand Transformations¶
Next Steps¶
- First Time?: Start with First Workflow
- Learning: Check out Examples & Recipes
- Production: Review Deployment Methods and Security
- Advanced: Explore Error Handling & Compensation