Skip to content

Develop and Test (End-to-End)

This guide walks through the full developer loop: local setup, validation, branch workflow, CI/CD promotion, and manual Actions when you need them.

Prerequisites

Before you start, confirm:

  • Prerequisites — Kubernetes access (for cluster deploy), Docker, Git, Python 3.11+
  • Forgejo account with access to the repository
  • Optional: SSH alias to the cluster control plane (Cluster SSH Access)

Repository Setup

git clone https://code.meshflows.org/meshflowsgpl/meshflows.git
cd meshflows

# Enable local workflow validation before every commit
git config core.hooksPath .githooks

Optional commit message template for workflow changes:

git config commit.template .gitmessage

Local Engine and Flows

Engine (Docker Compose)

cd engine
cp .env.example .env
# Edit .env — generate secrets with: openssl rand -hex 32

docker-compose up -d

Verify:

curl http://localhost:8080/v1/status
curl http://localhost:8083/readyz

See Local Development for ports, reload tokens, and troubleshooting.

Flows (workflows, triggers, artifacts)

Workflows live under flows/workflows/; start contracts under flows/triggers/. Test against the local engine:

curl -X POST http://localhost:8080/v1/run/minimal \
  -H "Content-Type: application/json" \
  -d '{"xml": "<root/>"}'

Validate locally:

python flows/scripts/validate_workflow_headers.py --changed-only
python flows/scripts/validate_triggers.py --changed-only

After editing workflow files, reload the orchestrator:

curl -X POST http://localhost:8083/admin/reload \
  -H "X-Reload-Token: <ORCH_RELOAD_TOKEN from .env>"

pytest and pre-commit

pytest (engine tests)

cd engine
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r services/orchestrator/requirements.txt
pip install pytest pytest-asyncio httpx

python -m pytest -q tests -m "not legacy_broken"

CI runs the same gate via test.yml (structure, pytest smoke, flows YAML validation, K8s yamllint).

pre-commit hook

The repo ships .githooks/pre-commit (enabled via core.hooksPath). It runs:

  • flows/scripts/validate_workflow_headers.py --changed-only
  • flows/scripts/validate_triggers.py --changed-only
  • JavaScript syntax check on dashboard static files (when node is installed)

Branch → dev

Event Build Auto-deploy target
Push to feature/fix/dev/docs branch Selective build + test gate dev
Pull request (any branch → main) Selective build + test gate dev only

Flow:

  1. Push your branch → build.yml detects changed paths, builds only affected engine/wiki images
  2. test.yml runs as the build test gate
  3. On success, build.yml dispatches deploy.yml to dev via the Forgejo API (trigger-deploy-dev)

PRs never promote to tst or lab — only dev.

PR → dev

Open a PR targeting main. The pipeline:

  1. Builds changed services with a PR-scoped release tag (vYYYY.MM.DD.<run>-pr<N>)
  2. Runs the full test gate
  3. Deploys to dev for integration verification

Review the build summary for release_id and deploy status. Fix failing tests before merge.

main → dev → tst + lab

After merge (push) to main:

  1. Phase 1: build + test gate → deploy dev (same as PR, but with production release tag)
  2. Phase 2: when dev deploy succeeds → tst and lab deploy in parallel (trigger-deploy-promotion)

This is the standard promotion path for engine and wiki changes. acc/prd remain manual.

flows-only → tst

When you push only flows/** changes (no engine rebuild):

  • deploy.yml triggers directly on push (not via build.yml)
  • Validates workflow headers and triggers
  • Uploads artifacts to storage and reloads the orchestrator
  • Target environment:
  • push to main → tst
  • push to develop, feature, or docs branches → dev

No new container images are built. Engine services are not rolled.

Cluster Build and Deploy Agents

CI runs on Forgejo Actions runners:

Label Role
meshflows Build, test, setup resolution
deploy-<env> kubectl deploy (deploy-dev, deploy-tst, …)

VM runners (build)

Setup: CI Runner README and engine/scripts/bootstrap-ci-runner.sh.

In-cluster runners (deploy)

Deploy runners into namespace meshflows-ci:

# Fill engine/runner.secrets.env first (not committed)
bash engine/scripts/deploy-ci-runners-k8s.sh --remote k8s-master

This installs build + deploy runners for lab, dev, and tst. acc/prd deploy runners are not yet provisioned — use manual deploy.yml with a deploy-capable host when needed.

See also Cluster SSH Access.

Runner image bijwerken

Na een nieuwe forgejo-runner build (via build.yml job build-forgejo-runner, of handmatig met publish-runner-image.sh):

  1. Noteer de image-tag uit de build summary (bijv. v2026.07.21.42 of latest).
  2. Actions → Redeploy CI Runners → kies runner_image_tag en target (all voor build + deploy runners).
  3. Controleer in Forgejo: Settings → Actions → Runners (online + juiste labels).

De workflow draait op deploy-dev, werkt deployments in namespace meshflows-ci bij via kubectl set image, en wacht tot pods Ready zijn. Geen SSH nodig.

Voor eerste install of registration-token rotatie blijft engine/scripts/deploy-ci-runners-k8s.sh nodig — de workflow past geen tokens of RBAC aan.

Manual Forgejo Actions

Use when auto-deploy is insufficient or you need a specific release:

Workflow When to use
Build & Push (build.yml) Full rebuild (full_build=true); no auto-deploy on manual run
Deploy to Cluster (deploy.yml) Pick environment + release_id; lab manual promotion from cluster index
Deploy Cluster Infrastructure (infra.yml) Gateway API CRDs, nginx Gateway Fabric, cert-manager, ClusterIssuers
Test & Validate (test.yml) Re-run validation without build
Security Audit / Smoke Weekly audit; runner health check
Redeploy CI Runners (redeploy-ci-runners.yml) Roll out new forgejo-runner image after Dockerfile.runner changes

Lab manual deploy (pick a release)

  1. Actions → Deploy to Cluster → environment=lab, leave release_id empty → lists recent releases from cluster index
  2. Re-run with chosen release_id — scope loads from index automatically

Redeploy / rotate secrets

Update the Forgejo environment secret → re-run deploy.yml with the same release_id. The Ensure required Engine auth secrets step reconciles K8s secrets and restarts dependent pods.

Full workflow reference: CI/CD Pipeline and .github/workflows/README.md.

Quick Reference

feature branch ──push──► build + test ──► dev
PR ────────────────────► build + test ──► dev
main (engine/wiki) ──────► build + test ──► dev ──► tst + lab (parallel)
main (flows only) ───────► validate ──────► tst
develop (flows only) ────► validate ──────► dev
lab/acc/prd ───────────── manual deploy.yml + release_id

Next Steps