Deploy MeshFlows Runtime Secrets¶
Use this howto when a workflow or connection needs credentials (API keys, passwords, OAuth client secrets). Keep values out of Git; commit only $secret:… references.
For Forgejo / Kubernetes platform secrets (JWT, image pull, Postgres, service tokens), see Secrets Management (Engine).
Runtime secret vs $secret:name¶
| Concept | What it is | Where it lives |
|---|---|---|
| Runtime secret | Named credential value (weatherstack-access-key → real key) |
Storage runtime secret store (per environment / namespace) |
$secret:name |
Placeholder in connection (or similar) YAML | Git under flows/connections/ — name only, never the value |
| K8s Secret | Cluster object for pods (auth tokens, MinIO, RabbitMQ broker user, …) | Namespace meshflows-<env> — not what $secret: resolves |
At load/run time the Orchestrator loads secrets from Storage and replaces $secret:foo with the stored value. Unresolved refs fail connection use / service_call steps.
flowchart LR
Git["flows/connections/*.yaml\naccess_key: $secret:my-key"]
Store["Storage runtime secrets\nname=my-key"]
Orch["Orchestrator reload"]
Egress["Egress provider"]
Git -->|"upload connection YAML"| Store
Store -->|"load secrets + resolve"| Orch
Orch -->|"resolved fields"| Egress
Create or update a secret¶
Secrets are per environment (lab / dev / tst / acc / prd → namespace meshflows-<env>). Creating a secret in Dashboard dev does not put it on prd.
Dashboard (preferred for app credentials)¶
- Open the Dashboard for the target environment.
- Go to Admin → Secrets (needs
meshflows:runtime:secrets:read; create/update needsmeshflows:runtime:secrets:write). - Enter name (must match the
$secret:…suffix exactly, e.g.weatherstack-access-key) and value. - Save. The UI reloads the Orchestrator when you have admin rights so connections pick up the new value.
- If reload was skipped, use Reload runtime (platform admin) or wait for the next deploy reload.
Do not paste real values into Git, tickets, or wiki pages.
API (Dashboard proxy)¶
Authenticated Dashboard session with write scope:
POST /api/runtime/secrets/{name}
Content-Type: application/json
{"value": "<your-secret-value>"}
List (metadata only): GET /api/runtime/secrets
Get one value: GET /api/runtime/secrets/{name}
Delete: DELETE /api/runtime/secrets/{name}
Storage internal API (ops / CI)¶
Trusted callers with Storage admin bearer token (cluster secret meshflows-storage-auth / admin_token):
POST /internal/secrets/{name}
Authorization: Bearer <storage-admin-token>
Content-Type: application/json
{"value": "<your-secret-value>"}
Same pattern as deploy sync for RabbitMQ / MinIO (see below). There is no separate end-user CLI; use Dashboard, curl to the API, or CI upload.
Reference secrets in connection YAML¶
Commit the connection under flows/connections/ with placeholders only:
# flows/connections/weatherstack.yaml
name: weatherstack
type: weatherstack
access_key: "$secret:weatherstack-access-key"
HTTP header example:
name: external-api
type: http
base_url: https://api.example.com
default_headers:
X-API-Key: $secret:api-key
OAuth client credentials:
name: salesforce_api
type: http
base_url: https://your-instance.salesforce.com
oauth:
token_url: https://login.salesforce.com/services/oauth2/token
client_id: "$secret:sf-client-id"
client_secret: "$secret:sf-client-secret"
scope: "api"
Rules:
- The string after
$secret:is the runtime secret name. - Connection YAML is uploaded by deploy as written — deploy does not expand
$secret:inside connection files. - Real values must already exist in the Storage secret store for that namespace (Dashboard, API, CI secrets upload, or auto-sync).
More connection patterns: Connections & Credentials.
How deploy picks secrets up (and what stays out of Git)¶
What Git must never contain¶
- Plaintext passwords, API keys, private keys, OAuth client secrets
- Rendered
secrets.yamlwith real values
Safe in Git: $secret:name (and non-sensitive connection fields such as hostnames when they are not secret).
Automatic sync (Engine deploy)¶
On Engine deploy, CI copies selected Kubernetes credentials into the Storage runtime store so connections can use $secret:…:
| Runtime secret name | Source K8s secret |
|---|---|
rabbitmq-user / rabbitmq-password |
meshflows-rabbitmq-auth |
minio-access-key / minio-secret-key |
meshflows-orchestrator-minio |
After broker or MinIO password rotation, re-run Engine deploy (or update the matching runtime secrets manually) so Storage stays in sync. See RabbitMQ.
Optional CI upload: engine/config/secrets.yaml¶
Workflows deploy (deploy-workflows in .github/workflows/deploy.yml) may upload:
- Render
engine/config/secrets.yamlwithengine/scripts/render-upload-substitutions.py POST /internal/upload/secretsto Storage- Reload Orchestrator
In that file, $secret:NAME / $var:NAME are filled from Forgejo environment variables:
- Secret
NAME→ envSECRET_<NORMALIZED>orSECRET_<NORMALIZED>_<ENV>(e.g.SECRET_SFTP_DEMO_PASSWORD_DEV) - Var
NAME→VAR_<NORMALIZED>/VAR_<NORMALIZED>_<ENV>
Normalization: non-alphanumeric characters become _, then uppercase (mijneigen_secret → MIJNEIGEN_SECRET). Env-specific keys win over the generic key when both exist.
Only names wired in the workflow env: block are available. Missing substitutions fail the render step. Prefer Dashboard for one-off app secrets; use Forgejo + secrets.yaml for demo/CI-seeded credentials that must land on every deploy.
Namespaces: meshflows-lab, meshflows-dev, meshflows-tst, meshflows-acc, meshflows-prd (prefix from MESHFLOWS_NAMESPACE).
Example: Forgejo MY_SECRET → runtime mijneigen_secret¶
The YAML key is the runtime secret name (what $secret:… in connections resolves). The $secret:… / $var:… placeholder only selects which Forgejo/env value is rendered at deploy time — the names do not have to match.
- Create a Forgejo Actions secret (repo or environment
dev-workflows/prd-workflows/ …), e.g.MY_SECRET. - Wire it into
deploy-workflowsenv:in.github/workflows/deploy.ymlunder the lookup name the render script expects:
# deploy-workflows env: (illustrative)
SECRET_MIJNEIGEN_SECRET: ${{ secrets.MY_SECRET }}
# optional per-env override:
# SECRET_MIJNEIGEN_SECRET_DEV: ${{ secrets.MY_SECRET }}
- Map it in
engine/config/secrets.yaml:
secrets:
mijneigen_secret: # ← runtime name used by connections as $secret:mijneigen_secret
value: $secret:mijneigen_secret # ← looks up SECRET_MIJNEIGEN_SECRET[_<ENV>]
Alternative when you want the placeholder to mirror the Forgejo name:
secrets:
mijneigen_secret:
value: $secret:my_secret # looks up SECRET_MY_SECRET → wire env SECRET_MY_SECRET: ${{ secrets.MY_SECRET }}
- Deploy workflows (or full deploy). CI renders the file and uploads to Storage; Orchestrator reload picks it up.
- In connection YAML use only the runtime name:
api_key: "$secret:mijneigen_secret".
Do not commit real values. Keep Forgejo secret values out of tickets and wiki pages.
Insert-only / no-overwrite (not supported today)¶
There is no create-if-not-exists flag on runtime secrets.
| Path | Behaviour today |
|---|---|
| Dashboard Admin → Secrets save | Always upsert (creates or overwrites) |
POST /api/runtime/secrets/{name} |
Always upsert |
POST /internal/secrets/{name} |
Always upsert (CI RabbitMQ/MinIO sync uses this) |
POST /internal/upload/secrets (secrets.yaml) |
Always upsert every listed name; optional ?prune=true also deletes secrets not in the file |
Platform K8s reconcile (“empty Forgejo value → keep existing K8s key”) does not apply to Storage runtime secrets.
Safest current practice when you must not overwrite an existing value:
- Prefer Dashboard once for app credentials that operators own — create the secret manually and do not list that name in
engine/config/secrets.yaml(otherwise every workflows deploy overwrites it with the Forgejo value). - CI insert-only pattern (custom job / one-off ops script), not built into MeshFlows:
# Pseudocode — Storage admin token; never echo the value
# 1) GET /internal/secrets/{name} → if HTTP 200: skip
# 2) if HTTP 404: POST /internal/secrets/{name} with {"value":"..."}
Same idea via Dashboard proxy: GET /api/runtime/secrets/{name} then POST only when missing. There is no official overwrite=false query/body on these endpoints.
Warnings:
- Putting a name in
secrets.yamlmeans every successful workflows deploy replaces the Storage value for that name with the rendered Forgejo value. ?prune=trueon secrets upload is destructive for names absent from the YAML (deploy currently uploads without prune; do not enable prune casually).- RabbitMQ / MinIO Engine sync always overwrites
rabbitmq-*/minio-*runtime secrets from K8s — treat those names as platform-owned. - Missing Forgejo/env substitution fails the render step (deploy aborts); it does not “skip and keep old”.
Platform secrets are a different pipeline¶
Forgejo secrets such as IDENTITY_JWT_SECRET, ORCH_RELOAD_TOKEN, STORAGE_SERVICE_* are reconciled into Kubernetes Secrets for pods. They are not automatically exposed as $secret: names unless a dedicated sync step (like RabbitMQ/MinIO) or secrets.yaml upload maps them. Details: Engine Secrets Management.
Verify¶
- Dashboard → Secrets — name exists in the correct environment (value hidden until View).
- Dashboard → Connections — open the connection → Test (
POST /api/connections/{name}/test). Failures often mean missing/unresolved$secret:or wrong value. - Run a workflow that uses the connection (
service_callwithconnection:). Check run / traces for secret-resolution or auth errors (values should stay masked). - After creating secrets via API without UI reload: trigger Orchestrator reload (
POST /api/runtime/reloadfrom Dashboard, or/admin/reloadwith reload token).
Common mistakes¶
| Mistake | Fix |
|---|---|
Commit plaintext credentials in flows/connections/ |
Use $secret:name only; store the value in Dashboard/Storage |
| Secret created in dev, workflow runs in prd | Create/update the secret in each target environment |
Name typo ($secret:api_key vs secret api-key) |
Names must match exactly (allowed: letters, digits, _, ., -) |
| Updated secret but still old behaviour | Reload Orchestrator / use Dashboard save+reload |
Expect $secret: in connection YAML to be filled by GitHub/Forgejo env |
Connection upload does not substitute; only engine/config/secrets.yaml render does |
Forgejo secret set but not in deploy-workflows env: |
Render cannot see it — add SECRET_<NORMALIZED>: ${{ secrets.… }} to the job env |
| Manual Dashboard value overwritten on next deploy | Name was also in secrets.yaml — remove it from CI upload or accept Forgejo as source of truth |
| Expect CI upload to skip existing secrets | Not supported — upsert always; use Dashboard-only or GET-then-POST insert-only script |
Confuse K8s Secret meshflows-rabbitmq-auth with runtime $secret:rabbitmq-password |
Broker user is K8s; connection resolution uses Storage — Engine deploy syncs them |
| Delete a secret still referenced by a connection | Check references / force-delete only when intentional; update or remove $secret: refs first |
| Missing RBAC | Need meshflows:runtime:secrets:read / :write (see IAM role templates) |
Related¶
- Connections & Credentials
- Connections reference
- Engine Secrets Management — Forgejo → K8s platform secrets
- ADR 0011 — Connection credential isolation
- Integration end-to-end