Secrets Management¶
MeshFlows uses two different secret layers. Pick the right one before you store a value.
| Layer | Purpose | Typical names | How to manage |
|---|---|---|---|
| Runtime secrets | Values that workflows / connections resolve via $secret:name |
weatherstack-access-key, rabbitmq-password, minio-access-key |
Dashboard Admin → Secrets, Storage API, optional CI upload |
| Platform (K8s) secrets | Pod env / mounts for Engine services | meshflows-identity-auth, meshflows-storage-auth, image pull |
Forgejo environment secrets → deploy.yml reconcile |
Developers deploying connection credentials: follow the full howto → Deploy MeshFlows Runtime Secrets.
This page covers platform / CI secrets and points at runtime sync that bridges the two layers.
Secret Types (platform view)¶
- Platform secrets — JWT keys, admin bootstrap credentials, encryption keys, service tokens.
- Connection / runtime secrets — API keys, OAuth client secrets, passwords referenced as
$secret:…in connections (stored in Storage, not as arbitrary K8s Secret keys). - Runtime service tokens — reload / invocation / internal auth between microservices (usually K8s Secrets wired into pods).
Never put plaintext credentials in workflow or connection YAML in Git.
Runtime secrets (summary)¶
- Create the value in the target environment’s Storage store (Dashboard or API).
- Reference it in
flows/connections/*.yamlas$secret:<exact-name>. - Deploy uploads connection YAML unchanged; the Orchestrator resolves secrets at reload/run.
- Engine deploy auto-syncs RabbitMQ and MinIO credentials into runtime names
rabbitmq-user/rabbitmq-passwordandminio-access-key/minio-secret-key.
Full steps, verification, and common mistakes: Deploy MeshFlows Runtime Secrets.
CI Path: Forgejo Environment Secrets¶
Deploy jobs in deploy.yml bind to Forgejo environments (lab, dev, tst, acc, prd). Environment-scoped secrets override repository defaults.
Before each engine rollout, the workflow step Ensure required Engine auth secrets reconciles Forgejo secrets into Kubernetes:
| Forgejo secret | K8s secret (key) |
|---|---|
IDENTITY_JWT_SECRET |
meshflows-identity-auth (jwt_secret) |
IDENTITY_ENCRYPTION_KEY |
meshflows-identity-auth (encryption_key) |
ORCH_RELOAD_TOKEN |
meshflows-orchestrator-auth (reload_token) |
RABBITMQ_USER / RABBITMQ_PASSWORD / RABBITMQ_INTERNAL_TOKEN |
meshflows-rabbitmq-auth |
STORAGE_SERVICE_READ_TOKEN / WRITE / ADMIN |
meshflows-storage-auth |
SES_SERVICE_TOKEN |
meshflows-ses-auth (service_token) |
CAPABILITY_REGISTRY_TOKEN |
meshflows-capability-registry-auth (token) |
SCHEDULER_RELOAD_TOKEN |
meshflows-scheduler-auth |
ORCHESTRATOR_POSTGRES_PASSWORD |
meshflows-orchestrator-postgres |
Reconcile rules:
- Empty Forgejo value → keep existing Kubernetes secret value.
- Neither Forgejo nor K8s value → CI generates a fallback (logged as hint — set explicitly for production).
- After rotation: update Forgejo secret → redeploy with the same
release_id→ dependent pods restart when tokens change.
Full table and mandatory vs optional secrets: .github/workflows/README.md.
Optional demo/CI runtime seeds (SECRET_SFTP_DEMO_*, SECRET_ACME_PROD_*, …) are rendered into engine/config/secrets.yaml during workflows deploy — not the same as the K8s table above. Mapping Forgejo → runtime name, and the fact that upload is always upsert (no insert-only): runtime secrets howto.
Setup checklist (Forgejo-only, no kubectl)¶
- Create environments: Settings → Environments →
lab,dev,tst,acc,prd. - Set repo-level shared secrets (
REGISTRY_*,WORKFLOW_DISPATCH_TOKEN,WIKI_PUSH_TOKEN). - Override per-env secrets on
prd/acc(JWT, storage, Postgres passwords). - Deploy via Actions or wait for auto-deploy after green build.
- Verify in deploy logs (secret reconcile + post-deploy verify steps).
Optional: kubeconfig helper scripts¶
For offline bootstrap or legacy tooling (not used by deploy.yml):
# Print base64 kubeconfig for manual paste into Forgejo Actions secrets
bash engine/scripts/generate-cluster-secret.sh --env DEV --kube ~/.kube/config
# Print curl/gh command for a specific forge host
bash engine/scripts/set-cluster-secret.sh \
--host https://code.meshflows.org --repo meshflowsgpl/meshflows --env DEV \
--file ~/.kube/config
Forgejo deploy runners use host kubectl; CLUSTER_<ENV> secrets are optional for normal CI.
Deploy runtime secrets via API¶
Runtime (connection) secrets can be created from a pipeline without opening the dashboard.
| Mode | Query / body | Existing name |
|---|---|---|
| Exclusive (safe create) | overwrite=false |
409 Conflict, value unchanged |
| Overwrite (rotate / upsert) | overwrite=true (default) |
updated |
Dashboard (session + meshflows:runtime:secrets:write):
POST /api/runtime/secrets/{name}
Content-Type: application/json
{"value": "…", "overwrite": false}
Storage (cluster-internal admin token, used by deploy.yml):
POST /internal/secrets/{name}?overwrite=false
Authorization: Bearer <STORAGE_ADMIN_TOKEN>
Content-Type: application/json
{"value": "…"}
Bulk YAML upload accepts the same flag: POST /internal/upload/secrets?overwrite=false. On conflict the whole document is rejected and nothing is written. prune=true is incompatible with exclusive mode.
Default remains upsert so dashboard save and existing CI secret sync keep working.
Dummy sample after a default deploy¶
Every engine deploy (dev / linux-cluster via deploy.yml) seeds two non-credential examples so you can see runtime secrets without bringing real keys:
| Name | File | Mode | Redeploy |
|---|---|---|---|
example-dummy-overwrite |
engine/config/secrets.yaml |
upsert (overwrite=true) |
value reset to dummy-not-a-real-credential |
example-dummy-exclusive |
engine/config/secrets.example-dummy-exclusive.yaml |
exclusive (overwrite=false) |
created once; later deploys log 409 and keep the existing value |
After deploy: Dashboard → Admin → Secrets. View the dummy names (needs meshflows:runtime:secrets:read). Deploy logs show Created exclusive dummy… or already present (not overwritten).
Change the exclusive dummy in the dashboard, redeploy, and confirm the dashboard value is unchanged — that is the #131 exclusive path. The overwrite dummy returns to the placeholder on every upload of secrets.yaml.
Local Development¶
Copy engine/.env.example to engine/.env and replace all placeholder values. Never commit .env.
Manual Kubernetes Pattern¶
For bootstrap or offline clusters without CI:
kubectl create secret generic meshflows-identity-auth \
--from-literal=jwt_secret=<value> \
--from-literal=encryption_key=<value> \
--namespace meshflows-dev
Offline seed files (/etc/meshflows/secrets.<env>.env + apply-cluster-secrets.sh) are optional bootstrap aids — not required when using Forgejo deploy.
Recommended Practice¶
- Store platform secrets in Kubernetes / Forgejo environment secrets — not in Git.
- Store connection credentials as runtime secrets; reference names only in connections.
- Rotate secrets on schedule and after incidents.
- Use unique credentials per production environment (
prd,acc).
Validation¶
- Ensure pods receive expected env vars after platform secret changes.
- Confirm traces/logs mask sensitive headers and values.
- Test credential-dependent workflows after each rotation (connection Test in Dashboard).
- Call admin endpoints with
X-Reload-Tokenafter changingORCH_RELOAD_TOKEN.
Related¶
- Deploy MeshFlows Runtime Secrets —
$secret:, Dashboard, deploy sync - Connections & Credentials
- Develop and Test — redeploy after secret rotation
- CI/CD Pipeline