Forgejo Container Package Cleanup¶
Operator runbook for reclaiming disk on Forgejo package storage (not Kubernetes node disk). MeshFlows engine images live as container packages under MeshFlowsGPL/meshflows packages.
This is a manual checklist. Automated purge was removed from build.yml because deleting tags that were still referenced in engine-images-lock caused ImagePullBackOff. Do not put a blanket delete back into CI.
Related: CI/CD Pipeline, .github/workflows/README.md, Operations. Issue: #138.
Policy¶
| Tag kind | Pattern | Example | Keep |
|---|---|---|---|
| Product alias | vX.Y.Z (semver, not year-based) |
v0.4.0 |
Always |
| Immutable build | vYYYY.MM.DD.<run> |
v2026.09.17.123 |
Last 30 plus every tag still used by deploy |
| PR / develop | suffix -pr<N> or -dev |
v2026.09.17.123-pr42 |
Last 5 optional; the rest may go |
| Untagged / hash-only | OCI digest leftover, 7+ hex | a1b2c3d |
Delete |
Deploy pins on the immutable build tag (release_id), not on the product alias. v0.4.0 is a human/registry alias published by build.yml (publish_product_version) or recovered via publish-product-version.yml. Deleting a build tag that a namespace still pulls will break that environment even if the product alias remains.
Do not touch Docker Hub / infra images (loki, tempo, rabbitmq, redis, postgres, minio, otel). Those are not MeshFlows Forgejo packages.
Leave meshflows/forgejo-runner unless you have already rolled a newer runner (redeploy-ci-runners.yml). Wiki (meshflows/wiki) follows the same keep/delete rules as engine services.
What is in scope¶
Forgejo container packages named meshflows/<service> for every build_name in engine/deploy/engine-images-manifest.yaml, plus meshflows/wiki.
Out of scope: Action run history (that is cleanup-action-runs.yml), git releases, and cluster PVC/node disk.
1. Measure disk (Forgejo host)¶
Package blobs live on the Forgejo data volume, not on Kubernetes worker nodes. Cluster kubectl top nodes does not tell you whether packages are filling the registry.
On the Forgejo host (or its volume mount), check the packages/data filesystem. The exact path depends on how Forgejo is installed; typical locations are the Forgejo data PVC / /var/lib/gitea / compose volume named data. Record used/free before and after a cleanup.
The package UI also shows per-version size: open a package → version list.
2. Build the keep-set (do this first)¶
Collect tags you must not delete. Union of:
- All product aliases — every tag matching
^v[0-9]+\.[0-9]+\.[0-9]+$(rootVERSIONis the current one; oldervX.Y.Zstay too). - Cluster lock per environment — ConfigMap
engine-images-lockinmeshflows-dev,meshflows-lab,meshflows-tst,meshflows-acc,meshflows-prd. - Release index — ConfigMap
meshflows-release-indexinplatform-data(last 30release_idvalues; lab picker uses this). - Latest lock artifact —
images.lock.jsonfrom a successful deploy (engine-images-lockartifact), if you are not looking at the cluster.
From a laptop with SSH to the control plane (Cluster Access via SSH):
ssh -o BatchMode=yes k8s-master 'for ns in meshflows-dev meshflows-lab meshflows-tst meshflows-acc meshflows-prd; do
echo "=== $ns ==="
kubectl get configmap engine-images-lock -n "$ns" -o jsonpath="{.data.images\.lock\.json}" 2>/dev/null || echo "(missing)"
echo
done
echo "=== release-index ==="
kubectl get configmap meshflows-release-index -n platform-data -o yaml 2>/dev/null || echo "(missing)"'
Keep every version / tag / release_id you see. If a service in one namespace still runs v2026.08.01.40, that tag stays for all packages that still need it (selective deploys mean services can pin different build tags).
3. List package versions¶
UI: Packages → open meshflows/<service> → Versions.
API (token from the environment only — never commit or echo it). Owner-scoped container listing:
$headers = @{
Authorization = "token $env:FORGEJO_TOKEN"
Accept = "application/json"
}
# List packages (page through if needed)
Invoke-RestMethod -Headers $headers -Uri "https://code.meshflows.org/api/v1/packages/MeshFlowsGPL?type=container&limit=50&page=1"
# Versions for one image (name is URL-encoded: meshflows%2Forchestrator)
Invoke-RestMethod -Headers $headers -Uri "https://code.meshflows.org/api/v1/packages/MeshFlowsGPL/container/meshflows%2Forchestrator/versions?limit=50&page=1"
Classify each version string:
- Product alias → keep
- Contains
-pror-dev→ delete candidate (keep newest 5 if you still debug those builds) - Matches
v202*without those suffixes → keep if in the keep-set or among the newest 30; otherwise delete candidate - Hash-only / untagged → delete candidate
scripts/cleanup-codeberg-packages.ps1 is a legacy listing/delete helper. Do not run it with -Perform. Its default VersionPrefix=v0 selects product aliases (v0.x.y) and would delete older vX.Y.Z tags — the opposite of this policy. It also ignores engine-images-lock and the release index.
4. Delete (UI first, one tag at a time)¶
Prefer the package UI for the first pass: open the version → Delete. Start with obvious -pr / -dev tags that are not in the keep-set.
API delete (same token header). This is irreversible:
# Example only after the tag is confirmed unused
$pkg = [uri]::EscapeDataString("meshflows/orchestrator")
$ver = [uri]::EscapeDataString("v2026.01.02.3-pr12")
Invoke-RestMethod -Method Delete -Headers $headers -Uri "https://code.meshflows.org/api/v1/packages/MeshFlowsGPL/container/$pkg/$ver"
Work package by package. A shared release_id is one tag name on many images; if you drop v2026.09.01.10 on orchestrator but gateway in lab still pins it, only gateway is safe.
After a batch, re-check disk on the Forgejo host. Forgejo may not free space until GC; if usage barely moves, wait or run Forgejo’s package/storage GC from the admin side (host-specific).
5. Verify nothing in cluster broke¶
ssh -o BatchMode=yes k8s-master "kubectl get pods -A | grep -E 'meshflows-|ImagePull'"
ImagePullBackOff on a MeshFlows workload after a delete means a still-used tag was removed. Recover by redeploying that environment with a release_id that still exists, or by rebuilding/publishing that tag. Product alias vX.Y.Z cannot substitute for a missing build tag in deploy.
Checklist¶
- [ ] Disk usage recorded on the Forgejo package volume (not cluster nodes)
- [ ] Keep-set collected from all env
engine-images-lockmaps +meshflows-release-index+ allvX.Y.Z - [ ] No Docker Hub / infra images in the delete list
- [ ]
meshflows/forgejo-runnerskipped (unless you intend to) - [ ]
-pr/-devand old build tags classified; product aliases untouched - [ ] Deletes done in UI or explicit per-version API calls (no CI purge, no
-Performon the legacy script) - [ ] Cluster pods still Running; no new
ImagePullBackOff
What this runbook does not automate¶
- No workflow that deletes package versions on build or on a schedule
- No “keep newest N of everything” script — that previously collided with
engine-images-lock - Action-run pruning is separate (
cleanup-action-runs.yml) and does not free package blob disk