Skip to content

Observability

MeshFlows observability combines service health, request traces, trigger events, and platform telemetry.

Observability Signals

  • Health and readiness endpoints per service (/healthz, /readyz).
  • Workflow execution traces (/api/traces*).
  • Trigger audit events (/api/trigger-events*).
  • OpenTelemetry traces exported to collector/backends.
  • Service logs from runtime containers.

Trace Correlation Model

  • X-Request-ID is used for end-to-end request correlation.
  • Gateway forwards correlation context to Orchestrator.
  • Orchestrator emits trace documents per run and per-step artifacts.

Suggested Stack

  • OpenTelemetry Collector for ingestion and routing.
  • Tempo for distributed tracing.
  • Loki (or equivalent) for log aggregation.
  • Optional dashboards/alerts in your preferred monitoring tool.

Common Diagnostic Workflow

  1. Confirm service readiness with /readyz.
  2. Locate failing run via /api/traces?status=failed.
  3. Inspect step-level data through trace step endpoints.
  4. Check trigger event stream for dropped/failed trigger attempts.
  5. Correlate with infrastructure logs and broker metrics.

Best Practices

  • Capture enough context to reproduce failures without logging secrets.
  • Retain traces long enough for incident review windows.
  • Alert on repeated trigger failures and rising latency percentiles.
  • Use consistent request identifiers across all integrations.

Business events (thin convention)

Technical traces do not replace business milestones (order placed, invoice posted). MeshFlows does not ship BAM / Invictus. Workflows emit a greppable line from the existing type: log step.

  • Convention + fieldset: Business events
  • Demo: business_event_demo (POST /v1/api/examples/business-event-demo)
  • Loki: {component="orchestrator"} |= "meshflows.business_event"
  • Metrics: LogQL count_over_time(... |= "meshflows.business_event" [5m]) — not a Prometheus counter or Grafana pack

Operations milestones (type: milestone / context.milestone) are a different surface: they roll up onto the runtime-root row in the Operations list, they are not Loki events.