Observability: Three-State Health Checks and a Tracing Abstraction

· · 4 min read

Three-state health (Healthy / Degraded / Unhealthy) carries more information than a binary up/down check. Tracing depends on a small Tracer interface, with OTel connected through an adapter.

Correction (2 November 2026, 18:45 Taipei): the comparison of the three states with K8s probes is replaced by a note that the endpoint decides how they map to probe results (suggested by Claude, checked by Codex (gpt-6-astra)). Checks are now said to need to honour context cancellation, or they hold up the batch and the next round; the claim that business code stays untouched is removed and the basic Tracer methods are now described as producing no OTel spans; the MemoryHealthCheck threshold now reads as allocated heap in whole MiB, with Degraded only above 80% (suggested by Codex (gpt-6-astra), checked by Claude).

Prerequisite: Interfaces and Composition

Three-state health checks

Traditional health checks are up/down, but the real world is often “still serving, but on the edge”. Gortex uses three states:

const (
    HealthStatusHealthy   HealthStatus = "healthy"
    HealthStatusDegraded  HealthStatus = "degraded"  // still serving, but near the limit
    HealthStatusUnhealthy HealthStatus = "unhealthy"
)

Aggregation is worst-wins: any check Unhealthy makes the whole Unhealthy; otherwise any Degraded makes it Degraded; all good is Healthy. The built-in MemoryHealthCheck shows it. It looks at allocated heap (runtime.MemStats.Alloc, in whole MiB, not the process RSS): over the limit is Unhealthy, and above 80% of the limit it flags Degraded as an early warning.

Kubernetes probes only report Success, Failure or Unknown (Unknown means the diagnostic didn’t complete and will be retried), with no degraded state; see the official docs. Gortex’s health package computes the three states but doesn’t say how they map to HTTP status codes. Whether Degraded makes readiness return 200 or 503 is up to whoever wires the /healthz endpoint, and readiness and liveness can use different rules.

Concurrent checks with timeouts

HealthChecker runs a background goroutine on a ticker (runChecks), caches the results, and endpoints like /healthz read the cache instead of re-running on every hit.

When they do run, all checks run concurrently, each with its own timeout, so the fast ones update the cache without waiting for a slow one:

for name, check := range checks {
    wg.Add(1)
    go func(n string, c HealthCheck) {
        defer wg.Done()
        checkCtx, cancel := context.WithTimeout(ctx, hc.timeout)
        defer cancel()
        result := c(checkCtx) // each check gets its own deadline
        // ...store result
    }(name, check)
}
wg.Wait()

A sync.WaitGroup waits for them all. The timeout only cancels the context and can’t force a check to stop. A check has to watch ctx.Done() and use context-aware I/O; one that ignores cancellation keeps wg.Wait() waiting, so Check doesn’t return and the background ticker’s next round is held up too. This is the concurrency from Goroutines and Channels and the sync primitives from Memory and Sync Primitives, applied to the operations side.

The Tracer interface and OTel adapter

This section is the interfaces post’s interface-driven design in the flesh. The framework depends on one small interface:

type Tracer interface {
    StartSpan(ctx context.Context, operation string) (context.Context, *Span)
    FinishSpan(span *Span)
    AddTags(span *Span, tags map[string]string)
    SetStatus(span *Span, status SpanStatus)
}
type EnhancedTracer interface { // interface embedding
    Tracer
    StartEnhancedSpan(ctx context.Context, operation string) (context.Context, *EnhancedSpan)
}

Three implementations ship: NoOpTracer (tracing off), SimpleTracer (in-memory, for development), and OTelTracerAdapter (wired to OpenTelemetry). Swapping the implementation isn’t enough to move from in-memory to OTel. The basic StartSpan and FinishSpan on OTelTracerAdapter only pass through to gortex’s own tracer and never create an OTel span (a *Span can’t carry one, and nothing could end it). To reach OTel, the code that opens spans has to call the adapter’s StartSpanWithOptions and call End() on the returned *SpanAdapter; with StartEnhancedSpan, it has to fetch the SpanAdapter from the context and end that. The adapter is still the one from the interfaces post, but the callers change. Inside, the adapter holds a severityMap that maps gortex’s eight severity levels (DEBUG=10 … EMERGENCY=80) onto OTel attributes.

Context propagation

How does a span follow the request chain? Through context, using a private contextKey type so keys can’t collide:

type contextKey string
const spanContextKey contextKey = "span"

func ContextWithSpan(ctx context.Context, span *Span) context.Context {
    return context.WithValue(ctx, spanContextKey, span)
}
func SpanFromContext(ctx context.Context) *Span { /* ctx.Value(spanContextKey) */ }

StartSpan reads the parent span with SpanFromContext, creates a child, and puts the child back into the ctx to pass down. The whole chain is threaded through context — no globals, no hand-passing.

Notes

  • Three-state health (Healthy / Degraded / Unhealthy) adds a degraded state to up/down; K8s probes only report Success / Failure / Unknown, so the endpoint decides the mapping; aggregation is worst-wins.
  • Checks run on a background ticker and are cached; when run, a WaitGroup runs them concurrently with per-check timeouts, so the fast ones update the cache first; checks must honour ctx cancellation, or the batch and the next round wait for them.
  • Tracing depends only on the Tracer / EnhancedTracer interfaces (interface embedding); NoOp / Simple / OTel implementations are swappable; exactly the interfaces post’s adapter. Producing OTel spans needs StartSpanWithOptions.
  • A span propagates via a private contextKey + context.WithValue down the chain; eight severity levels map to OTel.
  • Observability is built on interfaces; wiring in OTel means span-opening calls switch to the adapter’s methods.

Source: yshengliao/gortex.

Ideas and technical judgement by Sheng; drafted with Claude · Go 1.25 (gortex go.mod) · compiled retroactively · part of the 2026-06-13 blog renovation, paint still drying.