Observability: Three-State Health Checks and a Tracing Abstraction

Three-state health (Healthy / Degraded / Unhealthy) carries more information than a binary up/down check. Tracing depends on a small Tracer interface, with OTel connected through an adapter.

Prerequisite: Interfaces and Composition

Three-state health checks

Traditional health checks are up/down, but the real world is often “still serving, but on the edge”. Gortex uses three states, close to K8s readiness / liveness:

const (
    HealthStatusHealthy   HealthStatus = "healthy"
    HealthStatusDegraded  HealthStatus = "degraded"  // still serving, but near the limit
    HealthStatusUnhealthy HealthStatus = "unhealthy"
)

Aggregation is worst-wins: any check Unhealthy makes the whole Unhealthy; otherwise any Degraded makes it Degraded; all good is Healthy. The built-in MemoryHealthCheck shows it: over the limit is Unhealthy, and at 80% it flags Degraded as an early warning.

Concurrent checks with timeouts

HealthChecker runs a background goroutine on a ticker (runChecks), caches the results, and endpoints like /healthz read the cache instead of re-running on every hit.

When they do run, all checks run concurrently, each with its own timeout, so one slow check can’t drag the rest down:

for name, check := range checks {
    wg.Add(1)
    go func(n string, c HealthCheck) {
        defer wg.Done()
        checkCtx, cancel := context.WithTimeout(ctx, hc.timeout)
        defer cancel()
        result := c(checkCtx) // each check gets its own deadline
        // ...store result
    }(name, check)
}
wg.Wait()

A sync.WaitGroup waits for them all. This is the concurrency from Goroutines and Channels and the sync primitives from Memory and Sync Primitives, applied to the operations side.

The Tracer interface and OTel adapter

This section is the interfaces post’s interface-driven design in the flesh. The framework depends on one small interface:

type Tracer interface {
    StartSpan(ctx context.Context, operation string) (context.Context, *Span)
    FinishSpan(span *Span)
    AddTags(span *Span, tags map[string]string)
    SetStatus(span *Span, status SpanStatus)
}
type EnhancedTracer interface { // interface embedding
    Tracer
    StartEnhancedSpan(ctx context.Context, operation string) (context.Context, *EnhancedSpan)
}

Three implementations ship: NoOpTracer (tracing off), SimpleTracer (in-memory, for development), and OTelTracerAdapter (wired to OpenTelemetry). Moving from in-memory to Jaeger / OTel is just swapping an implementation, with business code untouched — that’s the adapter from the interfaces post. Inside, the adapter holds a severityMap that maps gortex’s eight severity levels (DEBUG=10 … EMERGENCY=80) onto OTel attributes.

Context propagation

How does a span follow the request chain? Through context, using a private contextKey type so keys can’t collide:

type contextKey string
const spanContextKey contextKey = "span"

func ContextWithSpan(ctx context.Context, span *Span) context.Context {
    return context.WithValue(ctx, spanContextKey, span)
}
func SpanFromContext(ctx context.Context) *Span { /* ctx.Value(spanContextKey) */ }

StartSpan reads the parent span with SpanFromContext, creates a child, and puts the child back into the ctx to pass down. The whole chain is threaded through context — no globals, no hand-passing.

Notes

• Three-state health (Healthy / Degraded / Unhealthy) maps to K8s probes better than up/down; aggregation is worst-wins.
• Checks run on a background ticker and are cached; when run, a WaitGroup runs them concurrently with per-check timeouts, so the slow don’t drag the fast.
• Tracing depends only on the Tracer / EnhancedTracer interfaces (interface embedding); NoOp / Simple / OTel implementations are swappable; exactly the interfaces post’s adapter.
• A span propagates via a private contextKey + context.WithValue down the chain; eight severity levels map to OTel.
• Observability is built on interfaces, so changing the backend doesn’t touch business code.

Source: yshengliao/gortex.

---

Ideas and technical judgement by Sheng; drafted with Claude · Go 1.25 (gortex go.mod) · compiled retroactively · part of the 2026-06-13 blog renovation, paint still drying.