Observability: Three-State Health Checks and a Tracing Abstraction

Three-state health (Healthy / Degraded / Unhealthy) maps to K8s probes better than up/down. Tracing depends on just a Tracer interface, with OTel as an adapter. This post covers interface-driven observability.

Prerequisite: A3 Interfaces and Composition.

Three-state health checks

Traditional health checks are up/down, but the real world is often “still serving, but on the edge”. Gortex uses three states, close to K8s readiness / liveness:

const (
    HealthStatusHealthy   HealthStatus = "healthy"
    HealthStatusDegraded  HealthStatus = "degraded"  // still serving, but near the limit
    HealthStatusUnhealthy HealthStatus = "unhealthy"
)

Aggregation is worst-wins: any check Unhealthy makes the whole Unhealthy; otherwise any Degraded makes it Degraded; all good is Healthy. The built-in MemoryHealthCheck shows it: over the limit is Unhealthy, and at 80% it flags Degraded as an early warning.

Concurrent checks with timeouts

HealthChecker runs a background goroutine on a ticker (runChecks), caches the results, and endpoints like /healthz read the cache instead of re-running on every hit.

When they do run, all checks run concurrently, each with its own timeout, so one slow check can’t drag the rest down:

for name, check := range checks {
    wg.Add(1)
    go func(n string, c HealthCheck) {
        defer wg.Done()
        checkCtx, cancel := context.WithTimeout(ctx, hc.timeout)
        defer cancel()
        result := c(checkCtx) // each check gets its own deadline
        // ...store result
    }(name, check)
}
wg.Wait()

A sync.WaitGroup waits for them all. This is A4’s concurrency and A5’s sync primitives applied to the operations side.

The Tracer interface and OTel adapter

This section is A3’s interface-driven design in the flesh. The framework depends on one small interface:

type Tracer interface {
    StartSpan(ctx context.Context, operation string) (context.Context, *Span)
    FinishSpan(span *Span)
    AddTags(span *Span, tags map[string]string)
    SetStatus(span *Span, status SpanStatus)
}
type EnhancedTracer interface { // interface embedding
    Tracer
    StartEnhancedSpan(ctx context.Context, operation string) (context.Context, *EnhancedSpan)
}

Three implementations ship: NoOpTracer (tracing off), SimpleTracer (in-memory, for development), and OTelTracerAdapter (wired to OpenTelemetry). Moving from in-memory to Jaeger / OTel is just swapping an implementation, with business code untouched — that’s A3’s adapter. Inside, the adapter holds a severityMap that maps gortex’s eight severity levels (DEBUG=10 … EMERGENCY=80) onto OTel attributes.

Context propagation

How does a span follow the request chain? Through context, using a private contextKey type so keys can’t collide:

type contextKey string
const spanContextKey contextKey = "span"

func ContextWithSpan(ctx context.Context, span *Span) context.Context {
    return context.WithValue(ctx, spanContextKey, span)
}
func SpanFromContext(ctx context.Context) *Span { /* ctx.Value(spanContextKey) */ }

StartSpan reads the parent span with SpanFromContext, creates a child, and puts the child back into the ctx to pass down. The whole chain is threaded through context — no globals, no hand-passing.

Takeaways

• Three-state health (Healthy / Degraded / Unhealthy) maps to K8s probes better than up/down; aggregation is worst-wins.
• Checks run on a background ticker and are cached; when run, a WaitGroup runs them concurrently with per-check timeouts, so the slow don’t drag the fast.
• Tracing depends only on the Tracer / EnhancedTracer interfaces (interface embedding); NoOp / Simple / OTel implementations are swappable — exactly A3’s adapter.
• A span propagates via a private contextKey + context.WithValue down the chain; eight severity levels map to OTel.
• Observability is built on interfaces, so changing the backend doesn’t touch business code.

Source: yshengliao/gortex.

---

Outline by Sheng, drafted with Claude · Go 1.25 (gortex go.mod) · compiled retroactively · part of the 2026-06-13 blog renovation, paint still drying.