# Observability: Three-State Health Checks and a Tracing Abstraction

> Gortex health checks report Healthy, Degraded or Unhealthy, closer to K8s probes than up/down; tracing depends on one Tracer interface, with OTel as an adapter.

Source: https://sheng.page/en/posts/gortex-observability-health-tracing/ · Published: 2026-06-06 · Author: Sheng

Three-state health (Healthy / Degraded / Unhealthy) carries more information than a binary up/down check. Tracing depends on a small `Tracer` interface, with OTel connected through an adapter.

> Prerequisite: [Interfaces and Composition](/en/posts/go-interfaces-and-composition/)

## Three-state health checks

Traditional health checks are up/down, but the real world is often "still serving, but on the edge". Gortex uses three states, close to K8s readiness / liveness:

```go
const (
    HealthStatusHealthy   HealthStatus = "healthy"
    HealthStatusDegraded  HealthStatus = "degraded"  // still serving, but near the limit
    HealthStatusUnhealthy HealthStatus = "unhealthy"
)
```

Aggregation is worst-wins: any check Unhealthy makes the whole Unhealthy; otherwise any Degraded makes it Degraded; all good is Healthy. The built-in `MemoryHealthCheck` shows it: over the limit is Unhealthy, and at 80% it flags Degraded as an early warning.

## Concurrent checks with timeouts

`HealthChecker` runs a background goroutine on a ticker (`runChecks`), caches the results, and endpoints like `/healthz` read the cache instead of re-running on every hit.

When they do run, all checks run concurrently, each with its own timeout, so one slow check can't drag the rest down:

```go
for name, check := range checks {
    wg.Add(1)
    go func(n string, c HealthCheck) {
        defer wg.Done()
        checkCtx, cancel := context.WithTimeout(ctx, hc.timeout)
        defer cancel()
        result := c(checkCtx) // each check gets its own deadline
        // ...store result
    }(name, check)
}
wg.Wait()
```

A `sync.WaitGroup` waits for them all. This is the concurrency from [Goroutines and Channels](/en/posts/go-goroutines-and-channels/) and the sync primitives from [Memory and Sync Primitives](/en/posts/go-memory-and-sync-primitives/), applied to the operations side.

## The Tracer interface and OTel adapter

This section is the interfaces post's interface-driven design in the flesh. The framework depends on one small interface:

```go
type Tracer interface {
    StartSpan(ctx context.Context, operation string) (context.Context, *Span)
    FinishSpan(span *Span)
    AddTags(span *Span, tags map[string]string)
    SetStatus(span *Span, status SpanStatus)
}
type EnhancedTracer interface { // interface embedding
    Tracer
    StartEnhancedSpan(ctx context.Context, operation string) (context.Context, *EnhancedSpan)
}
```

Three implementations ship: `NoOpTracer` (tracing off), `SimpleTracer` (in-memory, for development), and `OTelTracerAdapter` (wired to OpenTelemetry). Moving from in-memory to Jaeger / OTel is just swapping an implementation, with business code untouched — that's the adapter from the interfaces post. Inside, the adapter holds a `severityMap` that maps gortex's eight severity levels (`DEBUG=10` … `EMERGENCY=80`) onto OTel attributes.

## Context propagation

How does a span follow the request chain? Through `context`, using a private `contextKey` type so keys can't collide:

```go
type contextKey string
const spanContextKey contextKey = "span"

func ContextWithSpan(ctx context.Context, span *Span) context.Context {
    return context.WithValue(ctx, spanContextKey, span)
}
func SpanFromContext(ctx context.Context) *Span { /* ctx.Value(spanContextKey) */ }
```

`StartSpan` reads the parent span with `SpanFromContext`, creates a child, and puts the child back into the ctx to pass down. The whole chain is threaded through context — no globals, no hand-passing.


## Notes

- Three-state health (Healthy / Degraded / Unhealthy) maps to K8s probes better than up/down; aggregation is worst-wins.
- Checks run on a background ticker and are cached; when run, a `WaitGroup` runs them concurrently with per-check timeouts, so the slow don't drag the fast.
- Tracing depends only on the `Tracer` / `EnhancedTracer` interfaces (interface embedding); NoOp / Simple / OTel implementations are swappable; exactly the interfaces post's adapter.
- A span propagates via a private `contextKey` + `context.WithValue` down the chain; eight severity levels map to OTel.
- Observability is built on interfaces, so changing the backend doesn't touch business code.

Source: [yshengliao/gortex](https://github.com/yshengliao/gortex).

---

> Ideas and technical judgement by Sheng; drafted with Claude · Go 1.25 (gortex go.mod) · compiled retroactively · part of the 2026-06-13 blog renovation, paint still drying.