Protobuf as the Single Source of Truth: Serving gRPC and HTTP/JSON from One Go Port

A few years ago, a colleague and I were choosing a format for real-time messaging. Neither of us liked protobuf, so we settled on msgpack internally and JSON for browsers. Later I wrote my own contract-first gRPC framework, gortexa, with .proto as the single source of truth (SSOT) and one port serving gRPC, HTTP/JSON and MCP.

What changed my mind was having one contract. With gRPC-first, the types, HTTP routes and tools an AI agent can call all come from the same .proto. Binary encoding being faster than JSON is a bonus. Sharing a port comes at a cost, though: grpc.Server.ServeHTTP is an experimental API, all three protocols use the same http.Server timeouts, and a .proto won’t design your error semantics for you. I’ll use a minimal example run on Go 1.27.2 and grpc-go v1.84.0, then go through the problems I found while reviewing gortexa.

1. Why we disliked protobuf back then

Business logic changed quickly. Every field change meant editing the .proto, regenerating the code and updating the implementation. Binary packet captures couldn’t be read directly, which made debugging harder than with JSON, and browsers couldn’t speak gRPC directly. Copies of the .proto kept in more than one place brought another problem: ours often lagged behind. We’d build against the old version, discover a field had changed, and the work went in the bin.

msgpack worked well with Redis, so we used it between internal services and kept JSON for browsers. With no schema, there was no schema to keep in sync.

Binary packets and the lack of native gRPC in browsers are still limits of the format and protocol. Tools and gateways can work around them. The tedious field changes and stale .proto files had a different cause: we treated the .proto as another document. The code, the .proto and the frontend API docs were maintained separately, and someone had to chase up any missed update. Any contract that exists in two copies drifts out of sync sooner or later, and that is where the wasted work came from. Making .proto the contract source and generating each entry point’s code from it addressed that part of the problem.

2. One contract, everything else generated from it

Semantics Lost at the Boundary, and Why the Monolith Keeps Pulling argued that data crossing a service boundary becomes bytes. A contract then has to restore the meaning that the type system used to check. That post left the question of how unanswered; this one picks it up.

A contract can restore more meaning when it stays close to the code. OpenAPI can generate code too. The trouble comes when a spec is kept in sync by hand and drifts behind the implementation. If both sides generate their types from one contract file, renaming a field makes the compiler flag references that no longer exist after regeneration. A type change is caught only if the generated Go type changes too: int32 to sint32 leaves it as int32. That catches some type mismatches across service boundaries.

The compiler still can’t tell whether the other side has regenerated and deployed. Versioning rules and compatibility checks have to cover that.

gortexa generates three entry points from one .proto:

gRPC: the types and service interfaces from protoc-gen-go and protoc-gen-go-grpc.
HTTP/JSON: grpc-gateway reads google.api.http annotations and generates a reverse proxy that turns RESTful JSON into gRPC calls.
MCP: gortexa.ai.v1 annotations mark which RPCs are exposed as tools for AI agents, and tool calls go through the same interceptor chain as other requests.

According to the gortexa docs, both the gateway and the MCP bridge call back into the gRPC server over an in-process loopback. RPCs from all three entry points therefore pass through one interceptor chain and share an error model. A single port brings the entry points together; sharing auth and observability still requires each one to feed into that path. The tools/list case later in this post shows what happens when one doesn’t.

My earlier framework, gortex, took another route: routes declared in struct tags, with the Go code itself as the source of the contract.

ConnectRPC offers a similar approach. It joined the CNCF Sandbox in April 2024. A connect-go server accepts Connect, gRPC and gRPC-Web by default, and Connect runs unary and one-way streaming calls over HTTP/1.1 (bidi still needs HTTP/2), so one .proto serves several protocols here too.

3. Routing on content type over one port

grpc-go’s *grpc.Server implements http.Handler, so it can sit under the standard library’s http.Server. Its documentation gives the routing pattern: HTTP/2 requests whose Content-Type starts with application/grpc go to gRPC, and everything else goes to an ordinary mux.

Cleartext HTTP/2 (h2c) used to need a wrapper from golang.org/x/net/http2/h2c. Go 1.24 added the http.Server field Protocols: enable UnencryptedHTTP2 and you’re done, which is how gortexa handles it. The example uses grpc-go’s built-in health service, saving a protoc run. Its JSON endpoint calls the same health implementation directly to demonstrate routing. It bypasses the gRPC server’s interceptors; gortexa’s grpc-gateway setup forwards through a gRPC call.

package main

import (
	"encoding/json"
	"errors"
	"flag"
	"log"
	"net/http"
	"strings"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/health"
	healthpb "google.golang.org/grpc/health/grpc_health_v1"
	"google.golang.org/grpc/reflection"
)

func main() {
	readTimeout := flag.Duration("read-timeout", 2*time.Second, "http.Server.ReadTimeout")
	writeTimeout := flag.Duration("write-timeout", 0, "http.Server.WriteTimeout")
	flag.Parse()

	hs := health.NewServer()
	hs.SetServingStatus("demo.Orders", healthpb.HealthCheckResponse_SERVING)

	gs := grpc.NewServer()
	healthpb.RegisterHealthServer(gs, hs)
	reflection.Register(gs) // ServerReflectionInfo is a bidi stream, used to show ReadTimeout

	// The HTTP/JSON endpoint calls the same health implementation, with no separate state
	mux := http.NewServeMux()
	mux.HandleFunc("GET /v1/health/{service}", func(w http.ResponseWriter, r *http.Request) {
		resp, err := hs.Check(r.Context(), &healthpb.HealthCheckRequest{Service: r.PathValue("service")})
		w.Header().Set("Content-Type", "application/json")
		if err != nil {
			w.WriteHeader(http.StatusNotFound)
			_ = json.NewEncoder(w).Encode(map[string]string{"error": err.Error()})
			return
		}
		_ = json.NewEncoder(w).Encode(map[string]string{
			"status": resp.GetStatus().String(),
			"proto":  r.Proto,
		})
	})

	root := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if r.ProtoMajor == 2 && strings.HasPrefix(r.Header.Get("Content-Type"), "application/grpc") {
			gs.ServeHTTP(w, r)
			return
		}
		mux.ServeHTTP(w, r)
	})

	var protocols http.Protocols
	protocols.SetHTTP1(true)
	protocols.SetUnencryptedHTTP2(true)

	srv := &http.Server{
		Addr:              "127.0.0.1:8080",
		Handler:           root,
		Protocols:         &protocols,
		ReadHeaderTimeout: 5 * time.Second,
		ReadTimeout:       *readTimeout,  // deliberately short, to see its effect on streams
		WriteTimeout:      *writeTimeout, // 0 by default; a value cuts off long server streams
	}
	log.Printf("listening on %s (HTTP/1.1 + h2c)", srv.Addr)
	if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
		log.Fatalf("serve: %v", err)
	}
}

curl reaches the JSON endpoint over both HTTP/1.1 and h2c:

$ curl -s http://127.0.0.1:8080/v1/health/demo.Orders
{"proto":"HTTP/1.1","status":"SERVING"}
$ curl -s --http2-prior-knowledge http://127.0.0.1:8080/v1/health/demo.Orders
{"proto":"HTTP/2.0","status":"SERVING"}
$ curl -s http://127.0.0.1:8080/v1/health/demo.Payments
{"error":"rpc error: code = NotFound desc = unknown service"}

A gRPC client on the same port tries a unary call, a server stream and a bidi stream in turn, and the last two deliberately run past the server’s 2-second timeout:

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	healthpb "google.golang.org/grpc/health/grpc_health_v1"
	refpb "google.golang.org/grpc/reflection/grpc_reflection_v1"
	"google.golang.org/grpc/status"
)

func main() {
	if err := run(); err != nil {
		log.Fatal(err)
	}
}

func run() error {
	conn, err := grpc.NewClient("127.0.0.1:8080", grpc.WithTransportCredentials(insecure.NewCredentials()))
	if err != nil {
		return fmt.Errorf("new client: %w", err)
	}
	defer conn.Close()

	hc := healthpb.NewHealthClient(conn)

	// 1. unary
	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
	defer cancel()
	resp, err := hc.Check(ctx, &healthpb.HealthCheckRequest{Service: "demo.Orders"})
	if err != nil {
		return fmt.Errorf("check: %w", err)
	}
	fmt.Println("[unary] Check:", resp.GetStatus())

	// 2. server streaming: the client cancels after 3s, past the server's 2s timeout
	wctx, wcancel := context.WithCancel(context.Background())
	defer wcancel()
	time.AfterFunc(3*time.Second, wcancel)
	ws, err := hc.Watch(wctx, &healthpb.HealthCheckRequest{Service: "demo.Orders"})
	if err != nil {
		return fmt.Errorf("watch: %w", err)
	}
	start := time.Now()
	for {
		m, err := ws.Recv()
		if err != nil {
			fmt.Printf("[server-stream] Watch ended after %.1fs: %s\n", time.Since(start).Seconds(), status.Code(err))
			break
		}
		fmt.Println("[server-stream] Watch:", m.GetStatus())
	}

	// 3. bidi streaming: the second message is sent after 3s
	rc := refpb.NewServerReflectionClient(conn)
	bctx, bcancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer bcancel()
	bs, err := rc.ServerReflectionInfo(bctx)
	if err != nil {
		return fmt.Errorf("reflection stream: %w", err)
	}
	start = time.Now()
	for i := 1; i <= 2; i++ {
		if i == 2 {
			time.Sleep(3 * time.Second)
		}
		req := &refpb.ServerReflectionRequest{
			MessageRequest: &refpb.ServerReflectionRequest_ListServices{ListServices: ""},
		}
		if err := bs.Send(req); err != nil {
			fmt.Printf("[bidi] send #%d at %.1fs: %v\n", i, time.Since(start).Seconds(), err)
		}
		r, err := bs.Recv()
		if err != nil {
			fmt.Printf("[bidi] recv #%d at %.1fs: %s: %s\n", i, time.Since(start).Seconds(), status.Code(err), status.Convert(err).Message())
			return nil
		}
		fmt.Printf("[bidi] recv #%d at %.1fs: %d services\n", i, time.Since(start).Seconds(), len(r.GetListServicesResponse().GetService()))
	}
	return nil
}

I’ll leave the output for the timeout examples in the next section. ServeHTTP has its own limitations: the grpc-go source comment says ServeHTTP uses the Go standard library’s HTTP/2 implementation, entirely separate from grpc-go’s own HTTP/2 server. Performance and features may differ, and some gRPC features require grpc-go’s server. The API is marked EXPERIMENTAL and may be changed or removed.

gortexa’s README documents this trade-off in handler mode: external gRPC goes through ServeHTTP, with some transport behaviour differing from a dedicated gRPC port. It suggests cmux if you need full grpc-go semantics.

cmux also exposes one port, inspecting the first bytes of an accepted connection before handing it to grpc-go’s own server or to http.Server for HTTP. Routing per connection means that connection can’t mix the two protocols. TLS needs extra care too: the cmux README notes that handlers don’t get http.Request.TLS. Both approaches leave you with one external port and one load balancer and ingress configuration.

4. Three protocols, one set of timeouts

Sharing one http.Server means gRPC, JSON and MCP share the same ReadTimeout and WriteTimeout. Values that are sensible for ordinary HTTP break streaming RPCs.

The example server defaults to a 2-second ReadTimeout and a WriteTimeout of 0. With the client from the previous section:

### server (ReadTimeout=2s, WriteTimeout=0)
[unary] Check: SERVING
[server-stream] Watch: SERVING
[server-stream] Watch ended after 3.0s: Canceled
[bidi] recv #1 at 0.0s: 3 services
[bidi] send #2 at 3.0s: EOF
[bidi] recv #2 at 3.0s: Unavailable: i/o timeout

Watch stays open until the client cancels at 3 seconds. The second bidi message never gets through; the stream ends with Unavailable: i/o timeout. The net/http documentation defines ReadTimeout as the maximum time allowed to read the entire request, including its body. Under HTTP/2, each stream has a timer running from its start. When that expires, the request body is closed.

For client streaming and bidi, the body consists of messages the client keeps sending. Once the stream outlives ReadTimeout, later messages can’t be read. A server stream such as Watch sends one request and finishes its request body, so it isn’t affected.

Keep the 2-second ReadTimeout and add a 2-second WriteTimeout, and the server stream is cut off too:

### server -write-timeout=2s
[unary] Check: SERVING
[server-stream] Watch: SERVING
[server-stream] Watch ended after 2.0s: Internal
[bidi] recv #1 at 0.0s: 3 services
[bidi] send #2 at 3.0s: EOF
[bidi] recv #2 at 3.0s: Internal: stream terminated by RST_STREAM with error code: INTERNAL_ERROR

Only with ReadTimeout set to 0 does the bidi stream run to completion:

### server -read-timeout=0
[unary] Check: SERVING
[server-stream] Watch: SERVING
[server-stream] Watch ended after 3.0s: Canceled
[bidi] recv #1 at 0.0s: 3 services
[bidi] recv #2 at 3.0s: 3 services

I found the same problem when reviewing gortexa. The shared h2c server already set WriteTimeout to 0 for long-lived connections such as Health.Watch and SSE, but still had a 15-second ReadTimeout. Client streams and bidi streams were cut off after 15 seconds. The read timeout had been overlooked.

You can set ReadTimeout to 0 and keep ReadHeaderTimeout for HTTP/1.x headers and the initial h2c reads, leaving body deadlines to individual handlers. Or keep the timeout and clear the read deadline for gRPC requests at the routing point with http.ResponseController:

if r.ProtoMajor == 2 && strings.HasPrefix(r.Header.Get("Content-Type"), "application/grpc") {
	if err := http.NewResponseController(w).SetReadDeadline(time.Time{}); err != nil {
		log.Printf("clear read deadline: %v", err)
	}
	gs.ServeHTTP(w, r)
	return
}

With this added to the server using a 2-second ReadTimeout, the bidi message at 3 seconds gets through ([bidi] recv #2 at 3.0s: 3 services). The JSON endpoint keeps its original timeout. Clearing the deadline leaves the RPC deadline and the server’s own limits to control how long a stream stays open. Keepalive only checks whether a connection still responds; it does not cap a stream’s lifetime on a healthy connection. If the client supplies no deadline, the server has to set a limit.

The review also found two inconsistencies in the checks at each entry point:

The MCP bridge reads bodies through an io.LimitReader, while the gateway’s JSON entry point had no limit. An oversized JSON body would be read entirely into memory. http.MaxBytesReader suits JSON bodies, but wrapping it around everything before routing would limit all the messages in a gRPC stream together. Add r.Body = http.MaxBytesReader(w, r.Body, 32) at the top of the example’s routing handler and have the client send ListServices repeatedly on one bidi stream. Each request is 7 bytes including the gRPC frame: the first 4 succeed, then the fifth ends the stream with Unavailable: http: request body too large. On the gRPC side, use grpc.MaxRecvMsgSize (4 MB by default) for individual messages and manage the stream’s cumulative total in the handler.
MCP’s tools/list returns every tool without any auth, and only tools/call carries Authorization into the interceptor chain. The calls are protected, but the tool list publishes what the service can do. gRPC’s reflection service has the same problem; in production, either turn it off or require authentication for it too.

A later round of production hardening dealt with these. The ReadTimeout cutting off long-lived streams and the missing limit on gateway JSON bodies were both fixed, and the ServeHTTP limitations went into the docs. In the same round, gRPC reflection became a config switch that is off by default, and outgoing headers were narrowed to pass only X-Request-Id. tools/list still lists tools without auth; the docs state this as the current behaviour, with auth checked at tools/call.

5. Opening the proto files for others to import

A .proto package name can be anything while only the framework uses it. Expose it as a public API and the name starts to matter.

Go’s protobuf runtime keeps a global registry. According to the official protobuf FAQ, every declaration (messages, enums and so on) has a full name. Two declarations sharing that name in one program cause a namespace conflict, which has been a startup error since google.golang.org/protobuf v1.26.0. The protoregistry source also checks .proto file paths: registering two different files under the same path produces file ... is already registered in the global registry.

Sharing a package name is fine in itself. Repeating a full declaration name or file path causes the conflict, and generic package names make both more likely. gortexa’s MCP annotations originally lived in ai/v1, without a vendor prefix. If a project using the framework also had a file or declaration with the same name under ai/v1, linking both would panic at init. The FAQ recommends choosing globally unique .proto package names, for example by including a company or project name.

That meant changing ai/v1 to gortexa/ai/v1. The FAQ warns that renaming packages after the fact breaks backwards compatibility for extensions, google.protobuf.Any and gRPC services. Annotations are extensions, and their binary encoding only uses field numbers. If the number, the options message being extended and the field type stay the same, options in descriptors produced by old code still decode. The numbers were kept and pinned with a test to prevent accidental changes. Code referring to the old import path and names still needs updating; binary compatibility doesn’t preserve source compatibility.

All the generated code under gen/ was committed to the repo too. The Go Modules Reference says go mod tidy loads every package in the main module, the packages they import, and those packages’ tests, including tests in other modules. A project importing gortexa makes go mod tidy resolve packages all the way down into gen/. Without the generated code in the repo, that fails. Unit tests didn’t catch this. I found it by testing as a consumer in the example sandbox: I created a scratch module that imports gortexa, then ran go mod tidy, build and vet.

Committing generated code creates another job: catching .proto changes that haven’t been regenerated. CI regenerates and compares, but git diff --exit-code only checks tracked files. New files slip past:

$ echo b > gen/b.pb.go
$ git diff --exit-code -- gen; echo "git diff exit=$?"
git diff exit=0
$ git status --porcelain -- gen
?? gen/b.pb.go

An RPC added to an existing service usually changes existing output files, so git diff catches it. A new .proto produces new files that git diff misses. gortexa’s CI now uses git status --porcelain -- gen and fails if it produces any output, covering both cases.

Publishing generated code also means dealing with changes such as Protobuf Edition 2024. The Go blog introduced the Opaque API in December 2024: struct fields are no longer exported, so callers must use accessors. The protobuf docs specify api_level as API_OPAQUE by default in Edition 2024 and API_OPEN in Edition 2023. Between them, API_HYBRID keeps exported fields alongside accessors to allow a gradual migration. An edition upgrade in a project that publishes generated code directly affects its users’ code.

6. Error semantics still need a separate design

Later I used gortexa to prototype a system for a game provider. I only used three packages, kernel, health and observability. The error handling I ended up writing myself.

gortexa’s error code registry allows exactly one category per gRPC code, so one table can convert errors into the gRPC, HTTP and MCP formats. Several business errors can share a gRPC code. An insufficient balance and an order in the wrong state can both be FailedPrecondition, yet callers need to distinguish them. Custom codes for those cases can’t be registered. The Recovery interceptor then normalises errors, erasing a custom code even if the handler returned one.

I wrote my own interceptor and a domain error type implementing GRPCStatus(), with the business error code in google.rpc.ErrorInfo:

package main

import (
	"errors"
	"fmt"

	"google.golang.org/genproto/googleapis/rpc/errdetails"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"
)

// DomainError carries a business error code, placed in ErrorInfo when converted to a gRPC status.
type DomainError struct {
	Code   codes.Code
	Reason string // business error code, e.g. INSUFFICIENT_BALANCE
	Msg    string
	Meta   map[string]string
}

func (e *DomainError) Error() string { return e.Reason + ": " + e.Msg }

// GRPCStatus lets status.FromError get a status with details directly.
func (e *DomainError) GRPCStatus() *status.Status {
	st := status.New(e.Code, e.Msg)
	withInfo, err := st.WithDetails(&errdetails.ErrorInfo{
		Reason:   e.Reason,
		Domain:   "wallet.example.com",
		Metadata: e.Meta,
	})
	if err != nil {
		return st
	}
	return withInfo
}

func debit() error {
	de := &DomainError{
		Code:   codes.FailedPrecondition,
		Reason: "INSUFFICIENT_BALANCE",
		Msg:    "balance too low",
		Meta:   map[string]string{"required": "500", "available": "120"},
	}
	return fmt.Errorf("debit order 42: %w", de)
}

func main() {
	err := debit()
	st, ok := status.FromError(err)
	fmt.Println("ok:", ok)
	fmt.Println("code:", st.Code())
	fmt.Println("message:", st.Message())
	for _, d := range st.Details() {
		if info, ok := d.(*errdetails.ErrorInfo); ok {
			fmt.Println("reason:", info.GetReason(), "domain:", info.GetDomain(), "metadata:", info.GetMetadata())
		}
	}
	var de *DomainError
	fmt.Println("errors.As:", errors.As(err, &de))
}

ok: true
code: FailedPrecondition
message: debit order 42: INSUFFICIENT_BALANCE: balance too low
reason: INSUFFICIENT_BALANCE domain: wallet.example.com metadata: map[available:120 required:500]
errors.As: true

After wrapping once with %w, status.FromError still finds GRPCStatus(), preserving the code and ErrorInfo. The message, however, becomes the full err.Error() string. As the grpc-go documentation explains, wrapping an error includes its full text in the returned message, which can expose internal details to the client. The interceptor must decide how much to send.

Google’s AIP-193 defines the standard ErrorInfo field. Its reason is an UPPER_SNAKE_CASE machine-readable identifier; together with domain, it uniquely identifies an error and must remain stable over time. Dynamic information belongs in metadata. The gRPC docs list support for this richer error model in the C++, Go, Java, Python and Ruby libraries. They also warn that implementations can differ across languages, and ordinary proxies and logging tools can’t see the details. Check the libraries your callers use before relying on it.

A .proto describes requests and responses. It has no built-in syntax for declaring errors, so comments or custom options must spell out which errors can occur and how callers should handle them. A framework’s error model provides a default. You still have to design the business error categories and check that no interceptor erases them on the way out.

Notes

Our tedious field changes and stale .proto files came from maintaining several copies of the contract. Generating code from .proto as the single source reduces manual syncing. Binary packets that can’t be read directly and browsers without native gRPC remain limits of the format and protocol; an SSOT doesn’t fix them.
grpc.Server.ServeHTTP lets gRPC and ordinary HTTP share a port, and since Go 1.24 http.Server.Protocols enables h2c directly. The cost is that ServeHTTP is experimental and differs in places from grpc-go’s own server. cmux, which routes per connection, is another single-port approach.
A shared http.Server means shared timeouts: ReadTimeout cuts off client streams and bidi streams, WriteTimeout cuts off server streams, and ResponseController can clear the read deadline for gRPC requests at the routing point.
Body limits and auth need checking at every entry point. MaxBytesReader on a gRPC stream limits the stream’s cumulative total, while gRPC’s MaxRecvMsgSize limits a single message. MCP’s tools/list and gRPC reflection both reveal what a service can do.
When .proto files are opened for import, a repeated full declaration name or file path panics at init, so add a vendor prefix to the package name and pin extension numbers with a test. Commit generated code and catch drift in CI with git status --porcelain.
Error semantics need their own design alongside the data structures in .proto. Carry business error codes in a domain error that implements GRPCStatus() with ErrorInfo, and check that no interceptor normalises them away.