Unpredictable Game RNG in Production, Replayable in Tests: Separating the Two in Go

A project I worked on switched the PRNG used by its probability modules and game servers to a single AES-CTR implementation. Once it went live, the old probability package deadlocked. Every game that depended on it stopped responding. The new RNG package took the same lock twice along one call path, and Go’s sync.Mutex is not reentrant. We switched back to the old RNG straight away and tested the new one separately for stability before putting it back into service.

The failure came from how the RNG was called, along a path the tests had never exercised. Production needs results nobody can predict, while tests and dispute handling need to replay them. Making one RNG instance do both tends to leave neither working well. In the approach below, game logic depends only on a Source interface. Production supplies an unpredictable source; tests use a ChaCha8 seeded through HMAC. Dispute replays use the raw values each round actually drew, without re-running the RNG.

For production, crypto/rand lets the application avoid holding or managing a seed altogether. A CSPRNG package seeded from the operating system is another option. Section 2 compares the trade-offs.

Seed space, modulo bias, rejection sampling and the GLI-11 clauses were covered in Only 86 Million Ways to Shuffle: What Game RNG Certification Checks, so I won’t repeat them. Every example here was run on Go 1.24.7. At the time of writing the latest Go release is 1.27.2; the APIs used are all still there, and anything that needs Go 1.26 is flagged.

1. The same lock, taken twice

The smallest example of the failure uses an AES-256-CTR keystream, with the key and IV supplied by the operating system. Each Uint64 call takes the next 8 bytes. The type has no lock of its own:

type AESCTR struct {
	stream cipher.Stream
	buf    [8]byte
}

func NewAESCTR() (*AESCTR, error) {
	var key [32]byte
	var iv [aes.BlockSize]byte
	rand.Read(key[:]) // since Go 1.24 this never returns an error; on failure the program exits
	rand.Read(iv[:])
	block, err := aes.NewCipher(key[:])
	if err != nil {
		return nil, fmt.Errorf("new aes cipher: %w", err)
	}
	return &AESCTR{stream: cipher.NewCTR(block, iv[:])}, nil
}

func (a *AESCTR) Uint64() uint64 {
	clear(a.buf[:])
	a.stream.XORKeyStream(a.buf[:], a.buf[:])
	return binary.LittleEndian.Uint64(a.buf[:])
}

The package wraps it in one lock guarding all internal state. The public Uint64 method takes that lock. So does Draw, another public method that fetches several values at once. With the lock already held, it calls Uint64:

type LockedRNG struct {
	mu  sync.Mutex
	src *AESCTR
}

func (r *LockedRNG) Uint64() uint64 {
	r.mu.Lock()
	defer r.mu.Unlock()
	return r.src.Uint64()
}

// Draw fetches k values at once (say, five reels in one spin). It takes the lock,
// then calls Uint64, which takes the same lock.
func (r *LockedRNG) Draw(k int) []uint64 {
	r.mu.Lock()
	defer r.mu.Unlock()
	out := make([]uint64, k)
	for i := range out {
		out[i] = r.Uint64()
	}
	return out
}

func main() {
	src, err := NewAESCTR()
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
	r := &LockedRNG{src: src}
	fmt.Println("uint64:", r.Uint64())
	fmt.Println("draw:", r.Draw(5))
}

uint64: 9135510418585151819
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [sync.Mutex.Lock]:
...
sync.(*Mutex).Lock(...)
	/usr/local/go1.24.7/src/sync/mutex.go:46
main.(*LockedRNG).Uint64(0x4d1c20?)
	aes/locked.go:12 +0x4a
main.(*LockedRNG).Draw(0xc0000120f0, 0x5)
	aes/locked.go:24 +0xa5
main.main()
	aes/main.go:16 +0x114
exit status 2

The sync.Mutex documentation says that if the lock is already in use, the calling goroutine blocks until the mutex is available. The mutex has no idea who holds it. A goroutine that locks it a second time waits, and the goroutine it is waiting for is itself. In a standalone program the runtime reports a fatal error, which makes it look easy to catch.

Put it in a server and it gets harder to spot. The runtime only declares a deadlock when every goroutine is asleep, and a server always has goroutines waiting on network connections. The process carries on running; every request that touches the lock just sits there. Here is the same LockedRNG shared by two HTTP handlers. The client has a 2-second timeout and calls /coin, /slot and /coin in turn:

src, err := NewAESCTR()
if err != nil {
	return fmt.Errorf("new shared rng: %w", err)
}
shared := &LockedRNG{src: src} // one instance shared by every game

mux.HandleFunc("/slot", func(w http.ResponseWriter, _ *http.Request) {
	fmt.Fprintln(w, shared.Draw(5))
})
mux.HandleFunc("/coin", func(w http.ResponseWriter, _ *http.Request) {
	fmt.Fprintln(w, shared.Uint64()&1)
})

/coin  0
/slot  timeout after 2s
/coin  timeout after 2s
goroutines blocked in sync.Mutex.Lock: 2

The first /coin answers normally. /slot enters Draw and stalls there holding the lock, and after that even /coin, which only calls Uint64, queues behind the same lock. Only one call path deadlocked, yet every game sharing the instance is stuck. The process is still alive, and a health check that only tests whether the port is open sees nothing wrong.

One fix is to have public methods take the lock and call internal versions that don’t lock again. The internal names state what callers must do first:

func (r *LockedRNG) Draw(k int) []uint64 {
	r.mu.Lock()
	defer r.mu.Unlock()
	out := make([]uint64, k)
	for i := range out {
		out[i] = r.uint64Locked()
	}
	return out
}

// uint64Locked must be called with r.mu already held.
func (r *LockedRNG) uint64Locked() uint64 { return r.src.Uint64() }

Each round can also create its own RNG without a lock, used only by the goroutine handling that round. Giving it a separate copy of the double-locking object would still leave the same goroutine waiting on itself. Here each round creates an AESCTR directly, without a lock and with its own key and IV:

// Each round gets its own lock-free AESCTR, used only by the goroutine handling that round.
round, err := NewAESCTR()
if err != nil {
	return fmt.Errorf("new round rng: %w", err)
}

Both fixes were run across 8 goroutines for 10,000 rounds each, with -race on and a 30-second deadline. An unfinished run at that point counts as a likely deadlock:

$ go run -race .
done: 8 goroutines x 10000 rounds

With Security > Simplicity > Correctness > Performance as the priority order, I’d choose the second fix. No shared state means no lock and no lock ordering to get wrong. One round’s RNG state also can’t be observed from another round. The first fix depends on every maintainer remembering the naming convention, which gets harder to guarantee as the package grows.

Swapping the RNG replaces a dependency every game shares in one go. Check the package’s concurrency design before certification, and before release run at least one test that calls every public method from several goroutines at once. -race finds data races. It won’t find a deadlock, so the test also needs a deadline and only passes if it finishes in time. The 8-goroutine program above is the smallest version of that test. The features need a full regression pass as well: around the same time, the test environment started sending an empty board-recovery message the moment a player entered a game, which the logs eventually traced to account state and the scene-recovery logic.

2. What math/rand/v2 and crypto/rand are each for

Go 1.22 added math/rand/v2, the first v2 package in the standard library, with changes to how callers seed and use an RNG.

Top-level functions (rand.IntN, rand.Float64 and so on) use ChaCha8 by default and seed themselves. v2 removes the top-level Seed function, so they can’t be made reproducible. In v1, rand.Seed was already deprecated in Go 1.20.

For reproducibility, construct a ChaCha8 or a PCG yourself: NewChaCha8(seed [32]byte) and NewPCG(seed1, seed2 uint64). Both implement encoding.BinaryMarshaler, so their state can be saved and restored. ChaCha8 gained its Read method in Go 1.23.

The package documentation says it should not be used for security-sensitive work, yet the ChaCha8 type is described as cryptographically strong. The Go team’s article on ChaCha8Rand explains why: accidental misuse of math/rand is no longer a security catastrophe. For work that requires secure randomness, it still recommends crypto/rand.

crypto/rand changed in Go 1.24 too. Read is now guaranteed not to fail and always returns nil; if it truly can’t read, the program crashes instead of letting the caller carry on with a half-filled buffer. The same release switched to getrandom via vDSO on Linux 6.11 and later, which the release notes say is several times faster for small reads.

Here are the reproducible and non-reproducible sources side by side:

func first3(src mrand.Source) [3]uint64 {
	return [3]uint64{src.Uint64(), src.Uint64(), src.Uint64()}
}

func main() {
	seed := [32]byte{1, 2, 3}
	fmt.Println("chacha8 a:", first3(mrand.NewChaCha8(seed)))
	fmt.Println("chacha8 b:", first3(mrand.NewChaCha8(seed)))
	fmt.Println("pcg     a:", first3(mrand.NewPCG(1, 2)))
	fmt.Println("pcg     b:", first3(mrand.NewPCG(1, 2)))

	// save the state part-way, restore it, and the following output is identical
	c := mrand.NewChaCha8(seed)
	c.Uint64()
	state, err := c.MarshalBinary()
	if err != nil {
		fmt.Fprintln(os.Stderr, "marshal:", err)
		os.Exit(1)
	}
	var restored mrand.ChaCha8
	if err := restored.UnmarshalBinary(state); err != nil {
		fmt.Fprintln(os.Stderr, "unmarshal:", err)
		os.Exit(1)
	}
	fmt.Println("state bytes:", len(state))
	fmt.Println("original :", first3(c))
	fmt.Println("restored :", first3(&restored))

	for i := 0; i < 2; i++ {
		var b [8]byte
		rand.Read(b[:])
		fmt.Printf("crypto/rand %d: %x\n", i, b)
	}
}

chacha8 a: [18307998996574457915 11107991961950960290 3581772083949013674]
chacha8 b: [18307998996574457915 11107991961950960290 3581772083949013674]
pcg     a: [14192431797130687760 11371241257079532652 14470142590855381128]
pcg     b: [14192431797130687760 11371241257079532652 14470142590855381128]
state bytes: 48
original : [11107991961950960290 3581772083949013674 17560204186880880572]
restored : [11107991961950960290 3581772083949013674 17560204186880880572]
crypto/rand 0: 305916f32fc2d894
crypto/rand 1: 6684342646fec2db

On a second run the two crypto/rand lines became 62f8c833295b5f98 and 7ac6a41bfdf266bd. Everything else was identical. A fixed seed makes test results reproducible, so production needs to keep seeds and internal state from leaking. Here, 48 bytes of state are enough to compute every value that follows. Anything that can be serialised can end up in a log, a snapshot or a debug endpoint.

Chapter 3 of GLI-19 (the technical standard for interactive gaming systems, v3.0) is direct about this. 3.3.1 requires the RNG that determines game outcomes to be cryptographically strong. 3.3.2 requires that it can’t feasibly be compromised by an attacker who knows the source code, that it isn’t seeded from a time value alone, that games don’t share the same initial seed, and that it periodically modifies its state with external entropy. For the production source there are two common approaches:

Use crypto/rand directly. The application no longer has to manage random state itself: there is no seed to store and no reseeding schedule to write. The cost is that Go and the operating system choose the algorithm and entropy source, so a certification submission needs to explain their implementation in the deployed environment.
Use a CSPRNG package you control (for example one built around AES-CTR), seeded from the operating system and periodically reseeded with entropy. The algorithm and reseeding interval live in your own code, so you can walk through them line by line at submission. You also take responsibility for the seed, state, reseeding and concurrency. The deadlock in section 1 came from this layer.

Using the standard library or a well-known algorithm doesn’t prove GLI compliance. The underlying implementation in the deployed environment still needs checking against the submitted evidence; the laboratory decides whether it passes. Generators that accept a seed, such as ChaCha8 and PCG, belong in unit tests, e2e tests and offline probability simulations, where results need to be reproducible.

3. Game logic that only knows about Source

The same game logic needs to work with an unpredictable source in production and a fixed seed in tests. The simplest way is to depend only on the Source interface from math/rand/v2, which has a single Uint64() uint64 method. The caller chooses where the randomness comes from:

// CryptoSource is the production source: it asks the OS every time and keeps no seed.
type CryptoSource struct{}

func (CryptoSource) Uint64() uint64 {
	var b [8]byte
	rand.Read(b[:]) // since Go 1.24 this always returns nil; on failure the program exits
	return binary.LittleEndian.Uint64(b[:])
}

// Spin is the game logic: it only knows the Source interface, not where values come from.
func Spin(src mrand.Source, reelLen []int) []int {
	r := mrand.New(src)
	stops := make([]int, len(reelLen))
	for i, n := range reelLen {
		stops[i] = r.IntN(n)
	}
	return stops
}

The example uses crypto/rand in production. Your own CSPRNG package can replace it as long as it implements Uint64() uint64. With no fields, CryptoSource holds no state at the application layer and can be shared between goroutines without the locking problem from section 1.

Spin leaves the mapping from output to reel position to rand.New(src).IntN. Write that step yourself and you’ll need the line-by-line review for modulo bias described in the previous post.

4. Who holds the seed in tests

One e2e test in my game-provider system prototype checks that resending the same idempotency key returns the same round. Checking the round ID is enough to pass. Start checking the payout or balance change, though, and the production random source gives a different result every run. The demo slot game started out exactly like that: the e2e test depended on luck and now and then tripped a DUPLICATED_OPERATION, leaving me unsure whether the test or the code was wrong.

What fixed it was having the slot game use a derived replay seed in the test environment, built from a secret supplied by an environment variable plus the round and the idempotency key. After that, every e2e run gave the same results. The example below shows the derivation as seed = HMAC-SHA256(secret, round_id). The output is exactly 32 bytes and goes straight into NewChaCha8. The same secret and round ID always give the same round; change the round ID and you get a different one.

That gives reproducible results. Idempotency still needs separate checks: a resend must produce exactly one round and one debit. Duplicate execution can produce identical results too, so leaving out either assertion can hide the bug. The example below covers only seed derivation, with reels set to var reels = []int{32, 32, 40, 40, 48}:

// replaySecret reads the secret from the environment; if unset it generates and logs one,
// so a failing run can be repeated by setting the logged value.
func replaySecret(t *testing.T) []byte {
	t.Helper()
	if s := os.Getenv("RNG_REPLAY_SECRET"); s != "" {
		b, err := hex.DecodeString(s)
		if err != nil {
			t.Fatalf("decode RNG_REPLAY_SECRET: %v", err)
		}
		if len(b) < 32 {
			t.Fatalf("RNG_REPLAY_SECRET: need at least 32 bytes, got %d", len(b))
		}
		return b
	}
	b := make([]byte, 32)
	rand.Read(b)
	t.Logf("RNG_REPLAY_SECRET=%x", b)
	return b
}

// deriveSeed exists only in tests: the same secret and round ID always give the same seed.
func deriveSeed(secret []byte, roundID string) [32]byte {
	mac := hmac.New(sha256.New, secret)
	mac.Write([]byte(roundID))
	var seed [32]byte
	copy(seed[:], mac.Sum(nil))
	return seed
}

func TestSameRoundSameResult(t *testing.T) {
	secret := replaySecret(t)
	a := Spin(mrand.NewChaCha8(deriveSeed(secret, "round-0001")), reels)
	b := Spin(mrand.NewChaCha8(deriveSeed(secret, "round-0001")), reels)
	c := Spin(mrand.NewChaCha8(deriveSeed(secret, "round-0002")), reels)
	t.Logf("round-0001: %v %v", a, b)
	t.Logf("round-0002: %v", c)
	if !slices.Equal(a, b) {
		t.Fatalf("same round, different result: %v vs %v", a, b)
	}
}

Without the environment variable, a random secret is generated and logged. With the same secret set, and the same reel configuration, every run gives the same results (the secret in the second command is truncated here; in practice you paste all 64 hex characters):

$ go test -v -run SameRound .
    slot_test.go:46: RNG_REPLAY_SECRET=0c522080a19191276ebc9a045763d7b8c779081f7c5bfd3b6030811b1803375e
    slot_test.go:50: round-0001: [10 20 13 16 5] [10 20 13 16 5]
    slot_test.go:51: round-0002: [7 0 3 6 10]
--- PASS: TestSameRoundSameResult (0.00s)

$ RNG_REPLAY_SECRET=4464eb15... go test -v -run SameRound .   # identical output on two runs
    slot_test.go:50: round-0001: [22 28 8 22 14] [22 28 8 22 14]
    slot_test.go:51: round-0002: [26 31 30 24 24]
--- PASS: TestSameRoundSameResult (0.00s)

The source must stay in test builds, with the secret managed for that purpose:

deriveSeed lives in a _test.go file, so it never compiles into the production binary, and production has no path from a round ID to a seed. If an e2e test drives a running service, the derived-seed Source has to sit somewhere only a test build includes (a build tag, for instance), and the service should state clearly in its log at startup that it is using a test source.
Round IDs appear in bet records, game history and API responses, where anyone can see them. That’s why this uses HMAC instead of hashing the round ID directly. The secret has to be long enough and kept secret for HMAC to offer any protection, so the example requires at least 32 bytes. It also logs the secret. This is test code, and offers no safety guarantee if someone wires it into a shared environment by mistake.
The secret isn’t hard-coded. CI can change it on every run, and a failing run leaves enough in the log to repeat it.

For code that calls crypto/rand directly, bypassing Source, Go 1.26 added testing/cryptotest.SetGlobalRandom(t, seed). During a test, it replaces the randomness used by crypto/rand and inside the crypto/... packages with a deterministic source. This affects the whole process, so it can’t be used in parallel tests. The documentation also warns that how cryptographic algorithms consume randomness isn’t specified; output may change in later releases. I haven’t run it in this post’s Go 1.24.7 environment, so check the description on pkg.go.dev before relying on it.

5. Replaying disputed rounds from recorded draws

Tests can replay a round because they hold the seed. Production shouldn’t keep it. To reproduce a disputed round, record every raw value the game logic draws at the Source layer and store those values with the round. A replay feeds them back into the same game logic in the same order:

// Recorder records every raw value the game logic draws, stored together with the round.
type Recorder struct {
	Src   mrand.Source
	Draws []uint64
}

func (r *Recorder) Uint64() uint64 {
	v := r.Src.Uint64()
	r.Draws = append(r.Draws, v)
	return v
}

// Replay returns the stored values in order. It must not return a fake value when they
// run out: IntN's rejection sampling keeps redrawing on a constant 0 and never finishes.
type Replay struct {
	Draws []uint64
	pos   int
}

type errExhausted struct{ n int }

func (r *Replay) Uint64() uint64 {
	if r.pos >= len(r.Draws) {
		panic(errExhausted{len(r.Draws)})
	}
	v := r.Draws[r.pos]
	r.pos++
	return v
}

// ReplaySpin re-runs the same game logic on stored values and errors if the draw count differs.
func ReplaySpin(draws []uint64, reelLen []int) (stops []int, err error) {
	rp := &Replay{Draws: draws}
	defer func() {
		if v := recover(); v != nil {
			e, ok := v.(errExhausted)
			if !ok {
				panic(v)
			}
			stops, err = nil, fmt.Errorf("replay exhausted after %d draws", e.n)
		}
	}()
	stops = Spin(rp, reelLen)
	if rp.pos != len(draws) {
		return nil, fmt.Errorf("replay used %d of %d draws", rp.pos, len(draws))
	}
	return stops, nil
}

func TestRecordAndReplay(t *testing.T) {
	rec := &Recorder{Src: CryptoSource{}} // the production source
	got := Spin(rec, reels)

	again, err := ReplaySpin(rec.Draws, reels)
	if err != nil {
		t.Fatal(err)
	}
	t.Logf("live: %v  replay: %v  draws stored: %d", got, again, len(rec.Draws))
	if !slices.Equal(got, again) {
		t.Fatalf("replay mismatch: %v vs %v", got, again)
	}

	_, err = ReplaySpin(rec.Draws[:2], reels)
	t.Logf("too few draws: %v", err)
	_, err = ReplaySpin(append(slices.Clone(rec.Draws), 42), reels)
	t.Logf("too many draws: %v", err)
}

=== RUN   TestRecordAndReplay
    slot_test.go:65: live: [2 11 2 20 15]  replay: [2 11 2 20 15]  draws stored: 5
    slot_test.go:71: too few draws: replay exhausted after 2 draws
    slot_test.go:73: too many draws: replay used 5 of 6 draws
--- PASS: TestRecordAndReplay (0.00s)

Recording the final reel positions isn’t enough; the raw values need to be captured at the Source layer. To remove bias, IntN sometimes discards a value and draws again. The draw count can therefore differ from the reel count, and replaying the exact path requires the raw values.

When the values run out, Replay mustn’t return a placeholder such as 0. IntN’s rejection sampling keeps redrawing if it keeps receiving a value it rejects. A test supplied only 2 values and returned 0 once they ran out. The first two reels have length 32, a power of two, so a mask handles them. The third reel, of length 40, kept redrawing until the test hit its 5-second timeout. The current version panics on exhaustion and ReplaySpin recovers that into an error. Leftover values also produce an error.

Matching draw counts don’t prove that a replay is correct. The reel layout or payout logic can change without affecting how many values are drawn. Each round therefore needs to record its game logic version and paytable alongside the raw values, and the replay must use those versions. The mapping inside IntN is less of a worry: the Go team’s article on v2 states that the rand.Rand API is locked in to this value stream, even if faster algorithms are found later.

The recorded values come from an unpredictable source, so knowing them doesn’t let someone calculate future draws. Whether they can be made public still depends on what game information they reveal. Exposing a shared PRNG’s seed or state would let someone calculate the random values for every round that follows. GLI-19 2.8.2 requires the information for each game played to be maintained and backed up; the raw values can go in that same record.

6. Verifying the RNG as a separate module

Two GLI-19 clauses bear directly on how the code is split. 2.3.2 requires the system to verify, at least once every 24 hours and on demand, that critical control program components match the approved versions, using a hash algorithm with a digest of at least 128 bits. 2.3.3 requires a way for an independent third party to verify each critical component. The RNG is about as critical as components get, since it decides game outcomes. If it is tangled up with game and probability logic, every hash check and every third-party verification has to cover the whole bundle, and it becomes hard to say which part is actually the RNG. Splitting the RNG and probability logic into small, purely functional modules lets a verifier look at just that piece.

With the arrangement above, the RNG package depends on no game and only exposes a Source. Game logic gets its randomness through Source; fix the random sequence and other inputs, and its results can be reproduced. The derived test source and replay tooling stay outside the production binary. At the source-code level, changing one game’s paytable leaves the RNG files alone, and replacing the RNG means replacing one module.

The submitted artefact needs separate attention. If the RNG and games still compile into one Go executable, changing any game changes that file’s hash. To keep the RNG’s hash stable, it needs its own service or artefact.

Dependencies need pinning too. Before submission I ran go mod vendor to copy every dependency into vendor/, so the submitted build and the deployed build used the same dependency source. According to the Go modules reference, when the go version in go.mod is 1.14 or later and a vendor directory is present, builds use it by default, and if go.mod changes without vendor/modules.txt being regenerated, the go command reports an error rather than quietly picking other versions.

Notes

Go’s sync.Mutex is not reentrant, so public methods of an RNG package that call each other can deadlock on themselves. In a server there’s no fatal error; every game sharing the instance simply stops.
Fix it by splitting out internal functions that don’t lock, or by giving each round its own RNG without a lock. Before swapping RNG implementations, exercise every public method from several goroutines, with -race for data races and a deadline for deadlocks.
ChaCha8 and PCG from math/rand/v2 suit cases that need reproducibility. Production can use crypto/rand directly or a CSPRNG package you control; certification depends on the evidence submitted.
Game logic depends only on the Source interface: production injects an unpredictable source, tests inject a ChaCha8 seeded with HMAC-SHA256(secret, round_id), and the derivation code exists only in tests.
Dispute replays store each round’s raw draws and the game logic version, never the seed or PRNG state. A replay source that runs out must report an error, because a placeholder value sends IntN into an endless redraw.
Splitting the RNG and probability logic into their own modules lets them be verified independently later; for a clear hash scope, the submitted artefacts have to be separate too.