Treat Unset as Zero: Hard Caps, Approval Thresholds and Which Way a Kill Switch Fails

In early October Simon Willison made the case for hard budget caps being on by default for anything billed by usage. Once you hit the cap the service returns errors, and if you want to spend beyond it you have to opt in. Agents can now sign up for paid services on a whim, and he reckons most people would take an error over a surprise bill for thousands of dollars. The post picked up 645 points and 312 comments on Hacker News.

His examples are cloud bills, but systems that move money have the same problem with less room for error. Nobody looks at a limit, a threshold or a switch while things are working. They matter when the value isn’t there: it was never configured, the column is NULL, or whatever stores it has gone down. The default determines what happens next. I made that decision in a handful of places while building a game provider backend prototype in September. The snippets below are cut-down examples run on Go 1.27.1, with their output.

Amounts outside the rules are rejected

Amounts in the prototype are points, stored as numeric(38,15): at most fifteen digits after the decimal point. Before review I found and fixed a few problems in the code, including the lack of an upper limit on amounts. Anything positive was accepted.

I replaced that with two rules. Sixteen or more decimal places causes an error, as does any amount above the hard cap. Rounding is tempting because the request goes through, but the ledger then holds a number the caller never sent, leaving a tiny, unexplained difference at reconciliation. Clamping does more damage. A credit of a million and a bit is booked as exactly a million, and since neither side got an error, nobody finds out until the books are compared.

const maxScale = 15

var maxAmount = mustRat("1000000")

func parseAmount(s string) (*big.Rat, error) {
	if i := strings.IndexByte(s, '.'); i >= 0 && len(s)-i-1 > maxScale {
		return nil, fmt.Errorf("scale %d > %d", len(s)-i-1, maxScale)
	}
	r, ok := new(big.Rat).SetString(s)
	if !ok {
		return nil, errors.New("not a decimal")
	}
	if r.Sign() <= 0 {
		return nil, errors.New("must be positive")
	}
	if r.Cmp(maxAmount) > 0 {
		return nil, errors.New("over hard cap")
	}
	return r, nil
}

100.5                      ok
0.000000000000001          ok
0.0000000000000001         rejected: scale 16 > 15
1000000.000000000000001    rejected: over hard cap
-3                         rejected: must be positive

A million is just a number for the example. Where the cap sits is a business decision; the engineering decision is that crossing it produces an error the caller can see. Willison argues for the same behaviour: return an error rather than silently make a decision for the other party.

An unset approval threshold means zero

Manual adjustments and manual settlements in the back office need two people at or above a threshold: one requests, the other approves. Below it, one person can go ahead, and the action is still written to the audit log. Each operator has its own threshold, in points.

The mistake to watch for is what the code does before anyone has set the threshold. Go code tends to use a pointer for “may be missing”, and the natural thing to write is “if it’s nil, skip the check”. That means a freshly onboarded operator lets anyone with access make any adjustment alone until somebody remembers to configure it:

// needsApprovalNaive treats "not configured" as "no limit".
func needsApprovalNaive(c Casino, amt *big.Rat) bool {
	return c.ApprovalThreshold != nil && amt.Cmp(c.ApprovalThreshold) >= 0
}

// needsApproval treats "not configured" as zero: everything needs a second person.
func needsApproval(c Casino, amt *big.Rat) bool {
	t := c.ApprovalThreshold
	if t == nil {
		t = new(big.Rat)
	}
	return amt.Cmp(t) >= 0
}

For a 50-point adjustment:

configured=1000   naive=false default-zero=false
never configured  naive=false default-zero=true

So an unset threshold counts as zero and every amount needs a second person. On top of that, changing the threshold always needs a second person, whatever it is being changed to, because otherwise one person could raise it and then make a large adjustment on their own.

The price is a stream of approval requests in the first weeks, and some people will grumble. That is a problem everyone can see, and it gets the threshold set. Defaulting to no limit creates no extra work, but if one person later moves a large amount alone, the audit log is how you find out.

Retry limits depend on the operation

Bets, wins and refunds going to the operator pass through an outbox and are retried with backoff on failure. Without a limit, one operator whose system is down keeps the queue growing indefinitely. With a single limit for every operation, you give up on money that must be delivered.

The prototype treats them differently. A bet is dropped after eight attempts, which is fine because the hot path already issues a compensating refund, so the player is never out of pocket. Wins and refunds are money owed to the player: backoff tops out at 300 seconds, retries continue for 24 hours, and only then does a person pick it up. Every retry loop needs a limit, chosen according to who would be hurt if that payment were abandoned.

If the kill switch can’t be read

The kill switch is how you stop one operator or one game when something goes wrong. It is also a setting, and it can be unreadable: the database is unreachable, a cache node is restarting, the network has a bad moment.

The first design fell back to the database when Valkey failed. I later simplified this to an in-process cache with a five-second TTL, reloaded from the database when the entry expires. Outages shorter than five seconds never reach the code that has to decide how to handle a read failure. Beyond that, the decision depends on the path. Money writes fail closed: an unreadable switch is treated as tripped, and no new debits or credits are accepted. Read-only paths, balance lookups and reports, fail open and keep serving.

func (k *KillSwitch) Allow(p Path) (bool, string) {
	if !k.fetched.IsZero() && k.now().Sub(k.fetched) < k.ttl {
		return !k.val, "cache"
	}
	killed, err := k.load()
	if err == nil {
		k.val, k.fetched = killed, k.now()
		return !killed, "source"
	}
	if p == MoneyWrite {
		return false, "source down, fail-closed"
	}
	return true, "source down, fail-open"
}

Using a fake clock, with the source going away after t=0:

t=0s source up         money-write allow=true  (source)
t=0s source up         read-only   allow=true  (cache)
t=3s source down       money-write allow=true  (cache)
t=3s source down       read-only   allow=true  (cache)
t=6s source down       money-write allow=false (source down, fail-closed)
t=6s source down       read-only   allow=true  (source down, fail-open)

Wikipedia’s fail-safe article separates fail-safe from fail-secure, also called fail-closed, and points out that they can pull in opposite directions; electronic door locks in a fire are its example, since they have to release so people can get out. The kill switch splits the same way. Stopping is the safe outcome for a money write. Continuing is the safe outcome for a read, because support still needs to look up balances and operations still needs reports, and losing those in the middle of an incident only makes it harder to handle.

One of the prototype’s end-to-end tests covers this: it runs the kill switch with Valkey disconnected and expects fail-closed. Without that test, the next person to work on this code could easily switch everything to fail-open to improve availability.

Defaults for missing or unreadable settings

When a value is missing or can’t be read, the default should keep losses small and make the problem visible. An amount out of range is an error, an unconfigured threshold means two people for everything, retries stop eventually, and an unreadable switch halts money writes.

These defaults do make the system more awkward at times, mostly through errors and extra approvals. Willison accepts that too. People can lift a cap if they want, but it should take a deliberate step, and anyone who leaves the setting alone keeps the default protection.

Notes

Reject amounts beyond the allowed precision or the hard cap. Don’t round and don’t clamp, so both sides record the same number.
An unset approval threshold counts as zero, so everything needs two people, and changing the threshold always needs two people.
In Go, the common mistake with a nil pointer meaning “unset” is skipping the check, which quietly makes “no limit” the default.
Retry limits are set per operation: a bet with a compensating refund can be abandoned early, money owed to the player is retried for longer.
When the kill switch can’t be read, money writes fail closed and read-only paths fail open, with an end-to-end test checking that behaviour.