The 900-line PR Anthropic turned down: what a performance gain costs to maintain

In “How we made claude.ai 3x faster in two weeks”, published on 23 September 2026, Anthropic’s engineers mention a pull request of about 900 lines that took roughly 2 ms off every message sent. It also added a build plugin someone would have to look after. The post quotes a one-line rejection: “going to gavel that 2ms per send is not worth the complexity of maintaining this build plugin.” The post gives no measurement details for that 2 ms, and doesn’t say whether anyone challenged the figure. The quoted objection is about upkeep.

Knuth’s warning about premature optimisation also asks us not to pass up opportunities in the critical 3%. Even a measured saving of 2 ms needs to be weighed against upkeep after merge, a cost that is easy to overlook when a change is cheap to make.

The reply comes from the part of the post about steering: the team ran more than 150 narrow Slack threads at once, each focused on one benchmark or one user journey. Engineers decided what to prioritise, adjusting or closing threads when their work overlapped or the gains tailed off. When a model can hand you several variants of a change in a short time, optimisations that never made the schedule finally get measured. The post doesn’t say who wrote the 900-line PR. Either way, compatibility and every later edit stay with the team, and so does the call on whether to take it.

What 3.1 s to 0.55 s measures

One figure in the same post is time to a typeable page on a fresh load of claude.ai, at the 75th percentile, down from 3.1 seconds to 0.55. That is the moment the page will accept input. No message has been sent yet, so the model’s response time is not in it. The other journeys moved by different amounts: starting a Claude Code session went from 0.8 s to 0.3 s, loading a Claude Cowork cloud session from 2.6 s to 0.73 s.

Google picked the 75th percentile for Core Web Vitals because it wanted most visits to meet the target without letting a handful of outliers drive the result. A better p75 tells you three loads in four now finish within that time. It says nothing about the slowest quarter, and the post gives no p95 or p99, so you can’t tell what happened in the tail.

Where the clock starts and stops

Before changing anything, shrink “the site is slow” down to a single interaction. The page appearing and the input box accepting keystrokes are different moments. After you press send, the wait can be before the request leaves, on the server, or after the response comes back. Anthropic added instrumentation until their baselines were directly comparable: each one started with a user interaction, ended once the result was rendered, and separated client work from server work.

Say the backend gets 100 ms faster while the frontend still makes the user wait 500 ms. Whether the user notices the improvement depends on how much the whole interaction speeds up. Go the other way and make the input box usable earlier, and you have to check that whatever people type in that window actually gets sent. Otherwise the number improves and the experience gets worse.

Numbers need their test conditions

Use the same data volume and concurrency before and after a change, record the device and keep cold-cache results separate from warm-cache results. Repeatable conditions make it easier to distinguish an effect of the code from a change in the test environment. One Anthropic chart specifies “4-vCPU container, headless Chrome, no CPU throttling; 2–3 runs per value”. Two or three runs may reveal a large difference, but they give little basis for estimating the spread. For pure-JS hot paths the team counted instructions under Valgrind with node --predictable and compared them against a checked-in baseline, “one run, no statistics needed”, meaning that counts were repeatable under their fixed test conditions. They then checked those counts against wall-clock time.

After shipping, look at real traffic and confirm the improvement landed on the interaction you meant to fix. Their field comparison was 13 August against 27 August, both Thursdays, broken down by platform and product. After each deploy, Claude read field data by build and platform. The two days need not contain the same users, and a change in the mix of devices or regions can affect p75 independently of the code. More than 3,000 changes landed between those dates, so the two numbers alone can’t tell you which change did what. If only the lab got faster, go looking for the difference between lab and production rather than relaxing the acceptance criteria because the code is already written.

Whether you can roll back depends on how the change ships. They classified nearly two hundred feature flags as kill switches or ramps, sent high-risk changes to employees first, then to 1% of users, then to everyone, and report no customer-facing incident or rollback across those 3,000-plus changes.

The behaviour you need belongs in the task

Caching is the usual example. Query time can drop at once, but you still have to check how long an update takes to become visible, whether a user who no longer has the permission can still read the old result, and what happens to requests when the cache is empty or invalidated, or the backend is down.

Write those conditions into the task before you hand it to an AI. Without them, a faster benchmark may still leave the team checking whether the old behaviour survives, including permission checks and error responses. Put one query path into the task, along with the numbers from before the change and tests for the permission checks and error responses, then compare latency and resource use. With one path changed at a time you can tell where the gain came from, and if it disappoints you revert the lot. Anthropic’s narrow threads were scoped the same way.

Upkeep before merge, measurement after

Allow for maintenance before you merge. Deleting a query you didn’t need removes work at run time and a layer someone has to read later. Adding a cache layer that needs a background job to keep it in sync means more places to look the next time something breaks. Two milliseconds can matter a great deal inside a hot loop and very little in something that runs once a session. Amdahl’s law puts a limit on the benefit: an improvement to one part is limited by that part’s share of the original running time. The post doesn’t give the total time of a send, so from outside there is no way to say what share 2 ms represents.

Anthropic turned two of their instruction-count gains into CI ratchets. From then on, any PR that raised the instruction count on either path failed CI, and a daily job lowered each ceiling whenever the count went down. The counts were repeatable under their test conditions. With noisy wall-clock measurements, a gate that allows no tolerance and only lowers its threshold can produce false failures: the ceiling keeps dropping to the lowest reading of each day, the lowest reading sits at the bottom of the noise, and most PRs that change nothing at all end up above it. A short simulation makes the point: the code never changes, only the measurement noise does. It assumes independent noise, a zero-tolerance gate, and a daily job that lowers the ceiling to the day’s lowest reading across all PRs:

import random

def false_fail_rate(noise_sd: float, days: int = 60, prs_per_day: int = 10) -> float:
    rng = random.Random(20261104)
    true_cost = 1000.0                      # the code never changes
    ceiling = true_cost + rng.gauss(0, noise_sd)
    fails = 0
    for _ in range(days):
        today = [true_cost + rng.gauss(0, noise_sd) for _ in range(prs_per_day)]
        fails += sum(x > ceiling for x in today)
        ceiling = min(ceiling, min(today))  # daily job: lower the ceiling on any new low
    return fails / (days * prs_per_day)

for sd in (0.0, 5.0, 20.0):
    print(f"noise_sd={sd:>4}: {false_fail_rate(sd):.1%} of PRs flagged as slower")

noise_sd= 0.0: 0.0% of PRs flagged as slower
noise_sd= 5.0: 98.7% of PRs flagged as slower
noise_sd=20.0: 98.7% of PRs flagged as slower

In this model any noise above zero pushes the false-failure rate close to 100%, and the size of the noise makes no difference: the ceiling tracks the minimum, so only the ordering of readings matters, and scaling the noise leaves the ordering unchanged. This result depends on the model’s assumptions; a real pipeline’s false-failure rate depends on its sampling and threshold rules. If wall-clock time has to be a gate, estimate its variance first, leave a tolerance and stop the baseline chasing every new low, as in “Setting tolerances for Monte Carlo tests: standard error, false rejections and bit-exact replay”, and accept that small regressions will slip through.

When you hand performance work to an AI, start with a journey users actually complain about, with a baseline and behaviour tests in place. If the change stays, keep the measurement as a regression check, using instruction counts where they have been shown to track elapsed time, so a later slowdown on the same interaction leaves a figure to check against.

Notes

Anthropic declined a 900-line PR that saved 2 ms per send because the build plugin it needed wasn’t worth maintaining.
The 3.1 s to 0.55 s figure is p75 time to a typeable page on a fresh load. It excludes model response time and says nothing about p95 or p99.
Comparing p75 on two dates may involve different users; traffic mix can affect the number independently of the code, so break it down by platform and product.
Whether 2 ms is worth it depends on its share of the whole, which the post doesn’t give.
CI ratchets that allow no tolerance and keep chasing the minimum can mistake measurement noise for a regression. The simulation shows that particular rule.