# How Qwenputer turns DeltaNet state into RAM: exact reads and writes, and what they do not promise

> Qwenputer fixes the gate and write strength at 1, using one-hot keys for exact RAM access. A numpy example separates this from inference reproducibility.

Source: https://sheng.page/en/posts/qwenputer-deltanet-as-ram/ · Published: 2026-11-08 · Updated: 2026-11-09 · Author: Sheng

Qwenputer is an experiment published by Divyansh Agrawal in October 2026. It compiles a 32-bit RISC-V CPU into the weights of Qwen3.5 4B, using Gated DeltaNet (GDN) layers as memory. The design fixes the gate at 1 to retain stored state and the write strength at 1 to overwrite the old value, then uses one-hot keys to select addresses. The numpy example below shows how these settings allow a value to be written and read back unchanged.

The rule guarantees the semantics of the state matrix and nothing more. Whether the whole model produces bitwise-identical output across runs and serving conditions is a separate question with separate causes: Thinking Machines traced the variation in its experiments to kernels that were not batch-invariant. The last section treats the two separately and uses a small example to show which layers that second question applies to.

## What Qwenputer does

Everything in this section comes from the original post. Nothing here was reproduced.

The base is Qwen3.5 4B. The author points out that Qwen3-Next and Qwen3.5 use GDN in three of every four attention layers, which is what makes the trick available at all. The CPU is not trained. A compiler writes the weights straight from a circuit description: lookups go into attention heads, arithmetic into feed-forward units, RAM into DeltaNet heads. The CPU takes 11 layers placed in front of Qwen's 32, and the hidden size is wider. The post says "Qwen's original weights stay exactly as they are", so the original weights are untouched, but with the extra dimensions and layers the architecture is no longer stock.

The core is RV32IM with the Zba extension and a small bignum unit. According to the post it passes the official riscv-tests, matches QEMU, and ran 18 million instructions free-running without a single error. It runs C and MicroPython, though the author is clear it is not a Linux-class chip. Each instruction shows up in the generated sequence as a few tokens: ADDR names the memory row to write, VAL carries the value, STEP ends the instruction. That stream is the execution trace, and the CPU is running while Qwen generates it.

The model talks to the CPU through the tool-call format it was already trained on, with no post-training. A small driver called RvChat loads firmware into the GDN state and keeps the trace tokens out of Qwen's own layers. On Hacker News one commenter objected to the "no external runtime" framing on the grounds that firmware loading and skipping Qwen's layers are still handled by a host-side runtime.

The post reports a checkpoint of 18.8 GB, roughly twice the base model, which the author says is mostly zero padding, and it runs on a single 24 GB GPU. The batched executor used for benchmarks manages about 3,200 CPU tokens per second on an L4. For correctness, about 230,000 output positions were replayed against a reference machine with zero mismatches, and ordinary text stays close to base Qwen at a KL of 6e-4 against a stated noise floor of 5e-4.

## Writing and reading one address

The Gated DeltaNet paper (Yang, Kautz and Hatamizadeh, 2024) gives the state update as

```text
S_t = S_{t-1} (α_t (I − β_t k_t k_tᵀ)) + β_t v_t k_tᵀ
o_t = S_t q_t
```

S is the state matrix, shape d_v by d_k. α_t is the gate, in (0, 1) in the paper, and sets how much of the old state survives. β_t is the write strength, also (0, 1) in the paper, and sets how much of the new value replaces the old. k, v and q are key, value and query, and a read is just the query multiplied into the state. Qwenputer's RAM conditions map onto this directly: α_t = 1, β_t = 1, k_t = e_i. Substitute them in and the bracket becomes I − e_i e_iᵀ, which zeroes column i of S and leaves every other column alone; the second term then fills column i with v. Reading with q = e_i pulls column i back out. The post talks about rows rather than columns, which is only a matter of how the matrix is laid out.

The example below follows the equation as written. It writes four addresses, overwrites one of them, then reads each back.

```python
import numpy as np

# Gated delta rule (Yang, Kautz, Hatamizadeh, 2024, eq. 10):
#   S_t = S_{t-1} (alpha_t (I - beta_t k_t k_t^T)) + beta_t v_t k_t^T
#   o_t = S_t q_t
# S has shape (d_v, d_k). A one-hot key k = e_i touches only column i.
D_K, D_V = 4, 3
I = np.eye(D_K, dtype=np.float32)


def step(S, k, v, alpha, beta):
    k = k.reshape(-1, 1)
    v = v.reshape(-1, 1)
    return S @ (alpha * (I - beta * k @ k.T)) + beta * v @ k.T


def write(S, addr, v):
    return step(S, I[addr], v, alpha=np.float32(1), beta=np.float32(1))


def read(S, addr):
    return S @ I[addr]


rng = np.random.default_rng(7)
S = np.zeros((D_V, D_K), dtype=np.float32)
vals = rng.integers(-100, 100, size=(D_K, D_V)).astype(np.float32)
for addr in range(D_K):
    S = write(S, addr, vals[addr])
S = write(S, 2, np.array([7, 7, 7], dtype=np.float32))  # overwrite addr 2
vals[2] = [7, 7, 7]

worst = 0.0
for addr in range(D_K):
    got = read(S, addr)
    err = float(np.max(np.abs(got - vals[addr])))
    worst = max(worst, err)
    print(f"addr {addr}: wrote {vals[addr]}  read {got}  max|err| {err}")
print("worst error over all addresses:", worst)
```

```text
addr 0: wrote [88. 25. 36.]  read [88. 25. 36.]  max|err| 0.0
addr 1: wrote [79. 15. 55.]  read [79. 15. 55.]  max|err| 0.0
addr 2: wrote [7. 7. 7.]  read [7. 7. 7.]  max|err| 0.0
addr 3: wrote [-40. -43.  74.]  read [-40. -43.  74.]  max|err| 0.0
worst error over all addresses: 0.0
```

For the values and operations in this example, the error is zero: each step multiplies by 1 or 0 and adds into a slot holding 0, so float32 has nothing to round. This illustrates the exact read/write rule at the level of the state matrix; a real kernel's numerical path and the precision settings inside the model are out of scope, and no CPU instruction runs here. The 230,000 positions with zero mismatches are the author's result, not reproduced here.

## Reading back with relaxed conditions

Relaxing each setting in turn (a gate of 0.9, a write strength of 0.5, and one key that is not one-hot) makes the same program read back different values.

```python
import numpy as np

# Same gated delta rule as exact_rw.py, but with the RAM conditions relaxed.
D_K, D_V = 4, 3
I = np.eye(D_K, dtype=np.float32)


def step(S, k, v, alpha, beta):
    k = k.reshape(-1, 1)
    v = v.reshape(-1, 1)
    return S @ (alpha * (I - beta * k @ k.T)) + beta * v @ k.T


def fill(alpha, beta, keys):
    S = np.zeros((D_V, D_K), dtype=np.float32)
    vals = np.array([[10, 20, 30], [40, 50, 60], [70, 80, 90], [1, 2, 3]],
                    dtype=np.float32)
    for addr in range(D_K):
        S = step(S, keys[addr], vals[addr], alpha, beta)
    return S, vals


def report(label, S, vals, queries):
    print(label)
    for addr in range(D_K):
        got = S @ queries[addr]
        print(f"  addr {addr}: wrote {vals[addr]}  read {np.round(got, 3)}")


one_hot = [I[i] for i in range(D_K)]

# (a) gate < 1: every later write shrinks what is already stored.
S, vals = fill(alpha=np.float32(0.9), beta=np.float32(1), keys=one_hot)
report("gate 0.9, write strength 1, one-hot keys", S, vals, one_hot)

# (b) write strength < 1: the new value is blended with the old one.
S, vals = fill(alpha=np.float32(1), beta=np.float32(0.5), keys=one_hot)
report("gate 1, write strength 0.5, one-hot keys", S, vals, one_hot)

# (c) a key that overlaps two addresses writes into both of them.
mixed = list(one_hot)
mixed[3] = np.array([0, 0, 1, 1], dtype=np.float32) / np.sqrt(np.float32(2))
S, vals = fill(alpha=np.float32(1), beta=np.float32(1), keys=mixed)
report("gate 1, write strength 1, key 3 = (e2 + e3) / sqrt(2)", S, vals, mixed)
```

```text
gate 0.9, write strength 1, one-hot keys
  addr 0: wrote [10. 20. 30.]  read [ 7.29 14.58 21.87]
  addr 1: wrote [40. 50. 60.]  read [32.4 40.5 48.6]
  addr 2: wrote [70. 80. 90.]  read [63. 72. 81.]
  addr 3: wrote [1. 2. 3.]  read [1. 2. 3.]
gate 1, write strength 0.5, one-hot keys
  addr 0: wrote [10. 20. 30.]  read [ 5. 10. 15.]
  addr 1: wrote [40. 50. 60.]  read [20. 25. 30.]
  addr 2: wrote [70. 80. 90.]  read [35. 40. 45.]
  addr 3: wrote [1. 2. 3.]  read [0.5 1.  1.5]
gate 1, write strength 1, key 3 = (e2 + e3) / sqrt(2)
  addr 0: wrote [10. 20. 30.]  read [10. 20. 30.]
  addr 1: wrote [40. 50. 60.]  read [40. 50. 60.]
  addr 2: wrote [70. 80. 90.]  read [35.707 41.414 47.121]
  addr 3: wrote [1. 2. 3.]  read [1. 2. 3.]
```

With a gate of 0.9, every later write multiplies everything already stored by 0.9 again, so address 0, written first, has decayed to 0.729 of its value. That decay lets GDN discard old state, but does not meet the RAM behaviour required here. With a write strength of 0.5, what lands is half the new value and half the old (zero here), so a read returns half. The third run shows interference between addresses. Address 3's key straddles e2 and e3. Its own read-back looks fine, but the write also wiped part of address 2 on the way through, which now reads 35.7 instead of 70. One-hot keys matter because they keep the projection in each write confined to a single address.

In a real model all three quantities are computed from the input by learnt parameters, which is why an ordinary GDN layer is not RAM. Qwenputer pins them to 1 and to unit vectors through compiled weights. Compared with the line from Schlag, Irie and Schmidhuber's 2021 result, where delta-rule linear attention acts as associative memory, the difference is that α, β and the key are pinned rather than learnt.

## Exact reads and reproducible inference are different layers

Wikipedia's definition of a deterministic algorithm is one that, given a particular input, always produces the same output and passes through the same sequence of states. Fix the parameters, the initial state and the input, and the update with decay is every bit as deterministic as the one without; α = 0.9 and α = 1 are equal on that score, and it makes no difference that the parameters were learnt. The drift in the previous section is semantic drift, and it drifts the same way every time you run it.

An LLM can produce different answers to the same prompt at temperature 0. Thinking Machines' 2025 post examines why this happens in its inference setup. It argues that floating-point non-associativity and GPU concurrency alone do not explain the variation: the forward path it examines requires no atomic adds, and the same kernel on the same batch is bitwise reproducible. It traces the user-visible differences to changing batch sizes, which cause RMSNorm, matmul and attention kernels to use different reduction orders. Its experiment ran a specific Qwen3-235B checkpoint at temperature 0 on one prompt 1,000 times and obtained 80 distinct completions; the first 102 tokens matched, with divergence at token 103. With batch-invariant kernels, all 1,000 completions matched. These results apply to the tested conditions and cannot simply be extended to every inference service.

Float32 addition shows how reduction order can change a result. The example below shows how summation order alone can affect inference reproducibility.

```python
import numpy as np

# The same float32 numbers, summed in a different order, do not always
# give the same float32 result. Fixed input, no randomness.
xs = np.array([1e8, 1.0, -1e8, 1.0], dtype=np.float32)

left_to_right = np.float32(0)
for x in xs:
    left_to_right = np.float32(left_to_right + x)

pairwise = np.float32(np.float32(xs[0] + xs[2]) + np.float32(xs[1] + xs[3]))

print("left to right:", left_to_right)
print("pairwise     :", pairwise)
print("equal        :", left_to_right == pairwise)

# A reduction over a bigger array, with the same values in two orders.
rng = np.random.default_rng(1)
big = rng.standard_normal(100_000).astype(np.float32) * np.float32(1000)
a = np.float32(0)
for x in big:
    a = np.float32(a + x)
b = np.float32(0)
for x in big[::-1]:
    b = np.float32(b + x)
print("forward sum  :", a)
print("reverse sum  :", b)
print("difference   :", np.float32(a - b))
```

```text
left to right: 1.0
pairwise     : 2.0
equal        : False
forward sum  : -459056.06
reverse sum  : -459057.84
difference   : 1.78125
```

Four numbers whose sum should be 2 give 1 when added left to right, because 1e8 + 1 is still 1e8 in float32. A hundred thousand numbers added forwards and backwards differ by 1.78. Both orders are deterministic on their own; the order that actually runs depends on the kernel and the batch size, and a service user may be unable to control it.

Order changes results, and which layers it affects depends on what each layer computes. The example below puts the RAM step from earlier next to an ordinary float32 dense layer and applies the same two perturbations to both: a shuffled reduction order, and a batch size other than 1. The RAM step is compared bit for bit with writing v straight into column 17; the comparison is on float32 bit patterns, so +0.0 and −0.0 count as different. At the end the same write is evaluated once more in the algebraically equal form S + (v − S e) eᵀ.

```python
import numpy as np

# Two float32 layers under the same two perturbations: a shuffled reduction
# order, and a batch size other than 1. The RAM step (alpha = 1, beta = 1,
# one-hot key) is compared bit for bit with direct column assignment.
rng = np.random.default_rng(2026)
f32 = np.float32
D_K, D_V = 64, 48


def bit_equal(a, b):
    """Same bit pattern in every float32 (unlike ==, tells +0.0 from -0.0)."""
    return np.array_equal(a.view(np.uint32), b.view(np.uint32))


def ordered_matmul(A, B, order):
    """A @ B with the reduction over the shared axis done in the given order."""
    out = np.zeros((A.shape[0], B.shape[1]), dtype=f32)
    for j in order:
        out = f32(out + np.outer(A[:, j], B[j]))
    return out


# (a) RAM step: S (I - e e^T) + v e^T must equal S with column addr replaced by v.
S = rng.standard_normal((D_V, D_K)).astype(f32) * f32(1000)
v = rng.standard_normal(D_V).astype(f32)
addr = 17
e = np.eye(D_K, dtype=f32)[addr]
proj = np.eye(D_K, dtype=f32) - np.outer(e, e)
expected = S.copy()
expected[:, addr] = v

# (b) dense layer: x @ W with learnt-looking weights, no exact zeros to help it.
W = rng.standard_normal((D_K, D_V)).astype(f32)
x = rng.standard_normal((1, D_K)).astype(f32) * f32(1000)
dense_1 = x @ W

ram_bad = dense_bad = 0
for _ in range(200):
    order = rng.permutation(D_K)
    ram_bad += not bit_equal(ordered_matmul(S, proj, order) + np.outer(v, e), expected)
    dense_bad += not bit_equal(ordered_matmul(x, W, order), dense_1)
print(f"200 shuffled orders   RAM step mismatches: {ram_bad}   dense layer mismatches: {dense_bad}")

for b in (1, 2, 7, 64, 513):
    ram_b = (np.broadcast_to(S, (b, D_V, D_K)).copy() @ proj + np.outer(v, e))[0]
    dense_b = (np.broadcast_to(x, (b, D_K)).copy() @ W)[0]
    print(f"batch {b:>3}   RAM step equals expected: {bit_equal(ram_b, expected)}   "
          f"dense row 0 equals batch 1: {bit_equal(dense_b, dense_1[0])}")

# (c) The same write evaluated as S + (v - S e) e^T, which is equal on paper.
delta_form = S + np.outer(v - S @ e, e)
print("RAM step in delta form   equals expected:", bit_equal(delta_form, expected),
      "  mismatched elements:", int((delta_form.view(np.uint32) != expected.view(np.uint32)).sum()))

# (d) Two candidate tokens whose logits are nearly tied: the order alone decides.
X = rng.standard_normal((20_000, D_K)).astype(f32) * f32(1000)
Wt = np.repeat(rng.standard_normal((D_K, 1)).astype(f32), 2, axis=1)
Wt[0, 1] = np.nextafter(Wt[0, 1], f32(np.inf))  # one weight differs by one ulp
fwd = ordered_matmul(X, Wt, range(D_K))
rev = ordered_matmul(X, Wt, reversed(range(D_K)))
print("near-tie logits, forward vs reverse order: argmax differs in",
      int((fwd.argmax(1) != rev.argmax(1)).sum()), "of 20000 rows")
```

```text
200 shuffled orders   RAM step mismatches: 0   dense layer mismatches: 200
batch   1   RAM step equals expected: True   dense row 0 equals batch 1: True
batch   2   RAM step equals expected: True   dense row 0 equals batch 1: False
batch   7   RAM step equals expected: True   dense row 0 equals batch 1: False
batch  64   RAM step equals expected: True   dense row 0 equals batch 1: False
batch 513   RAM step equals expected: True   dense row 0 equals batch 1: False
RAM step in delta form   equals expected: False   mismatched elements: 48
near-tie logits, forward vs reverse order: argmax differs in 3015 of 20000 rows
```

The RAM step matches direct column assignment bit for bit under all 200 orders and all five batch sizes, and the reason is in the matrix. With α = 1, β = 1 and an exact unit-vector key, column i of I − e_i e_iᵀ is all zeros and every other column holds a single 1 with exact zeros elsewhere, so every output element has at most one non-zero product and the rest of the sum is zeros; summation order cannot change the result. That holds for the values in this example; a −0.0 in the state would come out as +0.0 after adding 0.0, and the bit comparison would flag it. The dense layer has no such structure. Each output element sums 64 non-zero products, every one of the 200 orders produces a different bit pattern, and in this environment (OpenBLAS on CPU) row 0 at batch 2, 7, 64 and 513 differs from the batch-1 result. When two candidate tokens differ by a single-ulp weight, forward and reverse order pick a different token in 3,015 of 20,000 rows.

Whether this step is affected by either perturbation depends on how it is computed; the equation alone does not settle it. The same write as S + (v − S e) eᵀ is equal on paper, but float32 computes v − S e first and adds it back, and with magnitudes this far apart the subtraction rounds: all 48 elements of column 17 differ from the expected result. An exact key and gate are therefore necessary but not sufficient; which form the kernel uses for this step matters as well, and the post does not say. The matrices here have dimensions 64 and 48, toy sizes, with none of Qwenputer's kernels or precision settings, but the split still applies: evaluated in the projection form with an exact key and gate, the RAM step is unaffected by either perturbation, while the lookups compiled into attention and the arithmetic compiled into feed-forward have no such structure to protect them, so order and batch size can change their results and the real kernels need checking. A wrongly chosen token there may change the trace that follows.

For Qwenputer, the RAM rule gives the intended read/write behaviour under α = 1, β = 1 and one-hot keys, and when evaluated in the projection form that step is insensitive to reduction order and batch size. Two parts of the implementation need checking. On the RAM side, the compiled key must be an exact unit vector, the gate must multiply by exactly 1, and the kernel must compute the step in a form that does not subtract the old value and add it back, since that subtraction can round; which form Qwenputer uses is not stated. On the attention and feed-forward side, the layers that hold the lookups and arithmetic must give the same result at every batch size, and the post does not say whether those kernels are batch-invariant. The post reports 230,000 positions with no mismatches and 18 million free-running instructions without error. Those results support the tested conditions; the RAM equation alone cannot establish bitwise identity on another GPU or at another batch size.

[Game RNG that is unpredictable in production and replayable in tests](/en/posts/rng-unpredictable-in-prod-replayable-in-tests/) also discusses the conditions needed for replay.

## Limitations listed in the post

The author lists these limitations: the checkpoint is 18.8 GB, mostly zero padding; it needs the RvChat driver to boot firmware and does not yet work with a plain `generate()` call; each CPU instruction costs a few decoded tokens, so large calculations take seconds rather than milliseconds; it is not a Linux-class CPU; the filesystem is experimental. Future work named in the post is a `python()` tool backed by the in-weight MicroPython, and faster CPU execution.

## Notes

- The Gated DeltaNet update is S_t = S_{t-1}(α_t(I − β_t k_t k_tᵀ)) + β_t v_t k_tᵀ, with read-out o_t = S_t q_t.
- With α = 1, β = 1 and a unit-vector key, a write clears and refills one column and leaves the rest untouched; the float32 example has zero error.
- A gate below 1 decays older addresses on every write, a write strength below 1 blends new and old, and the mixed key in the example affects another address.
- According to the author, Qwenputer leaves the original weights alone but adds dimensions and 11 leading layers; the checkpoint is 18.8 GB and it depends on the RvChat driver.
- Learnt parameters and decaying state do not make a computation non-deterministic; Thinking Machines attributes the output variation in its experiments to kernels that lack batch invariance, and float32 summation order is one mechanism among others.
- Evaluated in the projection form with an exact key and gate, every output element of the RAM step has at most one non-zero product, so it is insensitive to reduction order and batch size; the algebraically equal delta form does not reproduce the intended write in this example. The dense computations compiled into attention and feed-forward can be affected and the real kernels still need checking, and the RAM rule holding does not imply bitwise-identical results across serving conditions.

## Further reading

- Divyansh Agrawal, [Qwenputer: A functioning, deterministic RISC-V CPU inside Qwen3.5](https://divagr.com/blog/qwenputer): source for the architecture, layer counts, gate and write-strength settings, one-hot addressing, checkpoint size, GPU requirement, limitations and RvChat; every case fact here comes from it.
- Hacker News, [Qwenputer discussion](https://news.ycombinator.com/item?id=50282838): where the objection to the "no external runtime" framing was raised.
- Yang, Kautz and Hatamizadeh, [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://arxiv.org/abs/2412.06464): source for the gated delta rule update and the definitions of α and β.
- Thinking Machines, [Defeating Nondeterminism in LLM Inference](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/): source for the batch-invariance argument and the Qwen3-235B numbers; primary research.
- Wikipedia, [Deterministic algorithm](https://en.wikipedia.org/wiki/Deterministic_algorithm): the definition used above.
- Schlag, Irie and Schmidhuber, [Linear Transformers Are Secretly Fast Weight Programmers](https://arxiv.org/abs/2102.11174) (ICML 2021): the origin of this family of delta-rule updates.

> Ideas and technical judgement by Sheng; drafted with Claude · examples run on Linux with Python 3.13.18 and numpy 2.5.3 · revision planned with Fable, reviewed with Codex, tone tuned with Grok.