Looping transformers on the most popular open-source models: nothing got smarter
This experiment is a Digisensus test of training-free looped transformers: running a few of a model's middle layers two or three times per token, with no retraining. We tried it on two of the most popular open-source models, Qwen3.8-27B and Qwen3.8-Flash-Next, a 125B mixture-of-experts model, both in vLLM. Here is what we built, how we checked it, what MMLU-Pro and GPQA-Diamond said, and one scoring trap that nearly fooled us.
Digisensus · Vilnius
Summary
Neither model got better. We looped layers in Qwen3.8-27B and in Qwen3.8-Flash-Next, a 125B mixture-of-experts model that uses only about 6B of its parameters per token. Across 34 Flash-Next configs and dozens of 27B configs, no setting raised MMLU-Pro accuracy beyond noise. That is the whole result. The rest is how we got there and what we tripped over on the way.
The idea comes from Training-Free Looped Transformers (Chen et al., 2026). Take a finished model and run a few of its middle layers more than once. No training, no new weights, just extra passes. The paper reports +2.6 pp on MMLU-Pro for Qwen3-4B, shrinking to at most +1.1 pp for Qwen3-30B-A3B. (“pp” is percentage points.)
That shrinking curve made us curious. Would anything be left at 27B? And would Flash-Next, with only about 6B active parameters, behave like the small models where the gains are biggest? We tried both in vLLM in October 2026. Nothing got smarter, but the trip was worth writing up.
How the looping works
Pick a run of neighbouring layers, call the group g, and run it K times instead of once, each time taking only a fraction of a step:
x ← x + (1 / K) · (g(x) − x)
Each pass moves the model’s internal state only 1/K of the way toward what the layers want. Those small steps are called damping. Plain repetition tends to throw the later layers off; damping keeps things on track. The paper found damping essential, and so did we.
We also had to protect the model’s memory of the text so far. The looped layers may not grow or rewrite it, so we used the paper’s cache=first setting: only the first pass leaves a lasting entry, and the extra passes leave nothing behind.
Both models add a twist the paper never met. Most of their layers are Gated DeltaNet (GDN) layers, which keep a running memory and update it in place. Run such a layer three times and its memory moves on three times. So we save that memory before the loop and put it back before each extra pass, so every pass starts from the same place.
Round 1: Qwen3.8-27B in vLLM
On the 27B, no choice of layers, window width or loop count improved accuracy. Several made it worse.
Setup. We changed vLLM 0.28.0’s qwen3_next.py to loop a chosen window of layers, with the settings in a small file we could edit while the server ran. 48 of the model’s 64 layers are GDN layers, so saving and restoring their running memory did most of the work.
Before trusting it, we checked it against the untouched model. With the extra passes running but their result thrown away (beta=1), the answers have to match exactly. They did, token for token: 735 of 735. A relief.
Benchmark. MMLU-Pro is a broad multiple-choice knowledge test. We gave no examples and kept thinking off, and scored the answer letter the model liked best at the start of its reply. Each config is compared with the baseline on the same questions using an exact McNemar test; its p-value says how easily chance alone could explain the difference. Running the same config twice on all 12,032 questions already flips 233 answers (−0.06 pp), so anything that small is noise.
| Config | Δ accuracy | p | ΔP(gold) |
|---|---|---|---|
| Baseline, run again | −0.06 pp | 0.69 | −0.03 pp |
| Layers 32–35, K=2 | −0.15 pp | 0.61 | +0.32 pp |
| Layers 32–35, K=3 | −0.21 pp | 0.49 | +0.06 pp |
| Layers 32–35, K=2, beta=0.5 | −0.47 pp | 0.08 | +0.08 pp |
| Layers 33–36, K=2 | −0.51 pp | 0.06 | +1.39 pp |
| Layers 30–33, K=2 | −0.59 pp | 0.04 | −3.48 pp |
Layer scan on a 2,000-question subset
Baseline 60.85%. On this smaller set, two runs of the same config can differ by about 0.7 pp, so read the numbers with that in mind.
- Position. Layers 0–3 are a disaster: accuracy drops to 11.00%. Layers 4–7 and 40–43 cost 1.6 to 2.8 pp. Everywhere else lands between −1.3 and +0.1 pp. Even the best windows (44–47 and 56–59 at K=3, +0.10 pp) are noise.
- Width. Wider is worse. Eight-layer windows lose 1.9 to 3.2 pp; a 16-layer window loses 1.25 pp of accuracy and 6.1 pp of P(gold). One- and two-layer windows change nothing.
- Loop count. More passes hurt: K=2 −0.6 pp, K=4 −1.25 pp, K=6 −1.7 pp, K=8 −1.35 pp.
- Damping. Plain repetition did worse than small steps (−1.95 pp vs −0.60 pp), just as the paper predicts.
So the paper’s pattern showed up: damping matters, the first layers must be left alone, and narrow middle windows are safe. The gain itself stayed away.
Round 2: Qwen3.8-Flash-Next in a patched vLLM
Flash-Next needed deeper changes to vLLM, and we trusted none of its scores until the patched engine had proved it matched the original.
In short, the patch does three things:
- Repeats the chosen layers. The model walks its layers in order, runs the chosen window K times, then carries on.
- Writes memory once. Only the first pass through the window updates the model’s memory of the text. The extra passes work on a scratch copy and leave nothing behind, and the GDN running memory is saved and put back the same way as on the 27B.
- Refuses what it can’t do. Anything the patch doesn’t cover stops with an error instead of quietly running wrong. That limited us to 32k context.
There was one more wrinkle. Flash-Next often opens its answer with “To…” instead of a letter, so the most likely first token isn’t always an option. We recorded the probabilities of the letters A–J at the first output token instead.
Four checks before any sweep
We set four checks and ran nothing until all four passed.
| Check | Result |
|---|---|
| Original engine, run twice | Exactly the same letter probabilities (0 of 1,000 answers flip) |
| Patched engine, loop off | Exactly the same as the original |
| K=1 on layers 32–35 (loop machinery only, no repeats) | 7 of 1,000 answers flip (+0.5 pp, p=0.13), from one rounding difference |
| K=3 with the looped result thrown away | Exactly the same as K=1: the restore leaves no trace |
The last check failed the first time: 13 of 40 answers matched instead of all 40. Short answers looked fine, which pointed at how the prompt is read in. A counter was stuck at 1 after the loop, so every later layer thought it was on a repeat pass and skipped its memory writes. One fix later, all four checks passed.
Without that check, every config would have measured a quietly broken model, and the whole sweep would have read as “looping hurts.” That one table saved the experiment.
Round 2 results: flat, then a trap
Flash-Next gave us the same quiet “nothing happened.” The sweep covered 34 configs on 1,000 MMLU-Pro questions. The patched engine is fully deterministic, so a repeat baseline flipped zero answers. Baseline accuracy was 62.30%.
| Looped window (layers) | ΔP(gold | letters), worst | ΔP(gold | letters), best | Reading |
|---|---|---|---|
| 4–11 | +0.10 pp | +0.20 pp | Within noise |
| 12–15 | −0.25 pp | −0.13 pp | Flat or slightly worse |
| 16–19 | −0.32 pp | −0.13 pp | Flat or slightly worse |
| 20–23 | −1.96 pp | −1.44 pp | Clearly hurt |
| 24–31 | −0.80 pp | −0.18 pp | Flat or slightly worse |
| 32–35 | −0.57 pp | −0.11 pp | Flat or slightly worse |
| 36–43 | −1.21 pp | −0.70 pp | Clearly hurt |
| 44–46 | −0.22 pp | −0.13 pp | Flat or slightly worse |
No accuracy change was significant anywhere; every p-value was 0.11 or higher. The best window (layers 16–19, K=3) gained +0.7 pp at p=0.39, which is a coin toss. Variants didn’t help either: K=4 and K=6 were flat or worse, 8-layer windows gave +0.07 pp and +0.16 pp, and beta=0.5 gave +0.07 pp.
One difference from the 27B: Flash-Next shrugs off looping layers 4–7, where the 27B lost up to 2.8 pp.
The letter-mass trap
Halfway through, it looked like we had something. Windows across layers 8–19 raised P(gold), the probability of the correct letter, by up to 0.35 pp. Undamped looping of layers 16–19 looked even better. We got a little excited.
It was a mirage. Remember that Flash-Next likes to open with “To…”. Looping makes that opening less likely and every answer letter more likely, right ones and wrong ones alike. Raw P(gold) counts that as progress.
The fix is to look only at the answer letters: P(gold | letters). With undamped looping, the letters gained +7.75 pp of total probability, but the corrected score moved only +0.15 pp. The model got more willing to answer with a letter, not better at picking the right one. The table above uses the corrected score.
Confirmation on fresh questions
We took the two best candidates and ran them again on 3,000 MMLU-Pro questions (2,000 of them new) and on all 198 GPQA-Diamond questions, a hard set of graduate-level science questions. The baselines were 62.93% and 50.51%.
| Config | MMLU-Pro 3,000: Δ accuracy | Fresh 2,000: Δ accuracy | Fresh 2,000: ΔP(gold | letters) | GPQA-Diamond: Δ accuracy | GPQA-Diamond: ΔP(gold | letters) |
|---|---|---|---|---|---|
| Layers 8–11, K=4 | +0.30 pp (p=0.51) | +0.10 pp | +0.43 pp | −2.02 pp (p=0.42) | −0.01 pp |
| Layers 16–19, K=3 | +0.10 pp (p=0.87) | −0.20 pp | −0.36 pp | +1.01 pp (p=0.82) | +0.34 pp |
Neither held up. Gains turned into losses between benchmarks, and nothing came near significance.
What we learned
Training-free looping is real, and it runs cleanly on these hybrid models. On knowledge tests at this size, it just doesn’t make them smarter.
- The mechanism transfers. Damping beats plain repetition, the earliest layers can’t be looped, and narrow windows are safe, all as the paper says. With the running memory saved and restored, these hybrid models loop exactly, which our checks showed bit for bit.
- The gains don’t. The paper’s own numbers shrink with model size, and at 27B dense and 125B MoE they reach zero. Flash-Next’s small active share didn’t rescue it. One guess: a mixture-of-experts model may pick different experts on every pass, so each pass does different work instead of refining the same work. Another: the gains need more than repetition alone. Both are guesses.
- Check what your score rewards. Raw P(gold) rewarded the model for answering with a letter instead of “To…”. Any first-token scoring of a model that doesn’t always answer with a letter needs P(gold | letters).
- Check before you measure. Four small checks caught the bug that would have quietly spoiled all 34 Flash-Next configs.
What we didn’t try: thinking-on reasoning tasks, where extra passes might pay off more than a one-letter answer does; the cache=last variant, which keeps the last pass’s memory instead of the first; and windows that straddle two of Flash-Next’s layer groups (three GDN layers plus one attention layer each). If looping helps anywhere at this size, reasoning with thinking on is where we’d look next.
Reference
- Chen et al. (2026). Training-Free Looped Transformers. arxiv.org/abs/2605.23872
Models: Qwen3.8-27B (dense, hybrid Gated DeltaNet and attention) and Qwen3.8-Flash-Next (125B mixture of experts, about 6B active parameters), both in vLLM. Benchmarks: MMLU-Pro and GPQA-Diamond, zero-shot, thinking off, scored at the first output token. No models, data or code were released with this experiment.