A language model opened another model's weights, found the direction that made it loop, and subtracted it. It changed 131,072 weights, 0.016% of Qwen3.5-0.8B, and the model's MATH-500 score went from 7.9% to 48.1%. There was no training and no gradients.
The result comes from the post "Sparks of Weight-Space Programming", by Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab at MIT CSAIL, published on November 2, 2026. The tweet summarizing it passed 5 million views on its first day. The figures in this article are the authors': we have not replicated them, and the post itself calls them early results.
1. What is weight-space programming?
It is the idea that a general-purpose language model reads and writes another neural network's weights directly, guided by a goal in natural language and with no optimizer. The authors call it Weight-Space Programming. The model they used appears in the post as GPT-6 Astra.
The post covers eight experiments. Four read weights (identifying which game a network plays or which language it writes) and four write or fix them. The most quoted one is Qwen3.5-0.8B, which is the one that matters to a company running small models on its own servers.
2. What they did with Qwen3.5-0.8B
The starting model has 24 layers and 0.8B parameters, with thinking mode on. Astra worked in a Python kernel over the weights, with no GPU, using practice problems from the GSM8K (7,469) and MATH (7,275) training sets. It had a budget of 10, 25, or 50 forward passes, and each pass let the model answer up to 512 problems.
The base model loops: of its first 96 practice answers, 85 ran to the 32,768-token limit. Astra traced the looping to one direction written by the MLPs in layers 15 and 19, and subtracted it from 64 output columns in each. That is 131,072 weights out of 0.8B.
The post also notes that Qwen's own model card already warns about this looping behavior in reasoning mode.
3. The results, by pass budget
Same sampling settings in every row (temperature 0.6, top-p 0.95, top-k 20, up to 32,768 tokens):
| Qwen3.5-0.8B | MATH-500 (%) | GSM8K (%) | IFEval (%) | Answers hitting the token limit (%) |
|---|---|---|---|---|
| Base | 7.9 | 13.8 | 21.0 | 93.0 |
| Edited, 10 passes | 12.3 | 25.7 | 49.0 | 87.2 |
| Edited, 25 passes | 14.1 | 25.1 | 39.9 | 86.4 |
| Edited, 50 passes | 48.1 | 67.1 | 62.3 | 33.1 |
The big gain arrives at 50 passes. At 10 and 25, MATH-500 only reaches 12.3% and 14.1%. IFEval, which measures instruction following, improves by more than 3x even though the edit did not target it, because looping hurts every task.
4. What the experiment does not show
The authors are clear about the limits, and they are worth reading before drawing conclusions:
- Fixing loops is not the same as improving math reasoning. The edit lets the model finish answers it could already solve.
- On Gemma 3 270M, which rarely loops (8.5% of its answers hit the limit, against 93% for Qwen), Astra could not improve anything. MATH-500 moved from 6.8% to between 7.0% and 7.7% depending on budget, a change of under one point.
- Gemma's failures were arithmetic mistakes and invalid algebra steps, with no single component to switch off. Hence the post's conclusion: editing weights can remove an obstacle that hides an ability, but it cannot add an ability the model lacks.
- Several experiments are single attempts or single replies, such as the control-policy fix (Kinetix) and the from-scratch weight writing.
- The authors are building a benchmark and a harness for the method. Neither is released yet.
5. Other results in the post
Three experiments help calibrate how far the idea goes:
- Writing weights from scratch: with a zero-initialized 600→32→32→6 network, Astra wrote the weights in a single reply to play MinAtar Breakout. It scored 517.4, against 25.3 for PPO trained for 500 million steps. On Seaquest (53.1 vs 88.9) and Asterix (53.2 vs 55.2) it landed below or on par with PPO at 500M.
- Reading weights: from the middle layers of three 7.16M-weight transformers, with no labels, Astra got the language right (Python, romaji, English) and flagged the copying heads that were later confirmed by running the model.
- Detecting sycophancy: from the weights alone of two Gemma 3 270M copies, one honest and one sycophantic, Astra failed in all four attempts. Once it could see attention patterns on texts it wrote itself, it identified the sycophantic model in all three pairs.
6. What it means for a small business
What follows is our reading, not a conclusion of the post.
First, diagnose before you buy. If your local model gives bad answers, measure what share of them hit the token limit. In Qwen that number (93%) explained most of the problem. A model whose issue is looping needs a fix, not a bigger model or a new server.
Second, the cost changes nature. The workflow in the post does not train and uses no GPU to edit, but it runs the model dozens of times to check the changes. Spending moves from training to inference and to a strong reasoning model directing the process. That model is a paid one, and the post does not say how to access GPT-6 Astra.
Third, there is no tool you can install today. Code and logs from the experiments are public, but the harness and benchmark are not, and the results depend on a model you may not be able to use. For a production project this week, the realistic path is still to choose the model well, tune sampling, and measure.
Fourth, the direction is worth tracking. If editing weights with a general model becomes reliable, a small model with one specific defect could be fixed in hours without repeating a fine-tune. With your own data and models running inside your network, that would lower the cost of maintaining local AI. Today it is a hypothesis that the authors also describe as a belief, not a result.
What to do this week
In your current tests, measure the share of answers cut off by the token limit and the share that repeat text. If a meaningful share of answers gets cut off or repeats, first check the sampling settings in the model card and the token limit. If the problem persists in one specific model, keep that case with examples: it is the kind of failure this method could target once the harness is available.
Book your 60-minute technical consultation here and we will review which model, hardware, and settings fit your use case today, with tools that already exist.
Frequently asked questions
Is it true that an AI improved another AI without training it?
According to the authors, yes: Astra edited 131,072 weights of Qwen3.5-0.8B (0.016%) with no gradients or training, and MATH-500 went from 7.9% to 48.1% with 50 forward passes. It is a single group's result, described as early. The full logs are published on GitHub so others can check them.
Can I do this with my company's model?
Not today. The method uses a very strong reasoning model (GPT-6 Astra in the post) and a harness the authors have not yet released. You can apply the diagnostic idea: measure how many of your model's answers end in a loop or get cut off by the token limit.
Does it work on models that already perform well?
The post says no. On Gemma 3 270M, which rarely loops, Astra did not improve math. Weight editing fixed a specific Qwen defect and did not add capability.
What is a reasoning loop?
It happens when a model in reasoning mode repeats the same text and never reaches a final answer. In Qwen3.5-0.8B, 85 of the first 96 practice answers used up the 32,768 allowed tokens. The model card already warns about this behavior.
Should I change my AI strategy because of this result?
No. It is a line of research worth following, with promising results and stated limits. To decide today, use published models, measure with your own cases, and revisit your strategy once the method is available and someone has replicated it.