GPT-6 Astra Qwen Math Score Explained

GPT-6 Astra edited 0.016% of Qwen3.5-0.8B's weights, and the model's MATH-500 score went from 7.9% to 48.1%. Before the edit, 93.0% of its answers hit the output limit and stopped mid-thought. The headline reports the score, and the table reports the answers that never ended.

Verdict: half true. Token-limit hits fell from 93.0% to 33.1%, and 576 of 579 rescued practice answers had been cut off before the edit.

The sources sit one step from the primary

I did not have the original post. "Sparks of Weight-Space Programming," by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab of MIT CSAIL, is dated November 2, 2026. Neither it nor the released experiment logs was among my sources.

I had an independent explainer, which says "The authors of the sources did not write or review it." I also had secondary coverage. One piece of the coverage is vendor content from Apidog and another is labelled AI-generated.

Nearly every rescued answer had been cut off

Astra got a Python session with the weights loaded, no GPU, and 7,469 GSM8K plus 7,275 MATH practice problems. It never saw the test sets. No gradient descent was involved. Astra changed 131,072 weights, which is 64 output columns of down_proj in each of two MLP layers, 15 and 19.

The Qwen model card warns the model is prone to thinking loops. Astra traced the looping to one direction written by the MLPs. Neel Nanda reads the work as "old school mechanistic interpretability research rather than anything spooky."

The 10-pass edit reached 12.3% on MATH-500 with 87.2% of answers still hitting the limit. A 25-pass edit reached 14.1% and 86.4%. Only the 50-pass edit reached 48.1% and 33.1%. Loops "barely went away" at the lower budgets, per Apidog's write-up. These are three different edits, so I read the pattern as correlation, with no controlled dose curve behind it.

The control points the same way. Gemma 3 270M rarely loops, with 8.5% of answers hitting the limit. Its best edit moved MATH-500 from 6.8% to 7.7%. The same write-up relays the authors' conclusion: editing removed an obstacle hiding an ability and did not add a missing one.

A model that finishes its answers has gained something real

Here is the strongest objection. A model which failed to finish 93.0% of its answers now scores 48.1% on the authors' evaluation. Ahsen Khaliq of Hugging Face re-ran the full MATH-500 (500 problems, 4 samples each) and got 51.0%. An ability the user never sees is absent in practice.

Takehi describes the method as treating raw weights like readable, editable code, diagnosing failures and patching them directly. Under this frame, a patched failure is the capability, and my wording above demotes the achievement.

The mechanism is also unsettled. IFEval rose from 21.0% to 49.0% in the 10-pass edit while limit hits moved only from 93.0% to 87.2%. If loop removal explained everything, IFEval should have stayed put. The sources do not say which benchmark the limit column measures, so I infer the jump comes from shorter responses or a second failure.

The base row has its own puzzle: 93.0% of answers hit the limit, yet accuracy is 7.9%. Either some truncated answers were scored correct or the columns use different bases. Only the logs would settle both.

I still land on loop removal, with softer wording: the gain is consistent with removing a failure mode. The sources give no direct evidence of added math skill. Gemma, with little looping to remove, barely moved. Of 579 rescued practice answers, 576 had been truncated before the edit. Khaliq's run was a rebuild from the authors' logs, so I read it as confirming the checkpoint's score and nothing more.

Matthew Honnibal's critique is titled "The 'Astra can read and write weights to improve Qwen' post is misleading." Sean Goedecke's is titled "LLMs are not superhuman at reading and writing neural network weights." I had only the titles. Their arguments might undercut this essay, and I will not guess at them.

The cost is in hours, and no dollar figure appears

The strongest attempt took 158 conversational turns, 80 Python runs and 20.3 hours, including 4.7 hours waiting out a service outage. Some coverage cites a $1,000 cost. No source I had carries a dollar figure. An AI-generated summary says the paper does not show direct editing is cheaper or faster than training, and nothing in the other sources contradicts it.

From weights alone, Astra named the game for 4 of 4 PPO-trained game policies and the training language for 3 of 3 character-level models. It missed the sycophantic Gemma 3 270M copy until it saw attention patterns.

The numbers support a diagnosis. Astra traced the looping in a 0.8B model to one direction written by the MLPs, working from forward passes on practice problems. Whether weight editing beats training on cost is untested. The authors call every result "very early," and several rest on one or a few runs.