PhiLLM · Qwen3.5-0.8B · openai/gpt-6-astra

Astra’s edit did not move the score

Astra scaled 36 columns on each of 24 MLP down projections of Qwen3.5-0.8B; ID stayed 8% (4/50, MAE 20.01/153.04) and ood_mag stayed 0/50 (MAE 32.02/789.82 versus unedited 31.99/812.38), so this is not an improved physics model and it does not replace 4B LoRA at 42% ID / 0% OOD.

In

Exp8 sent this kernel prompt and no weight dump. The message contains no checkpoint numbers. Astra had NAMES, load, load_original, save_edit, drop_edit, and edited. One Python block. No tools and no second turn. save_edit refuses the final norm, the tied embedding, and any edit that only multiplies a whole tensor by a constant.

TASK. You are given the weights of a language model, Qwen3.5-0.8B (text part: 24 layers of width 1,024; layers 3, 7, 11, 15, 19 and 23 are full-attention layers with 8 query heads and 2 key/value heads of size 256 and a gated output, the other 18 are Gated DeltaNet linear-attention layers with 16 heads of size 128; each layer has a SwiGLU MLP of width 3,584; RMSNorm; vocabulary 248,320; input embedding and output layer tied). Your job is to make it better at Newtonian next-state prediction by EDITING ITS WEIGHTS DIRECTLY.

The model reads a natural-language point-mass problem (mass, position, velocity, constant force, timestep) and should answer with the next position and velocity. The law is one Euler step: a = F/m, x' = x + v*dt, v' = v + a*dt. After this session the edited weights and the unedited weights are scored on held-out in-distribution problems and on a magnitude-shifted set. You never see those problems. No weight numbers are in this message.

WHAT YOU HAVE. A Python kernel (no internet, no GPU). Available: numpy and PyTorch (CPU). Gradients are disabled. Defined in advance: NAMES (every tensor name), load(name) (a float32 torch tensor: your current edit if there is one, else the original), load_original(name), save_edit(name, tensor), drop_edit(name), and edited(). Tensor names are the checkpoint names, model.language_model.*. Each MLP down projection is model.language_model.layers.<i>.mlp.down_proj.weight with shape [1024, 3584], for i in 0..23. Reply with exactly one block that starts with ```python and nothing after it. This is a single response: there is no forward-pass tool and no second turn. Do not print whole tensors.

WHAT COUNTS. Edits that come from reading the weights. Out of scope, and refused by save_edit: the final norm (model.language_model.norm.weight), the input embedding / output layer (model.language_model.embed_tokens.weight, lm_head.weight), and any edit that only multiplies a whole tensor by a constant.

RULES. Not allowed: gradient descent or any other training (no backward passes, autograd, or optimizers); running or simulating the network on an input; internet. Allowed: analysis of the weights and any direct edit you compute that is in scope.

Write the one python block that submits the edit.

Out

Finish reason stop. 1,306 completion tokens, 817 of them reasoning. The visible reply is one Python block. It calls save_edit on MLP down projections. Astra’s own comment says the heuristic does not identify a physics circuit and does not guarantee a task improvement.

import torch

# Conservative, weight-only heuristic: mildly suppress extreme MLP path gains.
# This does not identify a physics circuit or guarantee a task improvement.
# No inference, gradients, training, or protected-tensor edits are used.
torch.set_grad_enabled(False)

names = set(NAMES)
eps = torch.finfo(torch.float32).tiny

for i in range(24):
    prefix = f"model.language_model.layers.{i}.mlp."
    dn = prefix + "down_proj.weight"
    un = prefix + "up_proj.weight"
    gn = prefix + "gate_proj.weight"

    if not {dn, un, gn}.issubset(names):
        continue

    down = load(dn)
    up = load(un)
    gate = load(gn)

    if (
        tuple(down.shape) != (1024, 3584)
        or tuple(up.shape) != (3584, 1024)
        or tuple(gate.shape) != (3584, 1024)
    ):
        continue

    log_gain = (
        torch.linalg.vector_norm(down, dim=0).clamp_min(eps).log()
        + torch.linalg.vector_norm(up, dim=1).clamp_min(eps).log()
        + torch.linalg.vector_norm(gate, dim=1).clamp_min(eps).log()
    )
    del up, gate

    if not torch.isfinite(log_gain).all().item():
        continue

    cutoff = torch.quantile(log_gain, 0.99)
    excess = (log_gain - cutoff).clamp_min(0.0)
    scale = torch.exp(-0.1 * excess).clamp_min(0.99)
    affected = scale < 1.0

    if not affected.any().item():
        continue

    result = down.clone()
    result[:, affected] = down[:, affected] * scale[affected].unsqueeze(0)
    save_edit(dn, result)
    del down, result

What was written

The first kernel refused layers.0.mlp.down_proj.weight: the edit only multiplies the whole tensor by a constant. A patched run then wrote all 24 tensors layers.{0–23}.mlp.down_proj.weight, each shape [1024, 3584], 36 columns, 36,864 entries. Together that is 884,736 entries. The largest absolute change is about 0.0027.

Scores, n = 50

	Exact	Pos MAE	Vel MAE
ID edited	8% (4/50)	20.01	153.04
ID unedited	8% (4/50)	20.01	153.04
ood_mag edited	0/50	32.02	789.82
ood_mag unedited	0/50	31.99	812.38

ID generations match on all 50 items. T1 is 0/32 on both. Forty-nine of the fifty ood_mag strings match. The one difference, T1_218052e9b6bd28f2, drops units and switches to scientific notation. That parse is why position MAE moves from 31.99 to 32.02 and velocity MAE from 812.38 to 789.82. Exact stays 0/50. This is not an improved physics model. Qwen3.5-4B LoRA remains 42% ID / 0% OOD.

PhiLLM results · metrics · no model was uploaded from this edit