For now, I tried a small experiment:
Short answer
I take your question to be about one precise intervention: retaining RMSNorm immediately before QKV and FFN projections, but switching its learned per-channel gain (elementwise_affine, often called gamma) ON or OFF, while leaving norm locations and the rest of the architecture unchanged. That is a useful distinction. It is not a LayerNorm-versus-RMSNorm test, a pre-norm-versus-post-norm test, or an experiment about adding or removing the final output norm.
I have not verified a published or unpublished ablation that exactly satisfies that contract among the sources I checked. I would not infer that none exists; the nearby papers and implementations below are useful for the mechanism or for architectural precedent, but they are not direct substitutes for a gamma-only comparison.
I ran a small, paired, exploratory comparison on a three-block byte-level Transformer. The intervention was limited to the six RMSNorm gains at the pre-attention QKV and pre-FFN locations; a separate final RMSNorm remained non-affine in every condition. The hidden 2D matrices used either AdamW or a custom FP32 NorMuon-style update, with gamma, embeddings and the output head on AdamW. I used three seeds and two global gradient-clip thresholds, for 24/24 completed arms, each trained for 1,200 updates.
At the final checkpoint, gamma ON had lower validation bits per byte (BPB) in all six AdamW paired comparisons and in five of six NorMuon-style comparisons, but the NorMuon effect varied much more by seed and clip setting. These are three seeds crossed with two clip settings, not six independent training seeds. In this one toy setting, removing gamma was not automatically loss-neutral at a matched update budget. That does not mean gamma-free RMSNorm is a bad design, nor that learned gamma is universally better.
The main conceptual point remains promising: when gamma feeds directly into a linear projection, W diag(gamma) can be represented by a single effective matrix. That is a statement about the forward function. It is not an assertion that optimizing W and gamma separately, possibly with different optimizers, traces the same path as optimizing the folded matrix alone.
The default path I would use
I would keep the gamma-free design on the table, especially if the goal is a cleaner matrix-only Muon/NorMuon parameter organization. Use a gamma-ON reference and an otherwise identical gamma-OFF run within the optimizer that matters for your project. Match the seed, common initial weights, sample order, evaluation windows, clipping definition, schedule and training budget. Then choose according to the engineering objective:
- If OFF reaches the target validation quality and stability tolerance, it can be a reasonable simpler parameterization even if another run slightly favors ON.
- If quality at a particular update budget is the priority, treat the observed ON/OFF loss gap as a real hypothesis to test with more seeds, better evaluation and optimizer-appropriate tuning.
- If equal compute is the priority, measure compute or wall time; equal training updates and sampled tokens do not answer the strict question.
- If late stability is the priority, extend the run and define gradient spikes or sustained loss oscillations explicitly. Do not diagnose late instability from a small validation-checkpoint rebound.
None of those branches requires deciding in advance that one parameterization must win. The point of a controlled ablation is to see where each one works, and at what cost.
Minimal setup before the numbers
| Factor |
This probe |
| Model |
Causal decoder, 3 blocks, width 128, 4 heads, context 96, byte vocabulary 256 |
| Changed |
Six pre-QKV/pre-FFN RMSNorm gamma vectors, all ON or all OFF; eps=1e-6 |
| Unchanged |
The separate final RMSNorm is always affine=False; same attention/FFN topology, no projection biases or dropout |
| Data |
Fixed contiguous 90/10 split of Tiny Shakespeare |
| Pairing |
Seeds 17, 41, 79; matched non-gamma initialized weights, train sample order and validation windows within each seed |
| Conditions |
Hidden 2D matrices on AdamW or NorMuon-style update; global clip at 1.0 or 3.0; gamma ON/OFF |
| Budget |
24 arms, 1,200 steps each, batch 16, context 96; validation at step 0 and every 100 steps |
| Metric |
Next-byte held-out cross-entropy divided by ln(2), in bits per byte (BPB); lower is better |
| Precision and optimizer |
T4, full FP32, custom Newton–Schulz NorMuon-style update; not an unmodified official NorMuon run |
That is 1,843,200 sampled token positions per arm (1200 × 16 × 96), not unique corpus positions; the same data windows can repeat. Validation uses 9,216 sampled positions per checkpoint per seed (6 × 16 × 96). The windows are shared for ON/OFF comparisons within a seed but differ between seeds. Hidden-matrix learning rates were 0.001 for AdamW and 0.02 for NorMuon-style; auxiliary AdamW LR was 0.001. These are paired parameterization tests, not a fairly tuned ranking of the two optimizers.
Paired final validation results
The quantity is BPB(gamma ON) minus BPB(gamma OFF) for the same seed, matrix optimizer and clip threshold. Negative means ON achieved lower BPB. Values are mean ± sample standard deviation over three seed pairs, not confidence intervals.
| Hidden-matrix optimizer |
Clip threshold |
ON minus OFF validation BPB |
Sign across 3 seeds |
| AdamW |
1.0 |
-0.021321 ± 0.005683 |
ON lower, 3/3 |
| AdamW |
3.0 |
-0.018895 ± 0.002949 |
ON lower, 3/3 |
| NorMuon-style |
1.0 |
-0.005642 ± 0.015067 |
ON lower, 2/3 |
| NorMuon-style |
3.0 |
-0.017173 ± 0.021250 |
ON lower, 3/3 |
Thus the two clip cells give AdamW 6/6 and NorMuon 5/6 negative signs, not six independent seed replications for each optimizer. One NorMuon clip-1 seed actually preferred gamma OFF. The NorMuon sample SDs are substantial compared with the means, and the checkpoint-time signs can change. Reporting only the aggregate mean, or calling the interaction statistically significant, would be stronger than the data support.
How I would answer the four questions
1. Final validation loss or downstream quality. The toy final validation BPB result is above. At this fixed endpoint, gamma ON was usually lower, most consistently under AdamW in this setup. But there was no downstream evaluation: no instruction following, reasoning, multilingual, generation-quality, long-context or separate task benchmark. A difference of hundredths of a BPB on one tiny character-like byte corpus cannot be translated into downstream improvement. Nor have I found a larger external gamma-only result that resolves that question.
2. Convergence speed at matched compute. I logged held-out BPB from initialization through step 1,200 at 100-step intervals. Early paired differences were small; the AdamW ON advantage generally grew by later checkpoints, whereas the NorMuon contrasts crossed zero at some intermediate points. This is matched update count, context and sampled positions, not measured FLOPs, optimizer overhead, throughput or wall-clock equivalence. Nor were independent LR and schedule sweeps performed. A strict matched-compute claim requires loss-versus-measured-compute curves, and a fair optimizer comparison requires appropriate tuning.
3. Gradient spikes and late oscillations. All 24/24 arms finished; there were zero logged nonfinite training losses, pre-clipping gradient norms or validation checkpoints. With clip threshold 3.0, there were 0 clipped steps out of 14,400 across both optimizer branches. With threshold 1.0, clipping affected 54/7,200 AdamW steps and 76/7,200 NorMuon steps, with the latest event at steps 11 and 15 respectively. There was no clipping in steps 601–1,200. Some NorMuon seed-17 arms ended slightly above their best step-1,100 validation BPB, but that observation alone is not proof of unstable training or late spikes. A longer horizon and an explicit event definition would be needed.
4. AdamW versus Muon/NorMuon. The gamma contrast looked more consistent in the AdamW branch and more variable in the NorMuon-style branch. The comparison that is controlled is gamma ON versus OFF within a fixed optimizer/clip/seed, not absolute AdamW versus NorMuon BPB. The hidden learning rates differ and my NorMuon matrix update is a custom FP32 adaptation for the T4, not the upstream package used unchanged. The official NorMuon repository recommends routing hidden matrices to the matrix optimizer and gains, biases, embeddings and head parameters to AdamW. That separate routing is precisely why the gamma question is interesting.
The boundaries of this evidence
This is one three-block toy model, one byte-level corpus, three seeds, 1,200 updates, and a small fixed validation window per seed. The clip settings reuse seeds and are not independent replications. The extra gamma parameters and their AdamW states are deliberately part of the architecture intervention; identical total parameter counts are not claimed. No strict FLOP matching or downstream metrics were measured, and no production-scale convergence or stability law follows. The NorMuon-specific part was adapted to FP32. An earlier preliminary short run used a runner whose exact bytes are unavailable for a parity audit, so I do not combine its numbers or infer why its clipping behavior differed.
The useful narrow result is that functional absorbability does not guarantee loss-neutral training at a fixed budget. It is still reasonable to test an affine-free design on the actual target architecture and optimizer. The details below separate the exact intervention, individual measurements, optimizer reasoning, related work and practical branches.
A. Exact model, data, pairing and optimizer protocol
The intervention
Each of three pre-norm decoder blocks contains a norm before the combined QKV projection and another before the FFN. These are the six and only six gamma toggles. The final output norm remains affine-free in all arms. The attention branch uses bias-free combined QKV and output projections and causal scaled-dot-product attention; the FFN uses two bias-free linear layers with a 4× hidden expansion and GELU. The embedding and output head remain in both variants. No dropout or LR schedule is used.
# Placement schematic; the runnable Notebook contains the full model.
pre_attn_norm = nn.RMSNorm(width, eps=1e-6,
elementwise_affine=gamma_enabled)
pre_ffn_norm = nn.RMSNorm(width, eps=1e-6,
elementwise_affine=gamma_enabled)
final_norm = nn.RMSNorm(width, eps=1e-6,
elementwise_affine=False) # unchanged
def block_forward(x):
x = x + attention(pre_attn_norm(x))
x = x + ffn(pre_ffn_norm(x))
return x
According to the PyTorch RMSNorm API, elementwise_affine=True creates a learned per-channel weight initialized to ones; with False it is absent. RMS normalization is still performed with gamma OFF. Here gamma ON adds six 128-element vectors, 768 trainable scalars, rather than changing how many norm operations occur. Since gamma begins at one, the ON and OFF models have identical initial forward values when their other weights match. They can then follow different optimization paths.
Data and matched random streams
The raw Tiny Shakespeare file was 1,115,394 bytes, SHA-256 86c4e6aa9db7c042ec79f339dcb96d42b0075e16b8fc2e86bf0ca57e2dc565ed. The runner checks that digest rather than silently accepting a different source file. The first 90% of the byte stream is training data; the final 10% is validation data. Each sampled 96-byte sequence predicts the following byte. The data are small, highly overlapping and not an independent modern-language-model benchmark.
For each seed, training start positions and validation start positions were generated once using separate deterministic CPU streams, then reused across that seed’s eight arms. The model seed was reset before each arm; a SHA-256 fingerprint of all common non-gamma initialized weights matched across the eight conditions, as did step-0 validation BPB. Validation positions stay the same across that seed’s checkpoints. The sample windows differ between seeds, so the reported three-seed SD contains both training randomness and evaluation-window variation.
This pairing helps isolate gamma from initialization and data-order noise. It does not guarantee bitwise GPU determinism, establish a population-level effect, or magically create more independent observations. An ON/OFF pair still differs in parameter count and optimizer state because that is the actual intervention.
Routing, hyperparameters and implementation identity
Each block contributes four hidden 2D matrices: QKV, attention output, FFN input, FFN output, making 12 matrices. They go to AdamW or the custom NorMuon-style optimizer according to condition. Token and position embeddings, the LM head and the six gamma vectors when present go to AdamW. The gamma group uses zero weight decay; other weights use decay 0.01. The routing code rejects omissions, duplicates or wrong tensor shapes, and the output contains 24 per-arm routing ledgers.
| Setting |
Hidden AdamW condition |
Hidden NorMuon-style condition |
| Hidden LR |
0.001 |
0.02 |
| Matrix update |
AdamW |
Momentum 0.95, per-row second moment beta 0.95, five FP32 Newton–Schulz iterations, norm/shape rescaling |
| Hidden decay |
0.01 |
0.01 |
| Auxiliary optimizer |
AdamW, LR 0.001, betas (0.9, 0.95), epsilon 1e-8 |
Same auxiliary AdamW configuration |
| Gamma decay |
0 |
0 |
| Gradient clipping |
Once globally across all model parameters, threshold 1.0 or 3.0 |
Same rule within the matched condition |
The NorMuon-style step follows the core update structure of the upstream implementation, including orthogonalization and a per-row second-moment correction. For this T4 experiment, however, Newton–Schulz was performed in FP32. The single-device implementation and parameter grouping are custom. The result must therefore be called a NorMuon-style reproduction/adaptation, not an official-package benchmark. The upstream code and README are the relevant independent reference for an implementation-fidelity follow-up.
Each step computes next-byte cross-entropy in nats, calls backward, obtains the pre-clipping global gradient norm from torch.nn.utils.clip_grad_norm_, and steps the optimizers. The return value of that clipping call is the original norm, so a clip event is recorded when it exceeds the threshold. Each condition logs per-step training loss, norm and event flag; checkpoints contain validation BPB and gamma statistics. The execution reported a Tesla T4 and PyTorch 2.11.0+cu130 with full-FP32 model computation.
The result ZIP contains the executed source, configuration, 24 records and routes, 28,800 traces, and file hashes. CRC, 56 internal SHA-256 entries and paired-statistic checks passed. That verifies captured-file integrity, not independent replication. A public Notebook/result link should only be added after actual upload.
B. Seed-level results, convergence checkpoints and stability
Twelve paired endpoint contrasts
Here are all final validation BPB differences, computed separately by seed: gamma ON minus gamma OFF. A negative entry favors retaining gamma; a positive entry favors removing it. The main table above summarizes each row with the mean and sample SD of three values.
| Hidden optimizer |
Clip |
Seed 17 |
Seed 41 |
Seed 79 |
| AdamW |
1.0 |
-0.017738 |
-0.027873 |
-0.018351 |
| AdamW |
3.0 |
-0.015613 |
-0.021323 |
-0.019751 |
| NorMuon-style |
1.0 |
-0.021220 |
-0.004558 |
+0.008854 |
| NorMuon-style |
3.0 |
-0.000198 |
-0.010317 |
-0.041005 |
AdamW was consistently negative. NorMuon-style shows a sign reversal, a near-tie and a larger ON advantage in different paired cells. That heterogeneity is important; neither its mean nor its one reversal establishes a universal winner.
One should also keep fixed-endpoint validation BPB separate from the minimum recorded BPB over multiple checkpoints. Selecting the best of 13 checkpoints after training changes the evaluation rule and can favor an arm because of validation-window noise. The table above uses step 1,200 in all cases, without picking whichever step was most favorable for a variant.
The checkpoint-time gamma effect
The next table reports the mean paired difference over the same three seeds at a fixed checkpoint. It is not a comparison of “best checkpoint for ON” against “best checkpoint for OFF.”
| Step |
AdamW clip 1 |
AdamW clip 3 |
NorMuon clip 1 |
NorMuon clip 3 |
| 100 |
-0.00113 |
-0.00116 |
-0.00366 |
-0.00313 |
| 200 |
-0.00192 |
-0.00176 |
-0.00414 |
-0.00121 |
| 400 |
-0.00915 |
-0.00630 |
+0.01437 |
+0.00304 |
| 600 |
-0.01649 |
-0.01602 |
-0.02897 |
+0.00374 |
| 800 |
-0.01901 |
-0.01968 |
+0.00282 |
-0.00714 |
| 1,000 |
-0.02356 |
-0.02228 |
-0.00780 |
-0.02515 |
| 1,200 |
-0.02132 |
-0.01890 |
-0.00564 |
-0.01717 |
The AdamW gamma gap is small near initialization and generally becomes more negative later in the observed window. The NorMuon-style mean is more variable, with changes of sign at intermediate steps. A crossing of ON minus OFF does not imply that either raw training-loss curve became numerically unstable; a difference of two finite losses can cross zero without a spike in either. The dataset, small evaluation window, different per-seed validation batches and averaging can all contribute to how these curves appear.
None of the curves is a matched-FLOP convergence curve. Equal sampled data and updates were enforced, but cost from extra gamma parameters and from matrix orthogonalization was not measured. Nor does the 1,200-step endpoint demonstrate asymptotic convergence. With a production-level study I would show loss versus steps and loss versus measured compute, with fixed evaluation windows and optimizer-specific tuning costs stated.
Clipping: a startup-only event in this v2 run
The following counts come directly from per-update pre-clipping global gradient norms, with the condition norm > clip_threshold. “Latest” means the highest training-step index with a recorded clip event across the indicated six arms.
| Hidden optimizer |
Threshold |
Clipped steps |
Latest clip |
Clips in steps 601–1,200 |
| AdamW |
1.0 |
54 / 7,200 (0.75%) |
11 |
0 |
| AdamW |
3.0 |
0 / 7,200 |
None |
0 |
| NorMuon-style |
1.0 |
76 / 7,200 (1.06%) |
15 |
0 |
| NorMuon-style |
3.0 |
0 / 7,200 |
None |
0 |
Thus threshold 3.0 did not intervene at all, while threshold 1.0 affected only the first few updates in this v2 study. The clip setting can still change later loss through those early steps, but we did not prove that early clipping caused any particular gamma gap. Neither a smaller clipping fraction nor a larger threshold automatically establishes better stability. A model-wide norm also does not identify which parameter or layer was responsible for an event.
The secondary clip-3-minus-clip-1 endpoint differences varied, especially under NorMuon. Those are clip contrasts, not gamma contrasts; this grid cannot establish an optimal threshold.
The logs show no nonfinite values or failed arms. All twelve AdamW arms had their best recorded validation BPB at the final step. For four NorMuon seed-17 arms, step 1,100 was the best observed checkpoint and step 1,200 was roughly 0.0029–0.0130 BPB higher. This is a modest endpoint rebound, not a sustained training-loss divergence diagnosis. It would take longer training, more seeds, and predefined spike/oscillation measures to address the late-stability question reliably.
Statistical and provenance boundaries
There are 12 gamma contrasts in four groups of n=3, reusing seeds across clip settings and evaluation windows across checkpoints. These are not 12 independent seeds; SDs are descriptive, not significance tests. Small evaluation windows, LR sensitivity and parameter routing remain limitations.
An older 200-step exploratory runner had shown very frequent late NorMuon clipping, unlike this v2 run. Its exact executable was lost with transient files, and no byte-for-byte parity audit is available. I would not pool v1 and v2 statistics, declare that the discrepancy was fixed by longer training, or attribute it to gamma or clipping. The internally consistent v2 comparison is the basis of the claims here. That historical discrepancy is a reminder to include exact executed source and optimizer routing when publishing training results.
C. Folding gamma versus following the same optimization path
The exact algebra, with a graph condition
Write z = RMSNorm_noaffine(x). If a learned per-channel gamma multiplies z immediately before a linear projection W, the projection computes
y = W @ (gamma * z)
= (W @ diag(gamma)) @ z
= W_eff @ z.
So the gain can be absorbed into the input columns of the following matrix. A pre-attention gamma shared by Q, K and V can be folded into each of their input projections, or into the corresponding columns of a combined QKV matrix; pre-FFN gamma can be folded into the FFN’s first linear layer. This shows local forward-function equivalence, provided the normalized representation is consumed through those linear paths and any shared consumers or constraints are handled. It does not assert equivalence if an intervening nonlinear operation, tied-weight restriction or another branch is silently ignored.
The functional point is important for your motivation. It explains why removing explicit gamma need not reduce what the following unconstrained linear projections can represent. But a function class is not an optimization algorithm, and there is no implication that two different parameterizations will reach the same function at the same training step.
Why the update rules need not commute with folding
With M = W diag(gamma) as the effective matrix, changes to the factorized parameters produce, to first order,
delta M ≈ (delta W) @ diag(gamma)
+ W @ diag(delta gamma).
The second term exists only when gamma is optimized separately. In a gamma-ON model the gain has its own gradient, AdamW moments and potentially its own LR and weight decay; in an OFF model that coordinate is absent. The matrix update in Muon/NorMuon is nonlinear in its matrix-shaped gradients and optimizer state. Applying orthogonalization, momentum and row-wise rescaling to W while AdamW updates gamma is generally not the same update as applying the matrix optimizer directly to W_eff.
Even without a matrix-specific optimizer, AdamW’s moments and decoupled decay are parameterization-dependent. Here gamma had decay 0 while the matrices had decay 0.01; folding a non-decayed gain into a decayed matrix does not reproduce those separate regularization steps. Global gradient clipping is another difference: adding gamma changes which gradients enter the norm, even when the threshold remains fixed. These are possible mechanisms, not measured attributions. Our factorial test did not isolate moment adaptation, decay, clipping, conditioning or the matrix update as the cause of the BPB differences.
Gamma ON began at one and later moved: the final gamma mean absolute deviation from one was roughly 0.08–0.09 over the ON arms. That confirms the gain parameters were not left frozen. It does not identify a privileged channel or prove that their learned scale could not be represented by a folded matrix. A small number of vectors can affect many downstream matrix columns while still adding only 768 scalars to this model.
A cheap test worth separating from training
Given a trained gamma-ON checkpoint, one can fold each gamma into its following QKV/FFN input matrix, replace the relevant RMSNorm with the non-affine form, and compare logits on a fixed batch in evaluation mode. Matching logits to numerical tolerance would check that the implementation of the functional rewrite is correct. It would not show equal training trajectories after a restart, because optimizer moments and update geometry may differ. This checkpoint-folding test was not run in my 24-arm study; it is a suggested verification, not an additional observed result.
D. Nearby papers and code, without treating them as direct ablations
The search criterion matters: an exact answer would toggle learned gamma only at pre-QKV/pre-FFN RMSNorm sites, with other norm locations, architecture, data and training conditions held sufficiently constant to compare the requested outcomes. A working gamma-free model, a LayerNorm/RMSNorm swap, or a changed final norm does not by itself satisfy it.
| Source |
Why it is useful |
Why it does not settle this exact question |
| NorMuon paper and official code |
Matrix update, auxiliary AdamW routing and implementation baseline |
Not verified as a matched six-site gamma ON/OFF study |
| PyTorch RMSNorm documentation |
Exact meaning of elementwise_affine |
API specification, not training-outcome evidence |
| Hägele et al., Magnitude–Direction Decoupling and author explanation |
Closely related distinction between learned scale and matrix direction under Adam/Muon |
Different parameterization/placement, not the same gamma-only test; both links describe one research lineage |
| Mosaic training notes |
Affine-free, bias-free weather-forecasting model that routes all weights through Muon |
Architectural precedent, not a language-model gamma-only ablation |
| MACRO |
Overlapping questions about normalization and matrix constraints |
Broader changes to constraints/norms |
| Outlier-Safe Pre-Training (ACL 2025) |
Muon and single-scale RMSNorm with one learned scalar, in an outlier/quantization study |
A scalar scale is not gamma OFF; multiple architectural interventions, not a gamma-only test |
The NorMuon README distinguishes matrices from auxiliary gains/biases, supporting the architectural motivation. The magnitude–direction work suggests a related mechanism; Mosaic shows a design precedent. None independently confirms this gamma-only experiment. A larger direct ablation with controlled norm placement and routing would supersede these narrow toy observations.
E. Reproduction checklist and conditional next steps
For a reproducible gamma-only comparison, record: exact pre-QKV/pre-FFN and terminal norm graph; gamma initialization and epsilon; dataset revision, tokenizer and split; model dimensions; common initial-weight hash; per-seed train order and fixed evaluation windows; optimizer routing, LR, momentum/betas, decay, precision and clipping rule; update and actual compute budget; final and best checkpoint BPB; per-step loss, pre-clipping gradients, clipped flags and nonfinite status. The original 24-arm result archive contains source, configuration, per-arm routes, traces, checkpoints and checksums. It is locally audited, but no public link is asserted here.
The cheapest next check is not necessarily a training sweep. If a saved ON model checkpoint is available (this result ZIP contains metrics, not model weights), fold gamma into QKV/FFN input matrices and compare fixed-batch logits with the non-affine form. For the training claim, rerun matched ON/OFF pairs under the actual target optimizer, preserving all other norm sites. These check different questions.
A decision tree for choosing further work:
- If the goal is matrix-only simplicity: start with gamma OFF, retain ON as a paired quality baseline, and accept OFF only within a predeclared loss/stability tolerance.
- If the goal is lower validation loss: use larger held-out evaluation, more independent seeds, and LR/schedule sweeps; first compare gamma within each optimizer.
- If the goal is strict compute efficiency: measure FLOPs and optimizer overhead plus throughput; compare loss-versus-compute, not only step counts.
- If the goal is late stability: train longer, predefine spike/oscillation events, and log layer-level diagnostics when the global norm is suspicious.
- If the goal is upstream fidelity or transfer: repeat with a pinned official NorMuon implementation and then on other model/data scales and downstream tasks.
Results that would change my provisional interpretation include the AdamW sign reversing with more seeds or a larger evaluation set; NorMuon becoming consistent after tuning; a different outcome under the official precision/update path; an equal-compute curve reversing the endpoint ordering; or a long run showing convergence of ON and OFF. Those are testable alternatives, not claims that any has already occurred.
For future readers: separate measured gamma effects, function-level algebra and untested optimization mechanisms. None alone is a universal training rule.
Takeaway
I would keep affine-free pre-QKV/pre-FFN RMSNorm as a valid research branch. Gamma can be folded into directly following linear projections at the function level, but that does not make separately optimized gamma training-equivalent to a folded matrix. In this specific short, three-seed toy experiment, gamma ON usually gave lower final BPB; the NorMuon-style contrasts were less consistent, and no strict compute, downstream or long-run stability conclusion follows. A matched test on the intended architecture and optimizer is the sensible deciding evidence.