Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation for LLM Training
Tags Research · AI
Researchers introduced Pion, a new optimizer for LLM training that uses orthogonal transformations to preserve weight matrix singular values during training, offering a stable alternative to Adam and Muon. Pion updates weight matrices through left and right orthogonal transformations, preserving singular values throughout training — unlike additive optimizers. The method modulates the geometry of weight matrices while keeping their spectral norm fixed. Empirical results show Pion is a stable and competitive alternative for both LLM pretraining and finetuning. Published on arXiv 2605.12492 on 2026-05-13.
Technical significance
Optimizer design is an underappreciated lever for improving LLM training efficiency. Pion's spectrum-preserving approach addresses a known issue with Adam and Muon: additive updates can distort weight matrix geometry over time. If Pion delivers on its promise of stable training at scale, it could become a standard tool in the LLM training toolkit, particularly for long-running pretraining jobs where optimizer instability accumulates.