NeurIPS 2026 Accepts Paper Proving Transformer In-Context Learning Implements Gradient Descent on Function Space
Tags AI · Research · Infrastructure
A paper accepted at NeurIPS 2026 proves that transformer in-context learning implicitly performs gradient descent on a function space defined by the attention mechanism. The authors show that the forward pass of a frozen transformer on a few-shot prompt computes the same update as one step of gradient descent on a kernel regression objective, with the attention weights corresponding to the kernel matrix. This provides a rigorous mathematical explanation for why transformers can "learn" new tasks from context without weight updates.
Technical significance
This theoretical result unifies two previously separate views of transformer behavior: in-context learning as implicit optimization, and attention as kernel methods. For practitioners, it suggests that prompt engineering is effectively designing the initialization and data for an implicit optimization process — meaning prompt format, example ordering, and label distribution directly affect the "gradient step" the model takes. The kernel perspective also implies fundamental limits: tasks requiring optimization landscapes not representable by the attention kernel cannot be learned in-context regardless of model size.