Research

I work toward a predictive science of deep learning: explaining learning mechanisms, predicting training dynamics, and guiding algorithm design. My research moves between mathematical analysis of tractable models and large-scale experiments, with each informing and challenging the other.

LLM Pretraining: Modeling, Prediction, and Design

We seek a macroscopic description of LLM pretraining through effective equations that capture the mechanisms governing loss dynamics. This approach helps explain training behavior and improve training efficiency.

  • Functional scaling laws (FSL). We introduced FSL: effective equations that describe learning dynamics through the competition between signal learning and noise forgetting (NeurIPS 2025). By analyzing how training choices alter this competition, we can explain and predict their effects:

  • ELR collapse. We identify the effective learning rate — the learning rate divided by parameter norm — as a more fundamental variable than the nominal learning rate for describing LLM loss dynamics (arXiv 2026).

  • Optimizer design. We develop optimizers for efficient LLM pretraining (ICML 2025, ICML 2026).

For more on FSL, see my talk slides.

SGD Dynamics: Stability, Sharpness, and Generalization

Why does SGD always find minima that generalize well, even when the model is overparameterized? We introduced dynamical stability as a mechanism for SGD's implicit regularization. Our work links stability constraints and SGD's noise geometry to minima selection, sharpness, and generalization.

For an overview of this line of work, see my talk slides.

Neural Network Theory: Approximation and Estimation

We introduced and named Barron spaces for two-layer neural networks, establishing approximation and generalization results within this function-space framework. Our broader work studies how neural networks efficiently learn high-dimensional functions and when and why they outperform linear methods.

For more on approximation–estimation duality, see my talk slides.