Topics in Deep Learning Theory

Course Description

This course builds a rigorous theoretical foundation for modern machine learning, tracing ideas from classical linear models to today’s large-scale neural networks. Through four connected modules, you will master the core mathematical tools, landmark results, and open research questions that shape deep-learning theory.

Prerequisites

• Completion of the course Mathematical Introduction to Machine Learning or familiarity with machine learning and deep learning concepts. A formal background in machine learning theory is not required.

• This course is mathematically rigorous and requires a solid background in linear algebra, mathematical analysis, and familiarity with high-dimensional probability and functional analysis. Please ensure you have the necessary preparation before enrolling.

Lecture Notes

• Part I: Foundational Framework

  • Lecture 0: Concentration Inequalities

  • Lecture 1: Generalization via Uniform Concentration

• Part II: Classical Theory

  • Lecture 2: Linear Methods for Regression

  • Lecture 3: Reproducing Kernel Hilbert Spaces

  • Lecture 4: Theoretical Analysis of KRR

  • Lecture 5: Learning in RKHS with SGD

  • Lecture 6: Random Feature Models

• Part III: Deep Learning Theory

  • Lecture 7: Two-Layer Neural Networks

  • Lecture 8: Optimization of Neural Networks

  • Lecture 9: Deep Neural Networks

  • Lecture 10: Implicit Bias/Regularization I

  • Lecture 11: How GD/SGD Converges to Flat Minima

  • Lecture 12: GD Converges to Max-Margin Solutions

• Part IV: Frontier Topics

  • Lecture 13: Feature Learning

  • Lecture 14: Neural Network Landscape

  • Lecture 15: Emerging Phenomena in LLM

Relevant Books and Courses

• Introduction to ML from a theoretical perspective:

  • Shai Shalev-Shwartz, Shai Ben-David, Understanding Machine Learning: From Theory to Algorithms.

  • Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar, Foundations of Machine Learning.

• More theoretical textbooks:

  • Francis Bach, Learning Theory from First Principles.

  • Roman Vershynin, High-Dimensional Probability

  • Ramon van Handel, Probability in High Dimension.

• Deep Learning Theory Courses:

  • Matus Telgarsky, Deep Learning Theory.

  • Joan Bruna's course: Mathematics of Deep Learning