Theoretical Deep Learning

Lecture notes

1. A brief introduction to supervised learning

2. Concentration inequalities

   1. Sub-Gaussian, Chernoff bound, Hoeffding's inequality, McDiarmid's inequalty

3. Uniform bounds and empirical processes

   1. Rademacher complexity, Covering number, Dudley entropy integral

4. Kernel methods, representer theorem and RKHSs

5. RKHS II

6. Two-layer neural networks and the Fourier analysis

7. The Barron space

8. Deep neural networks

   1. lecture note,

   2. Approximation theory of deep ResNets

   3. Depth separations

9. Training neural networks: Convergence?

   1. A brief overview of GD convergence

   2. Slide

10. Training neural networks beyond the kernel regime

    1. Slide

References

• Peter Bartlett's course: Statistical Learning Theory

• Matus Telgarsky's notes: Deep Learning Theory Lecture Notes

• S. Shalev-Shwartz and S Ben-David, Understanding Machine Learning: From Theory to Algorithms, 2014.

• Roman Vershynin, High-dimensional probability: An introduction with applications in data science, 2018.