https://www.di.ens.fr/~fbach/learning_theory_class/index.html

• Estimation error
• Approximation properties and universality 特别主题 Special topics -Generalization/optimization properties of infinitely wide neural networks -Double descent

### 相关内容

【导读】本文档包含加州大学伯克利分校机器学习Jonathan Shewchuk入门课程的课堂讲稿。它涵盖了许多分类和回归的方法，以及聚类和降维的方法。简洁明了，是非常合适的机器学习入门学习材料。

We give polynomial-time approximation schemes for monotone maximization problems expressible in terms of distances (up to a fixed upper bound) and efficiently solvable in graphs of bounded treewidth. These schemes apply in all fractionally treewidth-fragile graph classes, a property that is true for many natural graph classes with sublinear separators. We also provide quasipolynomial-time approximation schemes for these problems in all classes with sublinear separators.

Stochastic gradient descent (SGD) is one of the most popular algorithms in modern machine learning. The noise encountered in these applications is different from that in many theoretical analyses of stochastic gradient algorithms. In this article, we discuss some of the common properties of energy landscapes and stochastic noise encountered in machine learning problems, and how they affect SGD-based optimization. In particular, we show that the learning rate in SGD with machine learning noise can be chosen to be small, but uniformly positive for all times if the energy landscape resembles that of overparametrized deep learning problems. If the objective function satisfies a Lojasiewicz inequality, SGD converges to the global minimum exponentially fast, and even for functions which may have local minima, we establish almost sure convergence to the global minimum at an exponential rate from any finite energy initialization. The assumptions that we make in this result concern the behavior where the objective function is either small or large and the nature of the gradient noise, but the energy landscape is fairly unconstrained on the domain where the objective function takes values in an intermediate regime.

Python机器学习教程展示了通过关注两个核心机器学习算法家族来成功分析数据，本书能够提供工作机制的完整描述，以及使用特定的、可破解的代码来说明机制的示例。算法用简单的术语解释，没有复杂的数学，并使用Python应用，指导算法选择，数据准备，并在实践中使用训练过的模型。您将学习一套核心的Python编程技术，各种构建预测模型的方法，以及如何测量每个模型的性能，以确保使用正确的模型。关于线性回归和集成方法的章节深入研究了每种算法，你可以使用书中的示例代码来开发你自己的数据分析解决方案。

1. 监督学习，非监督学习，强化学习。

2. 机器学习泛化能力

3. 支持向量机，核机

4. 神经网络和深度学习

https://algorithmsbook.com/

Introduction

• PART I: PROBABILISTIC REASONING Representation
• PART II: SEQUENTIAL PROBLEMS Exact Solution Methods
• PART III: MODEL UNCERTAINTY Exploration and Exploitation
• PART V: MULTIAGENT SYSTEMS Multiagent Reasoning

1. 无线数据学习 Learning with infinite data (population setting)
• Decision theory (loss, risk, optimal predictors)
• Decomposition of excess risk into approximation and estimation errors
• No free lunch theorems
• Basic notions of concentration inequalities (MacDiarmid, Hoeffding, Bernstein)
1. 线性最小二乘回归 Linear least-squares regression
• Guarantees in the fixed design settings (simple in closed form)
• Guarantees in the random design settings
• Ridge regression: dimension independent bounds
1. 经典风险分解 Classical risk decomposition
• Approximation error
• Convex surrogates
• Estimation error through covering numbers (basic example of ellipsoids)
• Modern tools (no proof): Rademacher complexity, Gaussian complexity (+ Slepian/Lipschitz)
• Minimax rates (at least one proof)
1. 机器学习优化 Optimization for machine learning
• Generalization bounds through stochastic gradient descent
1. 局部平均技术 Local averaging techniques
• Kernel density estimation
• Nadaraya-Watson estimators (simplest proof to be found with apparent curse of dimensionality)
• K-nearest-neighbors
• Decision trees and associated methods
1. 核方法 Kernel methods
• Modern analysis of non-parametric techniques (simplest proof with results depending on s and d
1. 模型选择 Model selection
• L0 penalty with AIC
• L1 penalty
• High-dimensional estimation
1. 神经方法 Neural networks
• Approximation properties (simplest approximation result)
• Two layers
• Deep networks
1. 特别话题 Special topics
• Generalization/optimization properties of infinitely wide neural networks
• Double descent

【导读】UC.Berkeley CS189 《Introduction to Machine Learning》是面向初学者的机器学习课程在本指南中，我们创建了一个全面的课程指南，以便与学生和公众分享我们的知识，并希望吸引其他大学的学生对伯克利的机器学习课程感兴趣。

• Note 1: Introduction

• Note 2: Linear Regression

• Note 3: Features, Hyperparameters, Validation

• Note 4: MLE and MAP for Regression (Part I)

• Note 6: Multivariate Gaussians

• Note 7: MLE and MAP for Regression (Part II)

• Note 8: Kernels, Kernel Ridge Regression

• Note 9: Total Least Squares

• Note 10: Principal Component Analysis (PCA)

• Note 11: Canonical Correlation Analysis (CCA)

• Note 12: Nonlinear Least Squares, Optimization

• Note 13: Gradient Descent Extensions

• Note 14: Neural Networks

• Note 15: Training Neural Networks

• Note 16: Discriminative vs. Generative Classification, LS-SVM

• Note 17: Logistic Regression

• Note 18: Gaussian Discriminant Analysis

• Note 19: Expectation-Maximization (EM) Algorithm, k-means Clustering

• Note 20: Support Vector Machines (SVM)

• Note 21: Generalization and Stability

• Note 22: Duality

• Note 23: Nearest Neighbor Classification

• Note 24: Sparsity

• Note 25: Decision Trees and Random Forests

• Note 26: Boosting

• Note 27: Convolutional Neural Networks (CNN)

• Discussion 0: Vector Calculus, Linear Algebra (solution)

• Discussion 1: Optimization, Least Squares, and Convexity (solution)

• Discussion 2: Ridge Regression and Multivariate Gaussians (solution)

• Discussion 3: Multivariate Gaussians and Kernels (solution)

• Discussion 4: Principal Component Analysis (solution)

• Discussion 5: Least Squares and Kernels (solution)

• Discussion 6: Optimization and Reviewing Linear Methods (solution)

• Discussion 7: Backpropagation and Computation Graphs (solution)

• Discussion 8: QDA and Logistic Regression (solution)

• Discussion 9: EM (solution)

• Discussion 10: SVMs and KNN (solution)

• Discussion 11: Decision Trees (solution)

• Discussion 12: LASSO, Sparsity, Feature Selection, Auto-ML (solution)

65+阅读 · 2月25日

31+阅读 · 2月7日

32+阅读 · 2020年12月14日

84+阅读 · 2020年6月27日

98+阅读 · 2020年1月16日

Alain Desgagné,Pierre Lafaye de Micheaux,Frédéric Ouimet
0+阅读 · 5月10日
Yiming Xu,Akil Narayan
0+阅读 · 5月10日
Sho Takemori,Masahiro Sato
0+阅读 · 5月10日
Alexander Barg,Peter Boyvalenkov,Maya Stoyanova
0+阅读 · 5月7日
Diego Cifuentes,Ankur Moitra
0+阅读 · 5月7日
Brian Brubach,Nathaniel Grammel,David G. Harris,Aravind Srinivasan,Leonidas Tsepenekas,Anil Vullikanti
0+阅读 · 5月6日