A measure of the discrepancy between the machine’s prediction and the actual output. The Problem of Generalization
A source of data that produces random vectors, usually assumed to be independent and identically distributed (i.i.d.). The Nature of Statistical Learning Theory
At its heart, the nature of statistical learning is defined by four essential components: A measure of the discrepancy between the machine’s
The most famous practical outcome of this theory is the Support Vector Machine (SVM). Rather than just minimizing training error, SVMs are designed to maximize the "margin" between classes. This approach directly implements the theoretical findings of SLT, ensuring that the chosen model has the best possible guarantee of generalizing to new information. Rather than just minimizing training error, SVMs are
The "nature" of this field is essentially the study of the gap between these two. If a model is too simple, it fails to capture the data's structure (underfitting). If it is too complex, it "memorizes" the noise in the training set (overfitting), leading to low empirical risk but high expected risk. Capacity and the VC Dimension