Model evaluation in machine learning is the process of determining how well a trained model generalizes to new, unseen data. It helps in selecting the best model for a task, assessing its performance against expectations, and identifying any issues such as overfitting or underfitting.

Quantifying Predictive Performance

Predictive performance is measured in classification tasks through metrics such as accuracy, precision, recall, F1 score, and area under the ROC curve (AUC-ROC). For regression, commonly used metrics include mean squared error (MSE), root mean squared error (RMSE), mean absolute error (MAE), and $R^2$ coefficient of determination.

Overfitting and Underfitting

Overfitting occurs when a model is excessively complex, performing well on training data but poorly on new, unseen data. This is typically identified when there is a significant difference between the performance on the training and test sets. Cross-validation can help offset this.

Underfitting results from a model that is too simple, performing poorly on both training and test data. It can be recognized if even model’s performance on the training set is not satisfactory.

Common Techniques in Model Evaluation

• Train-Test Split: Initially, the dataset is divided into separate training and testing sets. The model is trained on the former and then evaluated using the latter to approximate how the model will perform on new data.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

• k-Fold Cross-Validation: The dataset is divided into k folds (typically 5 or 10), and each fold is used as the test set k-1 times, with the rest as the training set. This is repeated k times, and the results are averaged. It’s more reliable than a single train-test split.

from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X, y, cv=5)

• Leave-One-Out Cross-Validation (LOOCV): A more extreme form of k-fold where k is set to the number of data points. It can be computationally expensive but is helpful when there are limited data points.

• Bootstrapping: A resampling technique where multiple datasets are constructed by sampling with replacement from the original dataset. The model is trained and tested on these bootstrapped datasets, and the average performance is taken as the overall performance estimate.

  Jackknife: A specific type of bootstrapping where each dataset is generated by removing one data point.