Bias and Variance are two key sources of error in machine learning models.

Bias

• Definition: Bias is the model’s tendency to consistently learn the wrong thing by failing to capture the underlying relationships between input and output data.
• Visual Representation: A model with high bias is like firing arrows that consistently miss the bullseye, though they might still be clustered together.
• Implications: High bias leads to underfitting, where the model is too simplistic and fails to capture the complexities in the training data. This results in poor accuracy both on the training and test datasets. The model fails to learn even when provided with enough data.
  • Example: A linear regression model applied to a non-linear relationship will exhibit high bias.
• Bias-Variance Tradeoff: Bias and variance are inversely related. Lowering bias often increases variance, and vice versa.

Variance

• Definition: Variance pertains to the model’s sensitivity to fluctuations in the training set. A model with high variance is overly responsive to small fluctuations in the training data, often learning noise as part of the patterns.
• Visual Representation: Think of a scattergun that fires haphazardly, hitting some data points precisely but straying far from the others.
• Implications: High variance leads to overfitting, where the model performs well on the training data but fails to generalize to unseen data. In other words, it captures the noise in the training data rather than the underlying patterns. Overfitting occurs when the model is too complex, often as a result of being trained on a small or noisy dataset.
  • Example: A decision tree with no depth limit is prone to high variance and overfitting.
• Bias-Variance Tradeoff: Adjusting a model to reduce variance often increases bias and vice versa. The goal is to find the optimal balance that minimizes the overall error, known as the irreducible error.

Code Example: Bias and Variance Tradeoff in Linear Regression

Here is the Python code:

# Import required libraries
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Set up data
np.random.seed(0)
X = np.linspace(0, 10, 100)
y = 2*X + np.random.normal(0, 1, 100)

# Instantiate models of varying complexity
models = {
    'Underfit': LinearRegression(),  # Normal Linear Regression
    'Optimal': LinearRegression(fit_intercept=False), # If there is no bias
    'Overfit': LinearRegression(copy_X=True) # Bias
}

# Train models and calculate error metrics
train_sizes, train_scores, test_scores = learning_curve(models[-1], X.reshape(-1, 1), y, cv=5)
train_errors, test_errors = [], []
for key, model in models.items():
    model.fit(X.reshape(-1, 1), y)
    train_pred, test_pred = model.predict(X.reshape(-1, 1)), model.predict(X.reshape(-1, 1))
    train_errors.append(mean_squared_error(y, train_pred))
    test_errors.append(mean_squared_error(y, test_pred))

# Visualize the data
fig, ax = plt.subplots(1, 1, figsize=(5,3))
ax.plot(train_sizes, train_scores, 'o-', color='r', label='Training Set')
ax.plot(train_sizes, test_scores, 'o-', color='g', label='Testing Set')
ax.set_xlabel('Model Complexity')
ax.set_ylabel('Performance')
ax.set_title('Learning Curve')
ax.legend()
plt.show()

# Print error metrics
print("Training Errors:\n", train_errors, '\n')
print("Testing Errors:\n", test_errors)

# I would recommend training the models with the proper parameters and if possible on a larger dataset for a more accurate result.