Cost functions in machine learning quantify how well a model fits the data and are used for optimization. By minimizing this function, a model aims to provide accurate predictions.

Core Concepts

• True Output: Represented as $y$, it’s the actual target from the dataset.
• Predicted Output: denoted as $\hat{y}$, it’s the output produced by the model.
• Loss: The discrepancy between the true and predicted outputs for a single data point; often denoted as $L(y, \hat{y})$.
• Cost: The average of losses across all the training data. Often the mean squared error (MSE) or binary cross-entropy is used in the context of regression and classification tasks, respectively.

Common Cost Functions

1. Mean Squared Error (MSE): Ideal for Regression Problems

   $$MSE = \frac{1}{n}\sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

   This function squares the errors which gives higher weight to larger errors.

2. Cross-Entropy: Tailored for Classification Tasks

   $$CE = -\frac{1}{n}\sum_{i=1}^{n} \left(y_i \log(\hat{y}_i) + (1 -y_i) \log(1 - \hat{y}_i) \right)$$

3. Hinge Loss: Common in SVMs for Classification

$$HL = \max(0, 1 - y\hat{y})$$

For multi-class problems, you would typically use the softmax function in the output layer and then employ the cross-entropy function for the cost.

An important concept to note is that different models might work better with different cost functions. For example, linear regression models often work well with the MSE, while logistic regression models are better suited for classification tasks and, hence, prefer the cross-entropy loss.