Logistic Regression is designed to deal with binary classification tasks. Unlike Linear Regression, it doesn’t use a direct linear mapping for classification but rather employs a sigmoid function, offering results in the range of 0 to 1.

Sigmoid Function

The sigmoid function is integral to logistic regression. It maps continuous input values (commonly called a linear combination of weights and features, or logits) to a probability range (0 to 1).

The sigmoid function, often denoted by $\sigma(z)$, is expressed as:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

where $z$ represents the linear combination:

$$z = w_0x_0 + w_1x_1 + ... + w_nx_n$$

• $x_n$ are the input features,
• $w_n$ are the co-efficients
• $w_0$ is the bias term.

Decision Boundary

Logistic Regression fits a decision boundary through the dataset. Data points are then classified based on which side they fall.

Using the sigmoid function, if $\sigma(z)$ is greater than 0.5, the predicted class is 1. Otherwise, it’s 0.

Probability & Thresholding

Logistic Regression provides a probability score for each class assignment. You can then use a threshold to declare the final class; a common threshold is 0.5.

Loss Functions

While linear regression uses the mean squared error (MSE) or mean absolute error (MAE) for optimization, logistic regression employs the logarithmic loss, also known as the binary cross-entropy loss:

$$\text{BCE}(y, \hat{y}) = -y\log(\hat{y}) - (1-y)\log(1-\hat{y})$$

where:

• $y$ is the actual class (0 or 1)
• $\hat{y}$ is the predicted probability by the model.

Complexity and Training

Logistic Regression involves the iteration of minimization techniques like gradient descent. This training often converges faster than linear regression’s training process.

Regularization

Both linear and logistic regressions can integrate regularization techniques (e.g. L1, L2) to avoid overfitting. If needed, the emphasis can be shifted from minimizing the loss function to regulating the coefficients, ultimately enhancing model generalization.