Feature engineering is an essential part of building robust machine learning models. It involves selecting and transforming input variables (features) to maximize a model’s predictive power.

Feature Engineering: The Power Lever for Models

1. Dimensionality Reduction: High-performing models often work better with fewer, more impactful features. Techniques like PCA (Principal Component Analysis) and t-SNE (t-distributed Stochastic Neighbor Embedding) help visualize and choose top features.

2. Feature Standardization and Scaling: Data imbalance, where some features may have much wider ranges than others, can cause models like k-NN to favor certain features. Techniques like z-score standardization or min-max scaling ensure equal feature representation.

3. Feature Selection: Some features might not contribute significantly to the model’s predictive power. Tools like correlation matrices, forward/backward selection, or specialized algorithms like LASSO and Elastic Net can help choose the most affective ones.

4. Polynomial Features: Sometimes, the nature of a relationship between a feature and a target variable is not linear. Codifying this as powers of features (like $x^2$ or $x^3$ ), can make the model more flexible.

5. Feature Crosses: In some cases, the relationship between features and the target is more nuanced when certain feature combinations are considered. Polynomial Features creates such combinations, enhancing the model’s performance.

6. Feature Embeddings: Raw data could have too many unique categories (like user or country names). Feature embeddings condense this data into vectors of lower dimensions. This simplifies categorical data representation.

7. Missing Values Handling: Many algorithms can’t handle missing values. Techniques for imputing missing values such as using the mean, median, or most frequent value, or even predicting the missing values, are important for model integrity.

8. Feature Normality: Some algorithms, including linear and logistic regression, expect features to be normally distributed. Data transformation techniques like the Box-Cox and Yeo-Johnson transforms ensure this conformance.

9. Temporal Features: For datasets with time-dependent relationships, features like this season’s sale figures can improve prediction.

10. Text and Image Features: Dealing with non-numeric data, such as natural language or images, often requires specialized pre-processing techniques before these features can be generically processed. Techniques like word embeddings or TF-IDF enable machine learning models to work with text data, while convolutional neural networks (CNNs) are used for image feature extraction.

11. Categorical Feature Handling: Features with non-numeric representations, such as “red”, “green”, and “blue” in items’ colors, might need to be converted to a numeric format (often via “one-hot encoding”) before input to a model.

Code Example: Feature Engineering Steps

Here is the Python code:

from sklearn import datasets
import pandas as pd

# Load the iris dataset
iris = datasets.load_iris()
iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)

# Perform feature selection
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
X_new = SelectKBest(chi2, k=2).fit_transform(iris_df, iris.target)

# Create interaction terms using PolynomialFeatures
from sklearn.preprocessing import PolynomialFeatures
interaction = PolynomialFeatures(degree=2, include_bias=False, interaction_only=True)
X_interact= interaction.fit_transform(iris_df)

# Normalization with MinMaxScaler
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
X_normalized = scaler.fit_transform(iris_df)

# Categorical feature encoding using One-Hot Encoding
from sklearn.preprocessing import OneHotEncoder
ohe = OneHotEncoder(sparse=False)
iris_encoded = ohe.fit_transform(iris_df[['species']])

# Show results
print("Selected Features after Chi2: \n", X_new)
print("Interaction Features using PolynomialFeatures: \n", X_interact)