Data Preprocessing for ML

Data preprocessing is the work of turning raw, messy data into clean, numeric, consistently scaled input that a machine learning model can use. It is rarely glamorous, but it usually decides whether a model works. Models cannot learn from gaps, text categories, or wildly different scales, so preparing the data well matters as much as the algorithm.

Why preprocessing matters

A machine learning algorithm only sees numbers. If your data has missing values, text labels, or features on very different scales, the model will either error out or learn something distorted. The common saying "garbage in, garbage out" applies directly: a strong algorithm on poorly prepared data loses to a simple algorithm on well-prepared data. Most real-world ML projects spend more time here than on the model itself.

Handling missing values

Real datasets have gaps. You have two broad choices: remove them or fill them.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "age": [25, np.nan, 30, 28],
    "income": [40000, 52000, np.nan, 48000],
})

# Option 1: drop rows with any missing value (simple, but loses data).
clean = df.dropna()

# Option 2: fill missing values with the column average (keeps all rows).
filled = df.fillna(df.mean(numeric_only=True))
print(filled)

Filling with the average is a reasonable default for numbers, but the right choice depends on the data. The key rule: be deliberate, and apply the same logic to new data later.

Encoding categorical data

Models need numbers, so text categories must be converted. For unordered categories like city or colour, one-hot encoding is standard — it creates a separate 0/1 column for each category, avoiding any false ordering.

import pandas as pd

df = pd.DataFrame({"city": ["Pune", "Jalgaon", "Pune", "Nashik"]})

# One-hot encode: each city becomes its own 0/1 column.
encoded = pd.get_dummies(df, columns=["city"])
print(encoded.columns.tolist())
# ['city_Jalgaon', 'city_Nashik', 'city_Pune']

Avoid simply numbering categories (Pune=1, Jalgaon=2) for unordered data, because the model would wrongly treat the numbers as a ranking.

Scaling features

Features often live on very different scales — age in tens, income in tens of thousands. Distance- and gradient-based models can be dominated by the larger numbers. Scaling puts features on a comparable range. A common method is standardisation: rescale each feature to have mean 0 and standard deviation 1.

from sklearn.preprocessing import StandardScaler
import numpy as np

X_train = np.array([[25, 40000], [30, 52000], [28, 48000]])
X_test = np.array([[27, 50000]])

scaler = StandardScaler()
# Fit the scaler on TRAINING data only, then apply it to both sets.
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

print(X_train_scaled.round(2))

Notice the scaler is fit on training data only, then used to transform the test data. Fitting on the test data too would leak information and inflate your results.

Splitting the data

Before training, hold back part of the data as a test set the model never sees. This is how you measure honest performance.

from sklearn.model_selection import train_test_split
import numpy as np

X = np.arange(20).reshape(10, 2)   # 10 examples, 2 features
y = np.array([0, 1] * 5)           # labels

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=0
)
print(X_train.shape, X_test.shape)   # (8, 2) (2, 2)

Common mistakes

• Fitting scalers or encoders on the whole dataset. Always fit on training data only, then transform the test data. Otherwise information leaks and your scores look better than reality.
• Number-coding unordered categories. Use one-hot encoding for categories with no natural order, or the model invents a ranking.
• Forgetting to apply the same steps to new data. Whatever you do in training must be repeated, identically, when the model is used in production.
• Cleaning before splitting in a way that leaks. Decisions that use the whole dataset (like a fill value or a scaler) should be learned from the training split only.

FAQ

Is preprocessing different for every model? Somewhat. Tree-based models care less about scaling; distance- and gradient-based models care a lot. But cleaning and encoding matter for almost all.

How much of an ML project is preprocessing? Often the majority. Loading, cleaning, and shaping data with Pandas typically takes more time than building the model.

What is data leakage? Letting information from the test set (or the future) influence training. It produces falsely high scores that collapse on real data.

Keep learning

Explore the full AI & machine learning hub for more guides, or move on to build your first ML model. To learn this properly, join the waitlist for our Artificial Intelligence course at Infoplanet, Jalgaon, with mentor support.