Data Cleaning & Wrangling

Data cleaning is the process of fixing problems in raw data so it can be analysed reliably — handling missing values, removing duplicates, correcting wrong data types, and standardising messy text. It is the least glamorous part of data work but often the most important: analysis on dirty data produces confident, wrong answers.

Why cleaning matters so much

Real data is rarely tidy. A single spreadsheet might contain blank cells, the same customer spelled three ways, dates stored as text, and a stray total row. Working data scientists routinely report that preparing data takes the bulk of their time. Getting good at it is one of the highest-leverage skills you can build.

The common problems and their fixes

Missing values

Decide whether to drop the rows, drop the columns, or fill the gaps.

import pandas as pd  # pandas 2.x

df = pd.DataFrame({
    "name": ["Aarav", "Diya", "Kabir"],
    "age":  [21, None, 23],
})

# Count missing values per column first
print(df.isna().sum())

# Option A: fill missing ages with the column median
df["age"] = df["age"].fillna(df["age"].median())

# Option B (if appropriate): drop rows that still have missing values
# df = df.dropna()
print(df)

Duplicates

Repeated rows inflate counts and skew averages.

import pandas as pd

df = pd.DataFrame({"id": [1, 2, 2, 3], "value": [10, 20, 20, 30]})

# Remove exact duplicate rows, keeping the first occurrence
df = df.drop_duplicates()
print(df)

Wrong data types

Numbers or dates stored as text break calculations.

import pandas as pd

df = pd.DataFrame({"price": ["100", "250", "75"]})

Convert text to numbers; errors="coerce" turns bad values into NaN

df["price"] = pd.to_numeric(df["price"], errors="coerce") print(df.dtypes)

### Inconsistent text
"Jalgaon", "jalgaon ", and "JALGAON" should be treated as one value.

```python
import pandas as pd

df = pd.DataFrame({"city": [" Jalgaon", "PUNE ", "jalgaon"]})

# Strip spaces and standardise case so values match
df["city"] = df["city"].str.strip().str.title()
print(df["city"].unique())   # ['Jalgaon' 'Pune']

A repeatable cleaning checklist

1. Inspect with df.head(), df.info(), and df.describe().
2. Count missing values with df.isna().sum().
3. Remove or fill missing data deliberately.
4. Drop duplicates.
5. Fix data types.
6. Standardise text and category labels.
7. Re-check before analysing.

Once clean, your data is ready for exploratory data analysis.

Common mistakes

• Using df.append() to stack cleaned chunks. It was removed from modern pandas. Use pd.concat([part1, part2], ignore_index=True).
• Filling missing values blindly with zero. Zero is a real number and can distort your results; choose the median, mean, or a flag deliberately.
• Cleaning without inspecting first. Always look at the data before transforming it.
• Forgetting to recheck. Verify the fix actually worked before moving on.

FAQ

Should I always drop missing values? No. Dropping loses information. Filling sensibly is often better — it depends on the column and the question.

How do I know if data is "clean enough"? When the types are correct, categories are consistent, and obvious errors are gone. Perfection is not the goal; reliability is.

Is cleaning the same as wrangling? They overlap. Wrangling is the broader reshaping of data into a usable form; cleaning is the error-fixing part of it.

Keep learning

Cleaning is where most of the real work lives, so it is worth mastering early. Build on it with pandas for analytics and the full Data Science & Analytics hub.

Want hands-on practice with messy, real datasets? Join the waitlist for the Data Science & Analytics course at Infoplanet, Jalgaon.