Python Tutorial
Pandas Cleaning Data
Empty cells, wrong types, and duplicate rows break analysis. Drop, fill, or convert them before you group or plot.
Find Missing Values
import pandas as pd
import numpy as np
df = pd.DataFrame({
"name": ["Luna", "Kai", "Mia", "Kai"],
"score": [88, np.nan, 95, 88],
"date": ["2024/03/01", "20240305", "2024-03-12", "2024/03/01"],
})
print(df.isna().sum())Drop or Fill
print(df.dropna()) # drop rows with any NA
print(df["score"].fillna(df["score"].mean()))Wrong Format
df["date"] = pd.to_datetime(df["date"], format="mixed")
print(df["date"])Duplicates
print(df.duplicated())
print(df.drop_duplicates())📘 Real-World Deep Dive
Knowing <strong>Pandas Cleaning (pandas)</strong> well is what turns pandas from a curiosity into a daily tool — you'll reach for it in nearly every real project.
Real-Life Scenario
An end-to-end usage of Pandas Cleaning that you'd actually see in a data pipeline or analytics notebook.
Real-Life Example
import pandas as pd
df = pd.DataFrame({"price": ["12.5", "-", "7", None]})
df["price"] = pd.to_numeric(df["price"], errors="coerce")
print(df)
print("mean:", df["price"].mean())Expected Output
(see source)Common mistakes
- A
DataFrameindexing pattern likedf[df.col > 5]returns a copy — use.loc[row_mask, col]for assignment to avoidSettingWithCopyWarning. - Pandas infers
objectdtype for CSVs with mixed numeric/text columns; cast withpd.to_numeric/astype("category")for big speed/memory wins. df.iterrows()is O(n) and slow; iterate withdf.itertuples()or vectorise column-wise.- Treating Pandas Cleaning as a black box without reading the docs — the API has subtle defaults that bite when you scale.
🚀 Performance & Best Practices
- Enable the Arrow backend:
pd.read_csv("…", engine="pyarrow", dtype_backend="pyarrow")for faster, type-stable reads. - Use
categoricaldtype for columns with low-cardinality strings — sort/join/group-by speed up dramatically. - Switching a hot loop from row-wise Python to
df.eval("…")/df.query("…")often gives 5–50×. - When working with pandas, prefer vectorised / batched operations over Python loops.
🧪 Try It Yourself
- Reproduce the snippet on a representative slice of your own data.
- Profile the snippet with
cProfileortimeitand find the single biggest improvement. - Generalise the snippet into a small, reusable function you can drop into future projects.