Python Tutorial
Pandas Merging DataFrames
Combine tables like SQL joins: merge on keys, or concat rows and columns.
merge
import pandas as pd
left = pd.DataFrame({"id": [1, 2, 3], "name": ["Luna", "Kai", "Mia"]})
right = pd.DataFrame({"id": [1, 2, 4], "score": [88, 92, 70]})
print(pd.merge(left, right, on="id", how="inner"))
print(pd.merge(left, right, on="id", how="left"))how can be inner, left, right, or outer.
concat
a = pd.DataFrame({"n": [1, 2]})
b = pd.DataFrame({"n": [3, 4]})
print(pd.concat([a, b], ignore_index=True))📘 Real-World Deep Dive
Knowing <strong>Pandas Merge (pandas)</strong> well is what turns pandas from a curiosity into a daily tool — you'll reach for it in nearly every real project.
Real-Life Scenario
An end-to-end usage of Pandas Merge that you'd actually see in a data pipeline or analytics notebook.
Real-Life Example
import pandas as pd
users = pd.DataFrame({"id": [1, 2, 3], "name": ["a", "b", "c"]})
orders = pd.DataFrame({"user_id": [1, 1, 2], "amount": [10, 5, 7]})
joined = users.merge(orders, left_on="id", right_on="user_id")
print(joined)Expected Output
(see source)Common mistakes
- A
DataFrameindexing pattern likedf[df.col > 5]returns a copy — use.loc[row_mask, col]for assignment to avoidSettingWithCopyWarning. - Pandas infers
objectdtype for CSVs with mixed numeric/text columns; cast withpd.to_numeric/astype("category")for big speed/memory wins. df.iterrows()is O(n) and slow; iterate withdf.itertuples()or vectorise column-wise.- Treating Pandas Merge as a black box without reading the docs — the API has subtle defaults that bite when you scale.
🚀 Performance & Best Practices
- Enable the Arrow backend:
pd.read_csv("…", engine="pyarrow", dtype_backend="pyarrow")for faster, type-stable reads. - Use
categoricaldtype for columns with low-cardinality strings — sort/join/group-by speed up dramatically. - Switching a hot loop from row-wise Python to
df.eval("…")/df.query("…")often gives 5–50×. - When working with pandas, prefer vectorised / batched operations over Python loops.
🧪 Try It Yourself
- Reproduce the snippet on a representative slice of your own data.
- Profile the snippet with
cProfileortimeitand find the single biggest improvement. - Generalise the snippet into a small, reusable function you can drop into future projects.