As an engineer, you should always be validating your data at the boundaries to make sure it fits your internal data structures/types.
While you could use Pydantic to validate your dataframes, it doesn’t offer this functionality out of the box. Suppose we had a simple ETL script with a few assert statements:
# daily_revenue.py — runs every morning on the orders export
df = pd.read_csv("orders_2026_07_22.csv")
assert not df["order_id"].isnull().any()
assert (df["quantity"] >= 0).all()
df["revenue"] = df["price"] * df["quantity"]
report = df.groupby("region")["revenue"].sum()Let’s suppose that your CSV starts reporting quantity as a string instead of a int. Here, we’ll hit the assert statement and it won’t give us any information as to what’s going on.
Imagine if there wasn’t an assert statement - we’d see a NaN for the report since price * quantity produces this.
As mentioned before, we could use Pydantic to validate our data, but there’s some serious drawbacks with using this library for Pandas DataFrames.
Instead, you may want to check out Pandera (GitHub repo) instead. You’ll define a declaritive schema once: the expected columns, data types, and constraints in a Pydantic-style syntax.
The difference between Pydantic and Pandera
The core distinction between Pydantic and Pandera is record oriented vs frame-oriented. Pydantic validates one object at a time: you give it a dictionary (or JSON) and it checks that object against a model.
Pandera treats the DataFrame as a columnar structure and validates down the columns in a vectorized way. With Pydantic you could validate it like this:
from pydantic import BaseModel, Field, TypeAdapter
class Order(BaseModel):
order_id: str
region: str = Field(pattern="^(NA|EU|APAC)$")
price: float = Field(ge=0)
quantity: int = Field(ge=0)
records = df.to_dict(orient="records")
validated = TypeAdapter(list[Order]).validate_python(records)And while this is valid, there’s a performance issue. to_dict(orient=”records”) materalizes every row as a Python dictionary and loops in pure Python, so vectorization gets thrown out (and that’s kind of the point of Pandas to begin with, right?).
An example of Pandera
If we were to move away from Pydantic and implement Pandera for our example, our model would look something like this:
import pandera.pandas as pa
from pandera.typing import Series
class Orders(pa.DataFrameModel):
order_id: Series[str] = pa.Field(
nullable=False, unique=True
)
region: Series[str] = pa.Field(isin=["NA", "EU", "APAC"])
price: Series[float] = pa.Field(ge=0)
quantity: Series[int] = pa.Field(ge=0, coerce=True)
# Unexpected columns are flagged
class Config:
strict = TrueWe’re defining our columns under the Orders class and define each field’s data type (str, float, etc), then define the conditions with pa.Field.
So, if we’re trying validate our data, instead of turning it into Python-native data types and running a checker there, we can keep leveraging the power of vectorization:
df = pd.read_csv("orders_2026_07_22.csv")
try:
Orders.validate(df, lazy=True)
except pa.errors.SchemaErrors as exc:
# a DataFrame: column, check, failed value, row index
print(exc.failure_cases)
# ... continue with your logic » Note: lazy=True is included in the validate() method so that it shows every error, not just the first one.
Happy coding!
📧 Join the Python Snacks Newsletter! 🐍
Want even more Python-related content that’s useful? Here’s 3 reasons why you should subscribe the Python Snacks newsletter:
Get Ahead in Python with bite-sized Python tips and tricks delivered straight to your inbox, like the one above.
Exclusive Subscriber Perks: Receive a curated selection of up to 6 high-impact Python resources, tips, and exclusive insights with each email.
Get Smarter with Python in under 5 minutes. Your next Python breakthrough could just an email away.
You can unsubscribe at any time.
Interested in starting a newsletter or a blog?
Do you have a wealth of knowledge and insights to share with the world? Starting your own newsletter or blog is an excellent way to establish yourself as an authority in your field, connect with a like-minded community, and open up new opportunities.
If TikTok, Twitter, Facebook, or other social media platforms were to get banned, you’d lose all your followers. This is why you should start a newsletter: you own your audience.
This article may contain affiliate links. Affiliate links come at no cost to you and support the costs of this blog. Should you purchase a product/service from an affiliate link, it will come at no additional cost to you.
