Testing with Pydantic

Note

To ensure you are using compatible versions, install with the testfixtures[pydantic] extra.

When pydantic is installed, a comparer for BaseModel is automatically registered with ignore_eq=True. It compares models field-by-field using their declared attributes, so differences are shown clearly:

>>> from pydantic import BaseModel
>>> from testfixtures import compare
>>> class Point(BaseModel):
...     x: int
...     y: int
>>> compare(Point(x=1, y=2), expected=Point(x=1, y=3))
Traceback (most recent call last):
 ...
AssertionError: Point not as expected:

attributes same:
['x']

attributes differ:
'y': 3 (expected) != 2 (actual)

The ignore_eq=True registration is also needed whenever a model contains attributes whose type has a custom __eq__ that compare() has a registered comparer for, such as Polars or Pandas DataFrames. Without it, pydantic’s __eq__ calls == on each attribute value before testfixtures can intercept it, which for many such types raises an error or gives a misleading result.

For example, consider a model with a DataFrame attribute:

import polars as pl
from pydantic import BaseModel, ConfigDict

class Report(BaseModel):
    model_config = ConfigDict(arbitrary_types_allowed=True)
    name: str
    data: pl.DataFrame

r1 = Report(name='sales', data=pl.DataFrame({'x': [1, 2], 'y': [3, 4]}))
r2 = Report(name='sales', data=pl.DataFrame({'x': [1, 2], 'y': [3, 5]}))

Without the BaseModel registration, pydantic’s __eq__ fires first and raises a TypeError:

>>> compare(r1, expected=r2)
Traceback (most recent call last):
 ...
TypeError: the truth value of a DataFrame is ambiguous

Hint: to check if a DataFrame contains any values, use `is_empty()`.

With the registration in place, compare() hands off to the Polars comparer and produces a clear diff:

>>> compare(r1, expected=r2)
Traceback (most recent call last):
 ...
AssertionError: Report not as expected:

attributes same:
['name']

attributes differ:
'data': shape: (2, 2)
┌─────┬─────┐
│ x   ┆ y   │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1   ┆ 3   │
│ 2   ┆ 5   │
└─────┴─────┘ (expected) != shape: (2, 2)
┌─────┬─────┐
│ x   ┆ y   │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1   ┆ 3   │
│ 2   ┆ 4   │
└─────┴─────┘ (actual)

While comparing .data: DataFrames are different (value mismatch for column "y")
[left]: shape: (2,)
Series: 'y' [i64]
[
        3
        5
]
[right]: shape: (2,)
Series: 'y' [i64]
[
        3
        4
]

Testing validation errors

When a BaseModel is given invalid data, pydantic raises pydantic.ValidationError. This type has no public constructor that takes a plain message, so building an instance to hand to ShouldRaise for a full comparison is impractical.

Even setting aside how hard it is to construct one, comparing full instances would not catch anything useful anyway. Pydantic stores the details of what went wrong outside the args and __dict__ that compare() inspects for exceptions, so any two naturally raised pydantic.ValidationError instances compare equal regardless of what actually failed:

from pydantic import ValidationError

try:
    Point(x='not-an-int', y=2)
except ValidationError as e:
    exception_1 = e

try:
    Point(x=2, y='also-not-an-int')
except ValidationError as e:
    exception_2 = e

>>> compare(exception_1, exception_2)

The way to check a raised pydantic.ValidationError is therefore to check its str rendering with str_like(). match is a regular expression, so literal square brackets in pydantic’s rendering need escaping:

>>> from testfixtures import ShouldRaise, str_like
>>> with ShouldRaise(
...     str_like(
...         ValidationError,
...         match=(
...             "Input should be a valid integer, "
...             "unable to parse string as an integer "
...             r"\[type=int_parsing, input_value='not-an-int', input_type=str\]"
...         )
...     )
... ):
...     Point(x='not-an-int', y=2)

A short match pattern is easy to write but easy to get wrong too: it only has to be found somewhere in the rendering, so it can still pass even when a part of the message you care about isn’t what you expect. Where that matters, compare the whole rendering instead:

with ShouldRaise(
    str_like(
        ValidationError,
        "1 validation error for Point\n"
        "x\n"
        "  Input should be a valid integer, unable to parse string as an integer "
        "[type=int_parsing, input_value='not-an-int', input_type=str]\n"
        "    For further information visit https://errors.pydantic.dev/2.13/v/int_parsing"
    )
):
    Point(x='not-an-int', y=2)

The URL on the last line of the rendering is specific to the installed version of pydantic, so expect to update assertions like this when upgrading.