Replacing Marshmallow and Cerberus with Pydantic

The performance reality of the Rust rewrite

Pydantic v1 is not supported in Python 3.14. Teams using the latest Python releases must migrate to Pydantic v2 to ensure compatibility. The rewrite of pydantic-core in Rust provides speed improvements. For basic validator schemas, Pydantic v2.10.2 runs 6.7 times faster than v1.10.14. When you apply new v2 capabilities, the speedup over v1 reaches 14.3 times. The authors claim speed improvements of 5x to 50x compared to v1. The pydantic-core repository is now merged into the main Pydantic repository. The 2.13 beta release provides bug fixes and performance improvements for validation and serialization. FastAPI version 0.128.0 dropped support for the pydantic.v1 namespace, which previously helped with gradual migration. You can use the bump-pydantic tool to automate code changes. If you process 10,000 requests per second, Pydantic requires 6 CPU cores just for validation.

Breaking from Marshmallow and dataclasses

I see teams struggle when moving from Marshmallow to Pydantic because the libraries handle data differently. Marshmallow produces a cleaned dictionary after validation, whereas Pydantic returns a full Python object. While Pydantic handles OpenAPI and JSON schema generation natively, Marshmallow requires third-party tools like marshmallow-jsonschema to achieve similar results. You should use Pydantic at service boundaries to avoid excessive serialization debt. Since you already know the basics, use the bump-pydantic tool to automate code changes. I find that using Pydantic for every internal object creates heavy memory overhead and unnecessary computation.

If you require pure speed for object creation without needing any validation, Python dataclasses remain 6.5 times faster than Pydantic and use 2.5 times less memory. Both Marshmallow and Pydantic have approximately 5,000 GitHub stars. Marshmallow has 90 open issues, while Pydantic has 235 open issues. If you prefer dataclass behavior, the marshmallow-dataclass library can augment your classes to match Pydantic’s object-returning style. Marshmallow’s dump() method re-validates data, which can cause errors for manually created objects that contain invalid data. For pure serialization, packages like orjson or msgspec are faster. You can set Pydantic to use orjson to improve performance. Pydantic provides type safety throughout the entire lifetime of your objects. The geometric mean improvement for typical models is approximately 17x. This improvement comes from underlying optimizations at the Rust level.

Managing schema drift and parsing

Pydantic follows a philosophy of parsing rather than strict validation. It coerces data into the target type if a lossless interpretation exists. Pydantic turns a string "12345" into an integer without errors. This behavior helps engineers build an anti-corruption layer for web scraping data. Use AliasChoices to handle selector drift when a website changes a JSON key name. You can set extra=’forbid’ in the model configuration to detect schema explosion. Pydantic provides BeforeValidator, AfterValidator, and WrapValidator to clean data at the edge. You can use BeforeValidator to strip currency symbols from price strings before type checking. You can also use Annotated to create reusable types. The library uses pydantic-core to handle recursive traversal and type checking at the machine-code level. This allows the library to bypass the Python Global Interpreter Lock for significant portions of the workload. You can use FieldInfo within Annotated to provide metadata. Pydantic can also use create_model() to generate dynamic models. You can use the TypeAdapter to validate data against a type. Pydantic also provides the model_validate_json() method to handle JSON validation. Is the performance overhead of the Python wrapper acceptable for your specific high-throughput scraping needs?