mashumaro

repository·master·Indexed 21 days ago

https://github.com/fatal1ty/mashumaro

A fast and highly customizable Python serialization library (version 3.22) for converting complex Python objects, such as dataclasses and generic types, to and from JSON, YAML, TOML, and MessagePack. It provides high performance by generating specific decoders and encoders for given schemas via two primary approaches: Codecs for arbitrary types and Mixins for dataclass models.

Tokens
25.6K
Snippets
75
Records
92
Agent score
72%

What's inside mashumaro

  1. Use Dialects to handle different data formats

    master

    A Dialect allows you to define specific serialization and deserialization behaviors (like date formats or field naming) for a single dataclass without creating multiple dataclass definitions. This is useful when interacting with different APIs that require different data representations.

    To use a dialect:

    1. Create a class inheriting from Dialect.
    2. Override options like serialization_strategy, serialize_by_alias, omit_none, omit_default, namedtuple_as_dict, or no_copy_collections.
    3. Pass the dialect to to_dict(dialect=...) or from_dict(..., dialect=...).
    4. Alternatively, set a dialect in your BaseConfig to make it the default for that dataclass.

    Note: To use dialects, you must include ADD_DIALECT_SUPPORT in your code_generation_options within your config.

    from dataclasses import dataclass
    from datetime import date, datetime
    from mashumaro import DataClassDictMixin
    from mashumaro.config import ADD_DIALECT_SUPPORT
    from mashumaro.dialect import Dialect
    from mashumaro.types import SerializationStrategy
    
    class DateTimeSerializationStrategy(SerializationStrategy):
        def __init__(self, fmt: str):
            self.fmt = fmt
    
        def serialize(self, value: date) -> str:
            return value.strftime(self.fmt)
    
        def deserialize(self, value: str) -> date:
            return datetime.strptime(value, self.fmt).date()
    
    class EthiopianDialect(Dialect):
        serialization_strategy = {
            date: DateTimeSerializationStrategy("%d/%m/%Y")
        }
    
    class JapaneseDialect(Dialect):
        serialization_strategy = {
            date: DateTimeSerializationStrategy("%Y年%m月%d日")
        }
    
    @dataclass
    class Entity(DataClassDictMixin):
        dt: date
    
    class Config:
        code_generation_options = [ADD_DIALECT_SUPPORT]
    
    entity = Entity(date(2021, 12, 31))
    entity.to_dict(dialect=EthiopianDialect)  # {'dt': '31/12/2021'}
    entity.to_dict(dialect=JapaneseDialect)   # {'dt': '2021年12月31日'}
    Entity.from_dict({'dt': '2021年12月31日'}, dialect=JapaneseDialect)
  2. Extend JSON Schema generation with plugins

    master

    You can customize schema generation or add support for new types by implementing the BasePlugin class and overriding its get_schema method. Plugins can be passed to build_json_schema or JSONSchemaBuilder via the plugins argument.

    Multiple plugins can be applied sequentially to transform the schema in steps. If a plugin's get_schema returns None or raises NotImplementedError, the system moves to the next plugin.

    from mashumaro.jsonschema import build_json_schema
    from mashumaro.jsonschema.plugins import DocstringDescriptionPlugin
    
    # Using a built-in plugin to add descriptions from docstrings
    schema = build_json_schema(MyClass, plugins=[DocstringDescriptionPlugin()])
  3. Use `Discriminator` for Union and Hierarchy deserialization

    master

    The Discriminator class allows you to customize how a Union of dataclasses or a class hierarchy is deserialized by using a specific field (a "tag") to identify the correct variant.

    Parameters

    • field: The name of the input dictionary key used to distinguish variants.
    • include_subtypes: If True, allows deserializing subclasses.
    • include_supertypes: If True, allows deserializing superclasses.
    • variant_tagger_fn: A custom function (cls) -> Union[str, int, list] used to generate tag values from a class.

    Tagging Methods

    1. Class-level attribute: Define an attribute (e.g., type = "connected") in the subclass. This works with plain values, ClassVar, Final, Literal, or StrEnum.
    2. Custom Tagger: Use variant_tagger_fn to generate tags dynamically (e.g., lambda cls: cls.__name__).

    Example: Subclasses distinguishable by a field

    To use a discriminator, wrap the type in Annotated with a Discriminator instance. The discriminator field must be accessible from the __dict__ of the descendant class.

    from dataclasses import dataclass
    from ipaddress import IPv4Address
    from mashumaro import DataClassDictMixin
    
    @dataclass
    class ClientEvent(DataClassDictMixin):
        pass
    
    @dataclass
    class ClientConnectedEvent(ClientEvent):
        type = "connected"
        client_ip: IPv4Address
    
    @dataclass
    class ClientDisconnectedEvent(ClientEvent):
        type = "disconnected"
        client_ip: IPv4Address
  4. Use Discriminator with Union types

    master

    For better performance when deserializing a Union of types, use Discriminator. This prevents Mashumaro from attempting to deserialize every type in the Union sequentially. It is particularly useful when classes do not share a common superclass or when you only want to include a subset of possible subclasses via include_supertypes=True.

    @dataclass
    class Message(DataClassDictMixin):
        event: Annotated[
            Union[Event1, Event2],
            Discriminator(field="code", include_supertypes=True),
        ]
  5. Fallback to base class using include_supertypes

    master

    When using Discriminator, you can set include_supertypes=True to allow the deserializer to fall back to the base class if no suitable subclass matches the provided data. This is useful for handling unknown or generic types within a list of specialized objects.

    @dataclass
    class Plate(DataClassDictMixin):
        ingredients: List[
            Annotated[
                Ingredient,
                Discriminator(include_subtypes=True, include_supertypes=True),
            ]
        ]
  6. Use serialization hooks for lifecycle management

    master

    Mashumaro provides several hooks to perform actions during the serialization/deserialization lifecycle:

    Deserialization Hooks

    • __pre_deserialize__(cls, d: Dict[Any, Any]) -> Dict[Any, Any]: A class method called before deserialization. Use it to transform the input dictionary (e.g., normalizing keys).
    • __post_deserialize__(cls, obj: 'A') -> 'A': A class method called after an instance is created. Use it to modify the instance.

    Serialization Hooks

    • __pre_serialize__(self) -> 'A': An instance method called before serialization. Use it to modify the instance state.
    • __post_serialize__(self, d: Dict[Any, Any]) -> Dict[Any, Any]: An instance method called after the dictionary is created. Use it to modify the resulting dictionary (e.g., removing sensitive fields).
  7. Deserialize subclasses using a common field with Discriminator

    master

    When you have a hierarchy of classes where subclasses can be distinguished by a specific field value (e.g., a type field), use Annotated with Discriminator(field="...") to enable efficient deserialization. This allows Mashumaro to look at the value of the specified field and instantiate the correct subclass automatically.

    from typing import Annotated, List
    from mashumaro.types import Discriminator
    
    @dataclass
    class AggregatedEvents(DataClassDictMixin):
        list: List[
            Annotated[
                ClientEvent, Discriminator(field="type", include_subtypes=True)
            ]
        ]
  8. Deserialize subclasses without a common field

    master

    If your subclasses do not share a common field to distinguish them, you can use Discriminator without the field parameter. In this mode, Mashumaro will attempt to deserialize the data against all available subclasses until one succeeds. This behaves similarly to a Union type but with explicit support for class hierarchies.

    Note: This is less efficient than field-based discrimination because it requires traversing subclasses.

    @dataclass
    class Plate(DataClassDictMixin):
        ingredients: List[
            Annotated[Ingredient, Discriminator(include_subtypes=True)]
        ]
  9. Implement GenericSerializableType for custom type logic

    master

    For maximum flexibility, implement the GenericSerializableType interface. This allows you to define custom __packers__ (for serialization) and __unpackers__ (for deserialization) that depend on the types provided in the generic arguments. This is useful for complex wrappers like a dictionary that needs specific date formatting based on its key/value types.

    from dataclasses import dataclass
    from datetime import date
    from typing import Dict, TypeVar
    from mashumaro import DataClassDictMixin
    from mashumaro.types import GenericSerializableType
    
    KT = TypeVar("KT")
    VT = TypeVar("VT")
    
    class DictWrapper(Dict[KT, VT], GenericSerializableType):
        __packers__ = {date: lambda x: x.isoformat(), str: str}
        __unpackers__ = {date: date.fromisoformat, str: str}
    
    def _serialize(self, types) -> Dict[KT, VT]:
        k_type, v_type = types
        k_conv = self.__packers__[k_type]
        v_conv = self.__packers__[v_type]
        return {k_conv(k): v_conv(v) for k, v in self.items()}
    
    @classmethod
    def _deserialize(cls, value, types) -> "DictWrapper[KT, VT]":
        k_type, v_type = types
        k_conv = cls.__unpackers__[k_type]
        v_conv = cls.__unpackers__[v_type]
        return cls({k_conv(k): v_conv(k) for k, v in value.items()})
    
    @dataclass
    class DataClass(DataClassDictMixin):
        x: DictWrapper[date, str]
        y: DictWrapper[str, date]
  10. Use generic dataclasses with inheritance

    master

    You can use generic dataclasses by inheriting from them and providing concrete types for the TypeVar parameters. This allows Mashumaro to serialize and deserialize instances based on the specific types provided during inheritance.

    from dataclasses import dataclass
    from datetime import date
    from typing import Generic, Mapping, TypeVar, TypeVarTuple, Tuple
    from mashumaro import DataClassDictMixin
    
    KT = TypeVar("KT")
    VT = TypeVar("VT", date, str)
    Ts = TypeVarTuple("Ts")
    
    @dataclass
    class GenericDataClass(Generic[KT, VT, *Ts]):
        x: Mapping[KT, VT]
        y: Tuple[*Ts, KT]
    
    @dataclass
    class ConcreteDataClass(
        GenericDataClass[str, date, *Tuple[float, ...]],
        DataClassDictMixin,
    ):
        pass
    
    ConcreteDataClass.from_dict({"x": {"a": "2021-01-01"}, "y": [1, 2, "a"]})
    # ConcreteDataClass(x={'a': datetime.date(2021, 1, 1)}, y=(1.0, 2.0, 'a'))
  11. Implement a `SerializationStrategy` for third-party types

    master

    If you need to support a third-party type that you cannot modify, implement a SerializationStrategy. This is ideal for reusable logic or when you need different formatting for the same type in different contexts.

    Usage patterns:

    1. Instance-based: Pass a specific instance of a strategy to a field via field_options(serialization_strategy=...) to allow different formats (e.g., different date formats) for different fields.
    2. Global configuration: Register a strategy in a Config class's serialization_strategy dictionary to apply it to all instances of a type across your application.
    3. Annotations: Like SerializableType, strategies can use use_annotations=True to automatically convert input types (e.g., converting a string to a float before passing it to deserialize).
    from mashumaro.types import SerializationStrategy
    from mashumaro import field_options
    
    class FormattedDateTime(SerializationStrategy):
        def __init__(self, fmt):
            self.fmt = fmt
    
        def serialize(self, value: datetime) -> str:
            return value.strftime(self.fmt)
    
        def deserialize(self, value: str) -> datetime:
            return datetime.strptime(value, self.fmt)
    
    # Usage in a dataclass
    @dataclass
    class DateTimeFormats(DataClassDictMixin):
        short: datetime = field(
            metadata=field_options(
                serialization_strategy=FormattedDateTime("%d%m%Y%H%M%S")
            )
        )
  12. Pass context to serialization hooks

    master

    You can pass a custom context object (e.g., for flags like remove_sensitive_data) to to_* methods. To enable this, add ADD_SERIALIZATION_CONTEXT to your BaseConfig.code_generation_options. This allows your __pre_serialize__ and __post_serialize__ hooks to accept a context parameter.

    Example usage:

    from mashumaro.config import BaseConfig, ADD_SERIALIZATION_CONTEXT
    
    class BaseModel(DataClassDictMixin):
        class Config(BaseConfig):
            code_generation_options = [ADD_SERIALIZATION_CONTEXT]
    
    # In your hooks:
    def __post_serialize__(self, d: Dict, context: Optional[Dict] = None):
        if context and context.get("remove_sensitive_data"):
            d["password"] = "***"
        return d
    
    # Calling the method:
    obj.to_dict(context={"remove_sensitive_data": True})
    from dataclasses import dataclass
    from typing import Dict, Optional
    from uuid import UUID
    from mashumaro import DataClassDictMixin
    from mashumaro.config import BaseConfig, ADD_SERIALIZATION_CONTEXT
    
    class BaseModel(DataClassDictMixin):
        class Config(BaseConfig):
            code_generation_options = [ADD_SERIALIZATION_CONTEXT]
    
    @dataclass
    class Account(BaseModel):
        id: UUID
        username: str
        name: str
    
    def __pre_serialize__(self, context: Optional[Dict] = None):
        return self
    
    def __post_serialize__(self, d: Dict, context: Optional[Dict] = None):
        if context and context.get("remove_sensitive_data"):
            d["username"] = "***"
            d["name"] = "***"
        return d
    
    @dataclass
    class Session(BaseModel):
        id: UUID
        key: str
        account: Account
    
    def __pre_serialize__(self, context: Optional[Dict] = None):
        return self
    
    def __post_serialize__(self, d: Dict, context: Optional[Dict] = None):
        if context and context.get("remove_sensitive_data"):
            d["key"] = "***"
        return d
    
    foo = Session(
        id=UUID('03321c9f-6a97-421e-9869-918ff2867a71'),
        key="VQ6Q9bX4c8s",
        account=Account(
            id=UUID('4ef2baa7-edef-4d6a-b496-71e6d72c58fb'),
            username="john_doe",
            name="John"
        )
    )
    assert foo.to_dict(context={"remove_sensitive_data": True}) == {
        'id': '03321c9f-6a97-421e-9869-918ff2867a71',
        'key': '***',
        'account': {
            'id': '4ef2baa7-edef-4d6a-b496-71e6d72c58fb',
            'username': '***',
            'name': '***'
        }
    }