rmp (msgpack-rust)

repository·master·Indexed 23 days ago

https://github.com/3hren/msgpack-rust

A pure-Rust implementation of the MessagePack binary serialization format. The project consists of three crates: rmp for low-level, high-performance reading and writing with no-std support; rmp-serde for serialization and deserialization using the serde framework; and rmpv for handling arbitrary MessagePack messages via a universal Value enum. It features zero-copy decoding, stream-friendly encoding, and a self-describing format.

Tokens
14.6K
Snippets
30
Records
98
Agent score
79%

What's inside rmp

  1. Choose the right RMP crate for your needs

    master

    The RMP project is split into three distinct crates depending on your requirements for abstraction and control:

    • rmp-serde: Use this for easy serialization and deserialization of Rust data structures using the serde framework and derive attributes.
    • rmpv: Use this when you need to handle arbitrary MessagePack messages without a known schema. It provides a universal Value enum that can represent any MessagePack type.
    • rmp: Use this for low-level, high-performance reading and writing of encoded data. It provides full control over the encoding/decoding process and supports no-std environments without heap allocations.
  2. Key features of RMP

    master

    RMP provides several advantages for binary serialization:

    • High-level and Low-level APIs: Choose between convenient Serde integration or low-level control with no-std and zero-heap-allocation support.
    • Zero-copy decoding: Decode bytes from a buffer in a zero-copy manner using safe Rust.
    • Stream-friendly encoding: MessagePack uses <length><data> encoding, allowing values to be safely concatenated and read from a stream.
    • Self-describing format: Data is self-describing and extensible without requiring schema definitions.
  3. Efficiently store binary data (`&[u8]`)

    master

    By default, Serde's derived implementations represent types like &[u8; N] or Vec<u8> as arrays of objects rather than byte slices, which can result in approximately 50% storage overhead.

    To store blobs efficiently in MessagePack, you have two options:

    1. Wrap your byte data in the serde_bytes crate.
    2. Configure the rmp_serde::Serializer to force the use of byte slices using the .with_bytes() method.
  4. Serialize and deserialize data with rmp-serde

    master

    The rmp-serde crate provides integration between the Rust MessagePack library and serde. It allows for easy serialization and deserialization of built-in types, standard library types, and custom data structures.

    Use rmp_serde::to_vec to serialize a value into a MessagePack-encoded Vec<u8>, and rmp_serde::from_slice to deserialize from a byte slice.

    let buf = rmp_serde::to_vec(&(42, "the Answer")).unwrap();
    
    assert_eq!(
        vec![0x92, 0x2a, 0xaa, 0x74, 0x68, 0x65, 0x20, 0x41, 0x6e, 0x73, 0x77, 0x65, 0x72],
        buf
    );
    
    assert_eq!((42, "the Answer"), rmp_serde::from_slice(&buf).unwrap());
  5. Install the rmp crate

    master

    To use the low-level MessagePack implementation in your Rust project, add rmp to your Cargo.toml dependencies.

    Note: If you require Serde support for high-level serialization/deserialization, use the rmp-serde crate instead.

    ```toml
    [dependencies.rmp]
    rmp = "0.8"
    ```埋
  6. Deserialize MessagePack Enums

    master

    In rmpv, MessagePack enums are expected to be represented as arrays. The deserializer supports two formats:

    1. Unit Variants: An array containing a single element representing the variant ID (e.g., [id]).
    2. Newtype/Tuple/Struct Variants: An array where the first element is the variant ID, followed by the variant data (e.g., [id, data] or [id, field1, field2]).

    If the MessagePack value is not an array, deserialization will fail with an error indicating that an array, map, or integer was expected.

  7. Understand the recursion depth limit in rmpv

    master

    To prevent stack overflow attacks or excessive memory usage when decoding deeply nested MessagePack structures, rmpv enforces a maximum recursion depth. If the nesting exceeds this limit, a rmpv::decode::Error::DepthLimitExceeded is returned.

    The constant defining this limit is MAX_DEPTH.

    // The current limit is 1024
    const MAX_DEPTH: usize = 1024;
    // The maximum recursion depth before [`Error::DepthLimitExceeded`] is returned.
    pub const MAX_DEPTH: usize = 1024;
  8. Configure MessagePack serialization behavior with SerializerConfig

    master

    In rmp-serde, you can customize how the Serializer and Deserializer behave by using types that implement the SerializerConfig trait. These configuration wrappers allow you to control whether structs are serialized as maps (with field names) or tuples, whether the format is considered human-readable, and how byte slices are handled.

    Common configuration wrappers include:

    • DefaultConfig: The most compact representation. Writes structs as tuples (no field names), enum variants as integers, and uses binary format.
    • StructMapConfig<C>: Wraps an existing configuration C and forces structs to be serialized as maps with field names.
    • StructTupleConfig<C>: Wraps an existing configuration C and forces structs to be serialized as tuples without field names.
    • HumanReadableConfig<C>: Wraps an existing configuration C and sets is_human_readable to true.
    • BinaryConfig<C>: Wraps an existing configuration C and sets is_human_readable to false.
  9. Important: Handling I/O errors and non-blocking readers

    master

    When using rmp with custom readers:

    1. EINTR: Most functions in this module silently handle EINTR (interrupted system call) to remain consistent with std::io::Write::write_all.
    2. Non-blocking I/O: If your reader returns recoverable errors like EWOULDBLOCK (e.g., from a non-blocking socket), you must buffer the data externally (for example, using a BufRead reader). rmp does not automatically buffer data, and failing to do so may result in data loss or incomplete parsing.
  10. Handle MessagePack Extension types with `_ExtStruct`

    master

    MessagePack Extension types can be represented in Serde by using a specific newtype struct pattern. To make this work, you must rename your struct to _ExtStruct using the #[serde(rename = "_ExtStruct")] attribute. The struct should contain a tuple of (i8, bytes) representing the tag and the binary data.

    #[derive(Debug, PartialEq, Serialize, Deserialize)]
    #[serde(rename = "_ExtStruct")]
    struct ExtStruct((i8, serde_bytes::ByteBuf));
    
    // This will map to Msgpack's Ext(tag, binary) format
  11. How enums are serialized in `rmp-serde`

    master

    Because the MessagePack specification does not define a standard for enum encoding, rmp-serde uses the following convention:

    An enum value is represented as a single-entry map.

    • The key is the variant ID (the variant name).
    • The value is a sequence containing all associated data. If the enum variant has no associated data, the sequence is empty.