cbor-x

repository·master·Indexed 19 days ago

https://github.com/kriszyp/cbor-x

An ultra-fast and conformant CBOR (RFC 8949) implementation for NodeJS and JavaScript. It supports standard CBOR serialization and deserialization, high-performance streaming via EncoderStream and DecoderStream, and specialized extensions including record structures, structured cloning for cyclic references, and Packed CBOR encoding. It provides built-in support for preserving typed objects such as Error, Set, RegExp, and TypedArrays.

Tokens
11.7K
Snippets
50
Records
60
Agent score
63%

What's inside cbor-x

  1. Use Record/Object Structures for efficient encoding

    master

    cbor-x distinguishes between arbitrary Maps and well-defined object structures (records). By using the record extension, the encoder can reuse structure definitions, leading to more compact encodings and 2-3x faster decoding performance.

    By default, a new Encoder instance has the record extension enabled. For large object structures with repeating nested objects, this provides significant benefits.

    To disable this behavior and revert to standard MessageMap serialization (where objects are treated as maps and deserialized to JS Objects), set the objectsAsMaps property to true.

    import { Encoder } from 'cbor-x';
    let encoder = new Encoder();
    encoder.encode(myBigData);
  2. Enable Structured Cloning for complex types

    master
    By enabling structured cloning, cbor-x uses specific tags and extensions to preserve the types of complex JavaScript objects. This allows for the seamless serialization and deserialization of Set, Map, Error, RegExp, and ArrayBufferView objects.
  3. Compare CBOR vs MessagePack

    master

    If you are deciding between cbor-x and msgpackr (the MessagePack equivalent), consider these trade-offs:

    • MessagePack: Has wider adoption and msgpackr has broader usage.
    • CBOR: Has an official IETF standardization track (RFC 8949), and its tag system is a better philosophical fit for the proposed record extensions.
  4. Configure 32-bit float encoding modes

    master

    By default, non-integer numbers are serialized as 64-bit floats. To save space, you can use the useFloat32 option with one of the following constants provided by the module:

    • ALWAYS (1): Always encodes non-integers (absolute value < 2147483648) as 32-bit floats.
    • DECIMAL_ROUND (3): Always encodes non-integers as 32-bit floats and rounds to significant decimal digits during decoding.
    • DECIMAL_FIT (4): Only encodes non-integers as 32-bit floats if they can be unambiguously encoded without loss of precision, using decimal rounding during decoding.

    Note: DECIMAL_ROUND and DECIMAL_FIT may decrease performance by approximately 20-25%.

    import { ALWAYS, DECIMAL_ROUND, DECIMAL_FIT } from 'cbor-x'
    
    // Example usage in an Encoder
    const encoder = new Encoder({ useFloat32: DECIMAL_FIT });
  5. Use cbor-x in the browser

    master

    For browser environments, you can include the bundled UMD script. This will register a CBOR global or act as a module if possible.

    To minimize bundle size in module-based development, import only the specific functions you need directly from the subpaths.

    <script src="node_modules/cbor-x/dist/index.js"></script>
    // Recommended for module-based development to minimize dependencies
    import { decode } from 'cbor-x/decode';
  6. Implement Shared Record Structures for persistence

    master

    When storing data in databases or files, you can use shared structures to improve efficiency across different encoding sessions. You can provide a structures array to the Encoder constructor, which is automatically updated as new structures are encountered (up to a limit of 64).

    To handle persisted data robustly, use the getStructures() and saveStructures(structures) hooks. This allows you to load and save the generated shared structures in a way that is compatible with multiple processes or storage systems. The structures are added incrementally, so an object encoded with an earlier version of the structures can still be decoded with a later version.

    import { Encoder, decode, encode } from 'cbor-x';
    import { readFileSync, writeFileSync } from 'fs';
    
    let encoder = new Encoder({
    	getStructures() {
    		// Load structures from a file, DB, or KV store
    		return decode(readFileSync('my-shared-structures.cbor')) || [];
    	},
    	saveStructures(structures) {
    		// Persist the updated structures
    		writeFileSync('my-shared-structures.cbor', encode(structures))
    	},
    	structures: []
    });
  7. How the Decoder handles Maps vs Objects

    master

    By default, cbor-x can decode CBOR maps into either JavaScript Map objects or plain JavaScript objects. This is controlled by the mapsAsObjects option in the Decoder constructor.

    • mapsAsObjects: true (Default): Maps are converted to { key: value } objects. This is often more convenient for standard JSON-like data.
    • mapsAsObjects: false: Maps are converted to new Map() instances. This is necessary if your keys are not strings or if you need to preserve the distinction between a key and a property name.

    Note: If you use keyMap for decoding, mapsAsObjects is automatically set to true unless explicitly overridden.

    // Decodes maps as Map instances
    const decoder = new Decoder({ mapsAsObjects: false });
    const result = decoder.decode(buffer);
    console.log(result instanceof Map); // true
    
    // Decodes maps as plain objects
    const decoderObj = new Decoder({ mapsAsObjects: true });
    const resultObj = decoderObj.decode(buffer);
    console.log(typeof resultObj); // 'object'
  8. Enable Packed CBOR encoding

    master

    cbor-x supports decoding Packed CBOR automatically without any configuration.

    To generate Packed CBOR, pass the pack option to the Encoder. This causes the encoder to look for repeated strings within the data structure and store them in a packed table. While this reduces encoding size and can speed up decoding, it introduces extra encoding overhead and may reduce encoding performance compared to standard encoding.

    // Example of enabling packed encoding
    let encoder = new Encoder({ pack: true });
  9. How shared structures and packing work

    master

    The Encoder uses two primary mechanisms to reduce the size of serialized data:

    1. Shared Structures: Instead of repeating keys in every object, the encoder identifies common object shapes (sets of keys). It assigns a recordId to a specific set of keys. Subsequent objects with the same keys are encoded by referencing the recordId instead of the full key list. This is controlled via the structures option.

    2. Packing: Based on the IETF CBOR-packed draft, the pack option allows the encoder to identify repetitive values. These values are collected into a prefix array at the start of the encoded output, and subsequent occurrences of these values are replaced with small integer tags or references to the packed array.

  10. Configure Encoder and Decoder options

    master

    When instantiating an Encoder or Decoder, you can pass an options object to customize behavior. Key options include:

    • keyMap: An object used to map keys in source Objects/Maps to other keys (including integers) for more efficient encoding or Senml support.
    • useRecords: If false, disables the record extension. Objects are stored as CBOR maps (tag 259) and decoded as JS Objects for compatibility.
    • structures: An array of structures used for the record extension. This array is modified in place.
    • structuredClone: Enables structured cloning extensions for cyclic references and additional built-in types.
    • mapsAsObjects: If true, decodes CBOR maps/JS Objects as object properties. If false, decodes as JS Maps. (Note: behavior depends on useRecords).
    • useFloat32: Enables 32-bit floating point encoding for non-integers.
    • alwaysUseFloat: Forces all numbers (including integers) to be encoded as floats.
    • pack: Enables CBOR packing.
    • variableMapSize: Uses varying map size definitions for more compact encoding of small objects (slower encoding).
    • copyBuffers: If true, copies buffers during decoding instead of providing a slice/view.
    • bundleStrings: Uses a custom extension to bundle strings for faster decoding in browsers/Deno.
    • useTimestamp32: Encodes JS Dates in 32-bit format (dropping milliseconds).
    • sequential: Encodes structures and references them, expecting the decoder to read them in the same order via unpackMultiple.
    • largeBigIntToFloat: Encodes bigints larger than 64-bit as float-64 instead of throwing a RangeError.
    • useTag259ForMaps: Uses tag 259 for JS Maps. (Note: behavior depends on useRecords).
    • tagUint8Array: Uses tag 64 for Uint8Arrays.
    • int64AsNumber: Decodes uint64/int64 as standard JS numbers instead of bigint.
    • skipFunction: Skips functions during object encoding.