yyjson Documentation

repository·master·Indexed 26 days ago

https://github.com/ibireme/yyjson

A high-performance, ANSI C-compliant (C89) JSON library designed for rapid parsing and serialization of large JSON datasets. It strictly complies with RFC 8259 and supports JSON5 features, custom allocators, and accurate reading/writing of int64, uint64, and double numbers. The library provides both immutable structures for reading and mutable structures for building JSON, along with an incremental reading API for handling very large documents without freezing applications.

Tokens
17.2K
Snippets
35
Records
68
Agent score
87%

What's inside yyjson

  1. Overview of yyjson

    master
    yyjson is a high-performance JSON library written in ANSI C (C89). It is designed for speed, portability, and strict compliance with the RFC 8259 JSON standard. It supports accurate reading/writing of int64, uint64, and double numbers, and offers features like JSON5 support and custom allocators.
  2. Understand yyjson data structures

    master

    yyjson uses two distinct types of data structures depending on whether you are reading or building JSON:

    1. Immutable structures (yyjson_doc and yyjson_val): Returned when reading an existing JSON document. These cannot be modified after creation.
    2. Mutable structures (yyjson_mut_doc and yyjson_mut_val): Used when building a new JSON document from scratch. These can be modified (e.g., appending or removing elements).

    While yyjson provides functions to convert between these types, it is recommended to use the public API rather than accessing these structs directly.

  3. Understand yyjson data structures (Immutable vs Mutable)

    master

    yyjson uses two types of data structures depending on whether you are reading or building JSON:

    TypeImmutable (Reading)Mutable (Building)
    Documentyyjson_docyyjson_mut_doc
    Valueyyjson_valyyjson_mut_val

    Key Concepts:

    • Immutable: Returned when reading JSON. The document holds the memory for all values and strings.
    • Mutable: Used when building JSON.
    • Conversion: You can convert between them using yyjson_doc_mut_copy (doc $\rightarrow$ mut_doc), yyjson_val_mut_copy (val $\rightarrow$ mut_val), yyjson_mut_doc_imut_copy (mut_doc $\rightarrow$ doc), and yyjson_mut_val_imut_copy (mut_val $\rightarrow$ val).
    • API Pattern: For most immutable APIs, you can access the mutable version by adding mut after the prefix (e.g., yyjson_is_str $\rightarrow$ yyjson_mut_is_str).
  4. Use YYJSON_READ_INSITU for faster parsing

    master

    To improve reading speed, use the YYJSON_READ_INSITU flag. This allows the reader to use the input buffer itself to store string values.

    Critical Requirements:

    1. The input buffer must be padded with at least YYJSON_PADDING_SIZE bytes (typically 4).
    2. The caller must ensure the input buffer is not freed until yyjson_doc_free() has been called.
    3. The input buffer must be held until the document is no longer needed.
    size_t dat_len = ...;
    char *buf = malloc(dat_len + YYJSON_PADDING_SIZE); // create a buffer larger than (len + 4)
    read_from_socket(buf, ...);
    memset(buf + dat_len, 0, YYJSON_PADDING_SIZE); // set 4-byte padding after data
    
    yyjson_doc *doc = yyjson_read_opts(buf, dat_len, YYJSON_READ_INSITU, NULL, NULL);
    if (doc) {...}
    yyjson_doc_free(doc);
    free(buf); // the input data should be freed after the document.
  5. Build yyjson in freestanding mode

    master

    To build yyjson without a libc (e.g., for WebAssembly targets without a libc sysroot), define YYJSON_FREESTANDING as 1.

    When using this mode:

    • string.h functions like memcpy, memmove, memset, memcmp, and strlen are provided via built-in inline fallbacks.
    • File and FILE pointer APIs are disabled (equivalent to YYJSON_DISABLE_FILE).
    • malloc and free are unavailable. You must either pass a yyjson_alc allocator to compatible APIs or define a global default at compile time using -DYYJSON_CUSTOM_ALC=my_alc.
    • If you have custom implementations for string functions, you can define YYJSON_FREESTANDING_HEADER to point to your custom header.
  6. Use unsafe APIs to skip null checks

    master

    The standard public APIs perform null checks on every input parameter to prevent crashes. If you are certain a value is non-null and matches the expected type (e.g., when iterating over a known valid JSON structure), you can use the unsafe_ prefix API to improve performance.

    Example:

    size_t idx, max;
    yyjson_val *key, *val;
    yyjson_obj_foreach(obj, idx, max, key, val) {
        // using unsafe_ prefix to skip checks
        if (unsafe_yyjson_equals_str(key, "id") &&
            unsafe_yyjson_is_uint(val) &&
            unsafe_yyjson_get_uint(val) == 1234) {
            ...
        }
    }
  7. Handle reader errors and locate positions

    master

    When a reading function fails, pass a pointer to a yyjson_read_err struct to receive details.

    Error Details:

    • err.msg: Error message string.
    • err.code: Error code (yyjson_read_code).
    • err.pos: Byte position where the error occurred.

    Locating Line and Column: Use yyjson_locate_pos() to convert the byte position into human-readable line and column numbers. Note that line and column start from 1, while character starts from 0.

    char *dat = ...;
    size_t dat_len = ...;
    yyjson_read_err err = ...;
    
    // 1. Check error details
    if (!doc) {
        printf("read error: %s, code: %u at byte position: %lu\n", 
                err.msg, err.code, err.pos);
    }
    
    // 2. Get line/column/char
    size_t line, col, chr;
    if (yyjson_locate_pos(dat, dat_len, err.pos, &line, &col, &chr)) {
        printf("error at line: %lu, column: %lu, character index: %lu\n",
               line, col, chr);
    }
    char *dat = ...;
    size_t dat_len = ...;
    yyjson_read_err err = ...;
    
    if (!doc) {
        printf("read error: %s, code: %u at byte position: %lu\n", 
                err.msg, err.code, err.pos);
        // printed:
        // read error: trailing comma is not allowed, code: 7, at byte position: 40
    }
    
    yyjson_doc_free(doc);
  8. Manage memory for yyjson documents

    master

    Memory management in yyjson is document-centric. A JSON document (yyjson_doc or yyjson_mut_doc) owns the memory for all its contained values and strings.

    • Freeing memory: You must call yyjson_doc_free() for immutable documents or yyjson_mut_doc_free() for mutable documents when they are no longer needed.
    • Value lifetime: Individual JSON values (yyjson_val or yyjson_mut_val) share the lifetime of their parent document. You cannot free a value independently; it will be freed automatically when the document is freed.
  9. Read large JSON documents incrementally

    master

    To prevent freezing the program when reading very large documents, use the incremental reading API. This is slightly slower than standard reading but keeps the application responsive.

    Note: The incremental reader only supports standard JSON. Flags for non-standard features (comments, trailing commas) are ignored.

    Workflow:

    1. Create state with yyjson_incr_new().
    2. Call yyjson_incr_read() repeatedly with increasing byte lengths.
    3. If err.code == YYJSON_READ_ERROR_MORE, continue reading.
    4. Free state with yyjson_incr_free().
    const char *dat = your_file.bytes;
    size_t len = your_file.size;
    
    yyjson_read_flag flg = YYJSON_READ_NOFLAG;
    yyjson_incr_state *state = yyjson_incr_new(dat, len, flg, NULL);
    yyjson_doc *doc;
    yyjson_read_err err;
    size_t read_so_far = 0;
    
    do {
        read_so_far += 100000; // Increment by several KB/MB for efficiency
        if (read_so_far > len) read_so_far = len;
        
        doc = yyjson_incr_read(state, read_so_far, &err);
        if (err.code != YYJSON_READ_ERROR_MORE) break;
    } while (read_so_far < len);
    
    yyjson_incr_free(state);
    
    if (doc != NULL) {
        // ... use doc
        yyjson_doc_free(doc);
    }
    const char *dat = your_file.bytes;
    size_t len = your_file.size;
    
    yyjson_read_flag flg = YYJSON_READ_NOFLAG;
    yyjson_incr_state *state = yyjson_incr_new(dat, len, flg, NULL);
    yyjson_doc *doc;
    yyjson_read_err err;
    size_t read_so_far = 0;
    do {
        read_so_far += 100000;
        if (read_so_far > len)
            read_so_far = len;
        doc = yyjson_incr_read(state, read_so_far, &err);
        if (err.code != YYJSON_READ_ERROR_MORE)
            break;
    } while (read_so_far < len);
    yyjson_incr_free(state);
    
    if (doc != NULL) { ... }
    
    yyjson_doc_free(doc);
  10. Build JSON documents using the mutable API

    master

    Use yyjson_mut_doc and its related APIs to construct JSON documents from scratch.

    Important Considerations:

    • Memory Pool: yyjson_mut_doc uses a memory pool for all strings and values. The pool can only be created, grown, or freed in its entirety. This makes the API more suitable for write-once operations rather than frequent mutation of existing documents.
    • Ownership: JSON objects and arrays are composed of linked lists. Each yyjson_mut_val can only be added to one object or array at a time. Attempting to add a value that is already part of another container will result in incorrect behavior.
    // Build this JSON:
    //     {
    //        "page": 123,
    //        "names": [ "Harry", "Ron", "Hermione" ]
    //     }
    
    // Create a mutable document.
    yyjson_mut_doc *doc = yyjson_mut_doc_new(NULL);
    
    // Create an object, the value's memory is held by doc.
    yyjson_mut_val *root = yyjson_mut_obj(doc);
    
    // Create key and value, add to the root object.
    yyjson_mut_val *key = yyjson_mut_str(doc, "page");
    yyjson_mut_val *num = yyjson_mut_int(doc, 123);
    yyjson_mut_obj_add(root, key, num);
    
    // Create 3 string values, add to the array object.
    yyjson_mut_val *names = yyjson_mut_arr(doc);
    yyjson_mut_val *name1 = yyjson_mut_str(doc, "Harry");
    yyjson_mut_val *name2 = yyjson_mut_str(doc, "Ron");
    yyjson_mut_val *name3 = yyjson_mut_str(doc, "Hermione");
    yyjson_mut_arr_append(names, name1);
    yyjson_mut_arr_append(names, name2);
    yyjson_mut_arr_append(names, name3);
    yyjson_mut_obj_add(root, yyjson_mut_str(doc, "names"), names);
    
    // ❌ Wrong! the value is already added to another container.
    // yyjson_mut_obj_add(root, key, name1);
    
    // Set the document's root value.
    yyjson_mut_doc_set_root(doc, root);
    
    // Write to JSON string
    const char *json = yyjson_mut_write(doc, 0, NULL);
    
    // Free the memory of doc and all values created from this doc.
    yyjson_mut_doc_free(doc);
  11. Build yyjson using CMake

    master

    To build the library from source using CMake, create a build directory and run the build commands. You can choose to build either a static or a shared library.

    # Create build directory
    git clone https://github.com/ibireme/yyjson.git
    cmake -E make_directory build; cd build
    
    # Build static library
    cmake .. 
    cmake --build .
    
    # Build shared library
    cmake .. -DBUILD_SHARED_LIBS=ON
    cmake --build .
  12. Install yyjson using vcpkg

    master

    You can use the vcpkg dependency manager to install yyjson by following these steps:

    git clone https://github.com/Microsoft/vcpkg.git
    cd vcpkg
    ./bootstrap-vcpkg.sh  # ./bootstrap-vcpkg.bat for PowerShell
    ./vcpkg integrate install
    ./vcpkg install yyjson