DOCSEARCHbyAlgolia
Submit Docs

Lite³ Documentation

repository·main·Indexed 21 days ago

https://github.com/fastserial/lite3

A high-performance, zero-copy, JSON-compatible binary serialization format that uses a B-tree structure within a contiguous buffer. Lite³ allows for O(log n) access and mutation without traditional parsing or serialization steps. It features a low-level Buffer API for caller-provided memory management and a higher-level Context API for abstracted memory management. The library is schemaless, self-describing, and includes security measures such as bounds checks and recursion limits.

Tokens
2.8K
Snippets
7
Records
12
Agent score
24%

What's inside Lite³

  1. What is Lite³ and how does it work?

    main

    Lite³ is a zero-copy, JSON-compatible binary serialization format. It encodes data as a B-tree within a single contiguous buffer, acting as a serialized dictionary.

    Unlike traditional serialization formats, Lite³ blurs the line between memory and wire formats. Because the wire format is the memory format, you do not need to 'parse' or 'serialize' data. Once a Lite³ buffer is received (e.g., from a socket), you can immediately perform O(log n) operations such as:

    • Looking up keys and reading values via zero-copy pointers.
    • Inserting or overwriting arbitrary key/value entries.
    • Transmitting the buffer 'as-is' using memcpy() after mutations.

    Key characteristics:

    • Schemaless & Self-describing: No IDL or schema definitions required.
    • Zero-copy: Supports zero-copy reads and writes for any data size.
    • Memory Management: The library does not use malloc(); the caller provides the buffer.
    • JSON Compatibility: Supports conversion to/from JSON (requires the yyjson subdependency).

  2. Choose between the Buffer API and Context API

    main

    Lite³ provides two distinct APIs depending on your requirements for memory management and control:

    • Buffer API: Provides maximum control. It uses caller-supplied buffers, which is ideal for environments with custom allocation patterns where you want to avoid malloc().
      • Header: #include "lite3.h"
    • Context API: A higher-level wrapper around the Buffer API. It hides memory allocation from the user, making it more accessible and easier to use for beginners.
      • Header: #include "lite3_context_api.h

    You only need to include the header for the API you intend to use.

    #include "lite3.h"              // Buffer API
#include "lite3_context_api.h"  // Context API

  3. Security features of Lite³

    main

    Lite³ is designed to handle untrusted messages by implementing several security measures to mitigate risks associated with its pointer-chasing format:

    • Bounds Checks: All pointer dereferences are preceded by overflow-protected bounds checks.
    • Type Safety: Provides runtime type safety.
    • Recursion Limits: Implements maximum recursion limits to prevent stack exhaustion.
    • Dangling Pointer Prevention: Uses a generational pointer macro to prevent dangling pointers into Lite³ buffers.

  4. Install Lite³ via pkg-config

    main

    The easiest way to install Lite³ is using pkg-config. This method builds the static library and installs it to /usr/local, allowing you to use pkg-config flags for compilation.

    1. Clone the repository:
      git clone https://github.com/fastserial/lite3.git
cd lite3/
    2. Install the library:
      sudo make install -j
sudo ldconfig
    3. Verify the installation:
      pkg-config --modversion lite3

    To compile your program using the installed library, use the following command structure:

    gcc -o main main.c $(pkg-config --libs --cflags --static lite3)
    git clone https://github.com/fastserial/lite3.git
cd lite3/
sudo make install -j
sudo ldconfig
pkg-config --modversion lite3
gcc -o main main.c $(pkg-config --libs --cflags --static lite3)

  5. Enable library error messages for development

    main

    By default, Lite³ error messages are disabled. To receive feedback during development, you must explicitly enable them using one of two methods:

    1. Header modification: Uncomment the line // #define LITE3_ERROR_MESSAGES inside include/lite3.h.
    2. Compilation flag: Build the library or your application using the -DLITE3_ERROR_MESSAGES flag.

    If you have already installed the library via pkg-config, you must reinstall it to apply these changes:

    sudo make uninstall
sudo make clean
sudo make install
sudo ldconfig
    gcc -DLITE3_ERROR_MESSAGES ...

  6. Requirements for embedded and ARM platforms

    main

    Lite³ can be used on embedded or ARM platforms provided the following requirements are met:

    • Type Support: The platform must support the int64_t type and 8-byte doubles.
    • Compiler: A suitable C11 gcc or clang compiler is required.
    • C99 Compatibility: You can downgrade to C99 by removing all static assertions in the source.

    Note: As of the current documentation, the format has not been explicitly tested on ARM.

  7. Install Lite³ via manual linking

    main

    If you prefer not to install the library globally, you can link against the static library manually.

    1. Build the library in the project root:
      make -j
    2. In your source code, include the necessary headers:
      • include/lite3.h (for the Buffer API)
      • include/lite3_context_api.h (for the Context API)
    3. Compile by pointing to the include directory and the built static library:
    gcc -o main main.c -I/path/to/lite3/include /path/to/lite3/build/liblite3.a
    make -j
gcc -o main main.c -I/path/to/lite3/include /path/to/lite3/build/liblite3.a

  8. Build and run Lite³ examples

    main

    Examples are organized by API type in the following directories:

    • examples/buffer_api/*
    • examples/context_api/*

    To build all examples, run the following from the project root:

    make examples -j

    To run a specific example (e.g., a context API example):

    ./build/examples/context_api/01-building-messages
    make examples -j
./build/examples/context_api/01-building-messages

  9. Compare Lite³ with JSON and Protocol Buffers

    main

    When deciding whether to use Lite³ for your project, consider the following trade-offs:

    Lite³ vs. JSON

    Use Lite³ if you care about performance and can directly interface with C code. If you require high-level language support, you may need to wait for better language bindings.

    Lite³ vs. Protocol Buffers (Protobuf)

    • Choose Lite³ if you are CPU-constrained: Lite³ outperforms Protobuf in encode/decode performance due to its zero-copy advantage. However, because Lite³ is self-describing (it encodes field names), messages take up more space over the wire.
    • Choose Protocol Buffers if you are bandwidth-constrained: Protobuf minimizes message size but requires extra tooling, an Interface Definition Language (IDL), and handles ABI-breaking evolution differently.

  10. Use the Lite³ Low-Level API with caller-provided buffers

    main

    The low-level API allows you to manage your own memory by providing a pre-allocated buffer. This is ideal for high-performance scenarios where you want to avoid malloc() calls.

    To use this API, you must track the current length of the buffer (buflen) and the total size of the buffer (bufsz).

    Common functions include:

    • lite3_init_obj(buf, &buflen, bufsz): Initializes the object in the provided buffer.
    • lite3_set_str(buf, &buflen, offset, bufsz, key, value): Sets a string value.
    • lite3_set_i64(buf, &buflen, offset, bufsz, key, value): Sets a 64-bit integer value.
    • lite3_get_i64(buf, buflen, offset, key, &out_value): Retrieves a 64-bit integer value.
    #include <stdio.h>
#include <stdbool.h>
#include "lite3.h"

uint8_t buf[1024];

int main() {
    size_t buflen = 0;
    size_t bufsz = sizeof(buf);

    lite3_init_obj(buf, &buflen, bufsz);
    lite3_set_str(buf, &buflen, 0, bufsz, "app_name", "demo_app");
    lite3_set_i64(buf, &buflen, 0, bufsz, "max_retries", 3);
    lite3_set_bool(buf, &buflen, 0, bufsz, "debug_mode", false);

    int64_t max_retries;
    lite3_get_i64(buf, buflen, 0, "max_retries", &max_retries);
    printf("max retries: %li\n", max_retries);

    return 0;
}

  11. Use the Lite³ Context API for abstracted memory management

    main

    The Context API provides an abstraction layer where Lite³ handles memory management for you via a lite3_ctx object. This is useful for building complex, nested structures without manually tracking buffer offsets and sizes.

    Key workflow:

    1. Create a context with lite3_ctx_create().
    2. Initialize the object with lite3_ctx_init_obj(ctx).
    3. Set values using lite3_ctx_set_str, lite3_ctx_set_i64, etc.
    4. For nested objects, use lite3_ctx_set_obj(ctx, parent_offset, key, &child_offset) to get the offset of the new sub-object.
    5. Retrieve values using lite3_ctx_get_obj and lite3_ctx_get_str.
    6. Destroy the context with lite3_ctx_destroy(ctx).

    To print the structure as JSON, use lite3_ctx_json_print(ctx, offset). To access string data from a lite3_str returned by the API, use the LITE3_STR(ctx->buf, user_agent) macro.

    #include <stdio.h>
#include <string.h>
#include "lite3_context_api.h"

int main() {
    lite3_ctx *ctx = lite3_ctx_create();
    
    // Build message
    lite3_ctx_init_obj(ctx);
    lite3_ctx_set_str(ctx, 0, "event", "http_request");
    lite3_ctx_set_str(ctx, 0, "method", "POST");
    lite3_ctx_set_i64(ctx, 0, "duration_ms", 47);

    // Set headers
    size_t headers_ofs;
    lite3_ctx_set_obj(ctx, 0, "headers", &headers_ofs);
    lite3_ctx_set_str(ctx, headers_ofs, "content-type", "application/json");
    lite3_ctx_set_str(ctx, headers_ofs, "x-request-id", "req_9f8e2a");
    lite3_ctx_set_str(ctx, headers_ofs, "user-agent", "curl/8.1.2");

    lite3_ctx_json_print(ctx, 0); // Print Lite³ as JSON

    // Get user-agent
    lite3_str user_agent;
    size_t ofs;
    lite3_ctx_get_obj(ctx, 0, "headers", &ofs);
    lite3_ctx_get_str(ctx, ofs, "user-agent", &user_agent);
    printf("User agent: %s\n", LITE3_STR(ctx->buf, user_agent));

    lite3_ctx_destroy(ctx);
    return 0;
}

  12. Reference: Lite³ Make Commands

    main

    Available commands for managing the Lite³ build process:

    CommandDescription
    make allBuild the static library with -O2 optimizations (default)
    make testsBuild and run all tests (use VERBOSE=1 for stdout output)
    make examplesBuild all examples
    make installInstall library in /usr/local (for pkg-config)
    make uninstallUninstall library
    make cleanRemove all build artifacts
    make helpShow this help message