libcimbar

repository·master·Indexed 27 days ago

https://github.com/sz3/libcimbar

An experimental high-density 2D color icon matrix barcode format designed for air-gapped data transfer. It enables the transfer of files up to approximately 33MB by displaying animated barcodes on a screen and decoding them via a smartphone camera. The project includes CLI tools for encoding files to PNGs, real-time animation via cimbar_send, and a WASM-based web encoder. It utilizes image hashing, error correction, interleaving, and Wirehair fountain codes for robustness and order-independent reconstruction.

Tokens
6.1K
Snippets
18
Records
53
Agent score
91%

What's inside libcimbar

  1. Overview of libcimbar

    master
    libcimbar is an optimized implementation of the cimbar high-density 2D color barcode format. It uses a file encoding protocol based on fountain codes (wirehair) and zstd compression. It can encode files up to 33MB (compressed) into a series of images or videos. The system is designed to be resilient; even if some frames are lost or corrupted during transmission via a camera, the file can still be reconstructed.
  2. Understand the Cimbar encoding premise

    master

    Cimbar uses a grid of colored tiles based on a simple threshold image hashing algorithm. An 8x8 grid is encoded as a 64-bit number (left-to-right, top-to-bottom). Symbols are chosen such that they have a high Hamming distance from one another, ensuring they remain distinguishable even when blurry or corrupted.

    In a 6-bit Cimbar configuration (4 symbol bits and 2 color bits), a single image can contain approximately 12,400 tiles, resulting in roughly 9,300 bytes of data per image.

  3. Build libcorrect using CMake

    master

    libcorrect uses CMake for out-of-source builds. To build and install the library, ensure CMake is installed, then execute the following commands from the source directory:

    mkdir build && cd build && cmake .. && make && make install

    If your host system has <x86intrin.h> available, libcorrect will automatically build an SSE version. The SSE headers are provided under <correct-sse.h>. Note that libcorrect requires SSE functions up to and including SSE4, and the caller is responsible for ensuring SSE availability and usage.

  4. Use the Wirehair C API

    master

    Wirehair provides a simple C API for fountain coding (erasure coding). It allows you to produce an unlimited stream of error correction blocks from a data source.

    Key Workflow:

    1. Call wirehair_init() to initialize the library.
    2. Create an encoder with wirehair_encoder_create() using your message data and desired packet size.
    3. Create a decoder with wirehair_decoder_create().
    4. Use wirehair_encode() to generate blocks for transmission.
    5. Use wirehair_decode() to process incoming blocks. If it returns Wirehair_NeedMore, more blocks are required to recover the data.
    6. Once enough blocks are received, use wirehair_recover() to reconstruct the original message.
    7. Free resources using wirehair_free().
  5. Build Zstandard DLLs on Windows using MinGW+MSYS

    master

    You can create DLLs using MinGW+MSYS with the make libzstd command. This generates:

    • dll\libzstd.dll (the dynamic library)
    • dll\libzstd.lib (the import library, required only for Visual C++)

    To compile a project using gcc/MinGW, you need zstd.h and dll\libzstd.dll. The dynamic library must be added to your linking options.

    gcc $(CFLAGS) -Iinclude/ test-dll.c -o test-dll dll\libzstd.dll
  6. Build the Zstandard library

    master

    Use the provided Makefile to generate static and dynamic libraries.

    • make: Generates both static and dynamic libraries. By default, the dynamic library is multithreaded and the static library is single-threaded.
    • make install: Installs libraries and headers into target system directories.
    • make lib-mt: Forces multithreading for both static and dynamic libraries.
    • make lib-nomt: Forces single-threading for both static and dynamic libraries.

    To use a custom hash function for separating object files (useful when compiling multiple times with different flags), use the HASH variable. To specify a custom build directory, use BUILD_DIR.

    ```bash
    make
    make install
    make lib-mt
    make lib-nomt
    make HASH=xxhsum
    make BUILD_DIR=objectDir/v1
    ```埋
  7. Use Fountain (Wirehair) Codes for large files

    master

    Cimbar implements Fountain (wirehair) codes to handle files larger than the standard frame size or to increase robustness.

    Key Properties:

    • Overhead: Introduces a small amount of bookkeeping (e.g., 6 bytes per 744 bytes of real data in 6-bit Cimbar).
    • Order Independence: The decoder can reconstruct the file regardless of the order in which fountain frames are received.
    • Resilience: As long as $N+1$ frames are received (where $N$ is file_size/bytes_per_frame), the file can be reconstructed even if some frames are missing.

    Constraints:

    • File Size Limit: Currently capped at 33.55MB.
    • Memory Requirement: Wirehair requires the file contents to be stored in RAM.
  8. Build the WASM encoder using Docker

    master

    To build the WASM version of the encoder, use the package-wasm.sh script within an Emscripten Docker container. This ensures all dependencies are correctly configured.

    docker run --mount type=bind,source="$(pwd)",target="/usr/src/app" -it emscripten/emsdk:3.1.39
    
    # Inside the container:
    bash /usr/src/app/package-wasm.sh
  9. Build the WASM encoder with a local Emscripten setup

    master

    If you have Emscripten installed locally, you can build the project manually. This requires first building opencv.js and then using emcmake to configure the project.

    # 1. Build opencv.js
    cd /path/to/opencv
    mkdir opencv-build-wasm
    cd opencv-build-wasm
    python3 ../platforms/js/build_js.py build_wasm --build_wasm --emscripten_dir=/path/to/emscripten
    
    # 2. Build libcimbar
    mkdir build-wasm
    cd build-wasm
    source /path/to/emscripten/emsdk/emsdk_env.sh
    emcmake cmake .. -DUSE_WASM=1 -DOPENCV_DIR=/path/to/opencv
    make -j5 install
  10. Install libcimbar dependencies

    master

    Before building libcimbar, you must install OpenCV, GLFW, and OpenGL ES headers. On Ubuntu/Debian systems, you can install these using apt:

    sudo apt install libopencv-dev libglfw3-dev libgles2-mesa-dev
  11. Build the libfec compatibility shim for libcorrect

    master

    libcorrect can be built with a compatibility layer to act as a drop-in, BSD-licensed substitute for libfec. This allows libcorrect to be linked in place of libfec in existing projects.

    Warning: Running the shim build may overwrite an existing installation of libfec.

    make shim && make install