ESP32-targz Arduino Library

repository·master·Indexed 41 days ago

https://github.com/tobozo/esp32-targz

An Arduino library providing compression and decompression for .tar, .gz, and .tar.gz files. It supports ESP32, ESP8266, RP2040, and Teensy microcontrollers. Key features include the tarGzStreamUpdater() for automating binary extraction to OTA and SPIFFS partitions, as well as GzUnpacker, TarUnpacker, and TarGzUnpacker classes for handling archives. It includes the uzlib library for lightweight Deflate, Zlib, and Gzip bitstream processing.

Tokens
4.6K
Snippets
14
Records
19
Agent score
100%

What's inside ESP32-targz

  1. Overview of the uzlib library

    master

    uzlib is a lightweight compression/decompression library designed for deeply embedded systems with minimal code size and runtime memory requirements. It is compatible with Deflate, Zlib, and Gzip bitstreams.

    Key Capabilities:

    • Decompression: Can decompress any valid Deflate, Zlib, or Gzip bitstream.
    • Compression: Can compress data to a Deflate-compatible bitstream using LZ77 compression and static Deflate Huffman tree encoding. Note that the compression ratio may be lower than the standard Zlib Deflate algorithm.
  2. Overview of ESP32-targz

    master

    ESP32-targz is an Arduino library designed for compression and decompression on resource-constrained microcontrollers. It allows devices to handle .tar, .gz, and .tar.gz files.

    Supported Hardware:

    • ESP32
    • ESP8266
    • RP2040
    • Teensy (all versions)
  3. Extract binaries from .tar.gz using tarGzStreamUpdater()

    master

    The tarGzStreamUpdater(Stream *stream) function automates the extraction of binaries from a .tar.gz archive directly to OTA (Over-the-Air) and/or SPIFFS partitions. It utilizes the UpdateClass from the ESP32 Arduino core to perform the updates.

    Because the function accepts a Stream object, you can pass various stream types, such as an SD card file stream, an HTTP client stream, or a WiFi secure client stream.

    tarGzStreamUpdater(stream);
  4. Configure the destination filesystem

    master

    ESP32-targz requires you to set the output filesystem at compilation time. This is done by defining a specific macro before including the <ESP32-targz.h> header. This will create a filesystem object named tarGzFS that you can use in your code.

    Available destination macros:

    • #define DEST_FS_USES_SPIFFS
    • #define DEST_FS_USES_FFAT
    • #define DEST_FS_USES_SD
    • #define DEST_FS_USES_SD_MMC
    • #define DEST_FS_USES_LITTLEFS
    // Set **destination** filesystem by uncommenting one of these:
    //#define DEST_FS_USES_SPIFFS
    //#define DEST_FS_USES_FFAT
    //#define DEST_FS_USES_SD
    //#define DEST_FS_USES_SD_MMC
    #define DEST_FS_USES_LITTLEFS
    #include <ESP32-targz.h>
    // filesystem object will be available as "tarGzFS"
  5. Use PlatformIO Library Dependency Finder (LDF)

    master
    PlatformIO includes a Library Dependency Finder (LDF) that automatically scans your project's source files (e.g., #include <Foo.h>) to identify and include the necessary libraries. You do not need to manually register every private library in your platformio.ini if they are located in the lib/ folder.
  6. Organize private libraries in PlatformIO

    master

    When using PlatformIO, you can manage project-specific (private) libraries by placing them in the lib/ directory. Each library must reside in its own subdirectory within lib/. PlatformIO will automatically compile these as static libraries and link them into your executable.

    Directory Structure Example:

    |--lib
    |  |--Bar
    |  |  |--docs
    |  |  |--examples
    |  |  |--src
    |  |  |     |- Bar.c
    |  |  |     |- Bar.h
    |  |  |  |--library.json (optional)
    |  |
    |  |--Foo
    |  |     |- Foo.c
    |  |     |- Foo.h
    |
    |--platformio.ini
    |--src
       |- main.c
    // src/main.c
    #include <Foo.h>
    #include <Bar.h>
    
    int main (void)
    {
      // Your code here
    }
  7. Flash ESP32 from a .gz file or stream

    master

    ESP32 users can perform OTA-style updates using compressed files.

    From a file on filesystem: Use GzUnpacker::gzUpdater() to flash a .gz file to the device.

    From a Stream (e.g., HTTP/HTTPS): Use GzUnpacker::gzStreamUpdater() to flash directly from a stream without saving the file to disk first.

    // ESP32 Only: Flash from .gz stream
    fs::File file = tarGzFS.open( "/example_firmware.gz", "r" );
    if (!file) return;
    
    GzUnpacker *GZUnpacker = new GzUnpacker();
    GZUnpacker->haltOnError( true );
    GZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn );
    
    if( !GZUnpacker->gzStreamUpdater( (Stream *)&file, UPDATE_SIZE_UNKNOWN ) ) {
      Serial.printf("gzStreamUpdater failed with return code #%d\n", GZUnpacker->tarGzGetError() );
    }
  8. Expand contents from a .tar.gz file

    master

    Use the TarGzUnpacker class to unpack a .tar.gz archive. You can choose between two modes:

    1. With intermediate file: Uses a temporary file (default is /tmp/tmp.tar) during the process.
    2. Without intermediate file: Direct expansion (uses more heap memory).

    Note: For ESP32, direct expansion from a Stream (like an HTTP stream) is also supported via tarGzStreamExpander.

    // mount spiffs (or any other filesystem)
    tarGzFS.begin();
    
    TarGzUnpacker *TARGZUnpacker = new TarGzUnpacker();
    
    TARGZUnpacker->haltOnError( true );
    TARGZUnpacker->setTarVerify( true );
    TARGZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn );
    TARGZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback );
    TARGZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );
    TARGZUnpacker->setTarProgressCallback( BaseUnpacker::defaultProgressCallback );
    TARGZUnpacker->setTarStatusProgressCallback( BaseUnpacker::defaultTarStatusProgressCallback );
    TARGZUnpacker->setTarMessageCallback( BaseUnpacker::targzPrintLoggerCallback );
    
    // using an intermediate file (default is /tmp/tmp.tar)
    if( !TARGZUnpacker->tarGzExpander(tarGzFS, "/targz_example.tar.gz", tarGzFS, "/tmp") ) {
      Serial.printf("tarGzExpander+intermediate file failed with return code #%d\n", TARGZUnpacker->tarGzGetError() );
    }
    
    // or without intermediate file
    if( ! TARGZUnpacker->tarGzExpander(tarGzFS, "/targz_example.tar.gz", tarGzFS, "/tmp", nullptr ) ) {
      Serial.printf("tarGzExpander+intermediate file failed with return code #%d\n", TARGZUnpacker->tarGzGetError() );
    }
  9. Expand contents from a .tar file

    master

    Use the TarUnpacker class to unpack a .tar archive to a destination directory.

    Key features:

    • setTarVerify(true): Enables health checks (slower process).
    • setupFSCallbacks(...): Recommended to prevent partition overflow.
    • setTarProgressCallback(...): Prints progress for individual files.
    • setTarStatusProgressCallback(...): Prints filenames as they are expanded.
    // mount spiffs (or any other filesystem)
    tarGzFS.begin();
    
    TarUnpacker *TARUnpacker = new TarUnpacker();
    
    TARUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    TARUnpacker->setTarVerify( true ); // true = enables health checks but slows down the overall process
    TARUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    TARUnpacker->setTarProgressCallback( BaseUnpacker::defaultProgressCallback ); // prints the untarring progress for each individual file
    TARUnpacker->setTarStatusProgressCallback( BaseUnpacker::defaultTarStatusProgressCallback ); // print the filenames as they're expanded
    TARUnpacker->setTarMessageCallback( BaseUnpacker::targzPrintLoggerCallback  ); // tar log verbosity
    
    if( !TARUnpacker->tarExpander(tarGzFS, "/tar_example.tar", tarGzFS, "/") ) {
      Serial.printf("tarExpander failed with return code #%d\n", TARUnpacker->tarGzGetError() );
    }
  10. Extract content from a .gz file

    master

    Use the GzUnpacker class to decompress a single .gz file from a source filesystem to a destination filesystem.

    Recommended steps:

    1. Mount your filesystem using tarGzFS.begin().
    2. Initialize GzUnpacker.
    3. Set error handling, filesystem callbacks (to prevent partition overflow), and progress/logger callbacks.
    4. Call gzExpander() to perform the extraction.
    // mount spiffs (or any other filesystem)
    tarGzFS.begin();
    
    GzUnpacker *GZUnpacker = new GzUnpacker();
    
    GZUnpacker->haltOnError( true ); // stop on fail (manual restart/reset required)
    GZUnpacker->setupFSCallbacks( targzTotalBytesFn, targzFreeBytesFn ); // prevent the partition from exploding, recommended
    GZUnpacker->setGzProgressCallback( BaseUnpacker::defaultProgressCallback ); // targzNullProgressCallback or defaultProgressCallback
    GZUnpacker->setLoggerCallback( BaseUnpacker::targzPrintLoggerCallback  );    // gz log verbosity
    
    // expand one file
    if( !GZUnpacker->gzExpander(tarGzFS, "/gz_example.gz", tarGzFS, "/gz_example.jpg") ) {
      Serial.printf("gzExpander failed with return code #%d", GZUnpacker->tarGzGetError() );
    }
  11. Pack files to .tar

    master

    Use TarPacker to create .tar archives. You can first collect file entities using TarPacker::collectDirEntities() and then pack them to either a Stream or a destination fs::FS file.

    std::vector<TAR::dir_entity_t> dirEntities;
    TarPacker::collectDirEntities(&dirEntities, &LittleFS, "/folder/to/pack");
    auto packedSize = TarPacker::pack_files(&LittleFS, dirEntities, &LittleFS, "/my.archive.tar");