DOCSEARCHbyAlgolia
Submit Docs

flashDB Documentation

repository·master·Indexed 25 days ago

https://github.com/armink/flashdb

An ultra-lightweight embedded database for IoT and embedded products, featuring Key-Value (KVDB) and Time Series (TSDB) storage modes. It is designed for minimal RAM usage, power-off protection, and Flash wear leveling. The project includes support for various platforms including ESP32, ESP8266, STM32 (on-chip and SPI flash), and Linux, as well as the SFUD (Serial Flash Universal Driver) for abstracting SPI Flash hardware.

Tokens
18.1K
Snippets
68
Records
125
Agent score
82%

What's inside flashDB

  1. Overview of FlashDB features

    master

    FlashDB is an ultra-lightweight embedded database designed for resource-constrained environments. It provides two primary database modes:

    • Key-Value database: For general purpose storage of key-value pairs.
    • Time series database (TSDB): Optimized for storing and querying time-stamped data.

    Key characteristics include a small memory footprint and fast query speeds.

  2. Overview of FAL (Flash Abstraction Layer)

    master

    FAL is a Flash Abstraction Layer designed to manage and operate Flash devices and their partitions through a unified API. It provides an abstraction that decouples high-level components (like file systems, OTA, or NVM) from specific Flash hardware drivers.

    Key Features:

    • Supports statically configurable partition tables associated with multiple Flash devices.
    • Supports automatic loading of partition tables to avoid duplicate definitions in multi-firmware projects.
    • Lightweight and OS-independent; suitable for bare-metal platforms and bootloaders.
    • Unified interface improves driver reusability.
    • Includes Finsh/MSH test commands for byte-addressable Flash/partition operations (read/write/erase) via Shell.

  3. Overview of SFUD (Serial Flash Universal Driver)

    master

    SFUD is an open-source universal driver library for serial SPI Flash. It is designed to abstract the differences between various Flash brands and specifications (commands, capacities, etc.), allowing software to be reusable and extensible across different hardware platforms. This helps mitigate risks associated with Flash shortages or end-of-life (EOL) components.

    Key Features:

    • Supports SPI and QSPI interfaces.
    • Object-oriented design (supports multiple Flash objects simultaneously).
    • Highly configurable and can be trimmed for minimal footprint.
    • Supports 4-byte addressing.
    • Extensibility: Uses the JEDEC SFDP (Serial Flash Discoverable Parameters) standard to automatically detect Flash capacity, write granularity, erase commands, and address modes during initialization. If a Flash chip does not support SFDP, SFUD can fall back to a manual parameter table defined in the configuration.

    Resource Footprint:

    • Standard: RAM: 0.2KB, ROM: 5.5KB
    • Minimum: RAM: 0.1KB, ROM: 3.6KB

  4. Overview of FlashDB database modes

    master

    FlashDB is an ultra-lightweight embedded database designed for IoT products. It provides two distinct database modes depending on your data requirements:

    1. Key-Value Database (KVDB): A non-relational database that stores data as Key-Value pairs. It is ideal for storing product parameters, user configuration information, and managing small files. It supports string and blob types and allows for incremental upgrades after firmware updates.
    2. Time Series Database (TSDB): A database that stores data in chronological order using timestamps. It is optimized for high-volume data insertion and querying. It is ideal for storing structured dynamic data (e.g., sensor readings like temperature/humidity) and recording operational logs or error alerts.

  5. Key features of FlashDB

    master

    FlashDB includes several features designed for embedded reliability and efficiency:

    • Minimal Footprint: Extremely low RAM usage (nearly zero).
    • Multi-instance Support: Supports multiple partitions and instances to manage large datasets and reduce retrieval times.
    • Wear Leveling: Implements wear balance to extend the service life of the Flash memory.
    • Power-off Protection: Ensures high reliability and data integrity during power loss.
    • KV Data Types: Supports both string and blob types for Key-Value operations.
    • Incremental Upgrades: KVDB content supports automatic upgrades following firmware updates.
    • TSDB Record Management: Allows modifying the status of individual TSDB records for easier data management.

  6. Understand FlashDB core concepts

    master

    FlashDB consists of several key data models and access patterns:

    • Key-Value Database (KVDB): A non-relational database storing data as unique key-value pairs. It offers simple operations and high scalability.
    • Time Series Data (TSDB): A database designed for data stored in time sequence. It features timestamps and is optimized for high-performance insertion and querying.
    • Time series log (TSL): Represents an individual record within a TSDB.
    • Blob: A field type used in both KVDB and TSDB to store binary data, allowing compatibility with any variable type.
    • Iterator: An interface used to traverse every element in both TSDB and KVDB containers without needing to know the underlying implementation details.

  7. Simulate Timestamps for TSDB

    master
    In environments without a real-time clock (RTC) or network clock, you can simulate advancing time for TSDB. Use fdb_tsdb_control with the FDB_TSDB_CTRL_GET_LAST_TIME command to retrieve the last used timestamp. By incrementing this value manually in your get_time callback, you ensure that each new record has a unique, non-repeating timestamp.

  8. Build and run the FlashDB Linux demo

    master

    To test the KVDB (Key-Value Database) and TSDB (Time Series Database) implementations on a Linux platform, follow these steps:

    1. Build: Execute the make command in the terminal. The compiled executable will be placed in the out directory.
    2. Run: Navigate to the out folder and execute the ./FlashDBLinuxDemo binary. You can run this multiple times to observe different behaviors.
    3. Verify: The demo outputs its logs directly to the terminal for inspection.

  9. Use FinSH/MSH test commands for FAL

    master

    If you have enabled Finsh/MSH on RT-Thread, you can use the fal command to interact with Flash devices and partitions directly from the shell. This is useful for verifying driver integrity, reading/writing raw data, and performing performance benchmarks.

    To see the complete list of available commands, enter fal in the MSH shell.

    msh />fal

  10. Set up the STM32F405RG demo environment

    master

    Follow these steps to run the STM32F405RG on-chip flash demo:

    1. Connect Serial Terminal: Connect the board to your PC via the serial port and open a PC serial terminal.
    2. Open Project:
      • Keil MDK: Open RVMDK\FlashDB.uvprojx.
      • RT-Studio: Use the import wizard to import the project folder.
    3. Build and Download: Build the project and download the firmware to your board.
    4. Verify: Check the logs outputting to your PC serial terminal.

  11. Compile and flash the ESP8266 SPI-Flash demo

    master

    Follow these steps to build and deploy the demo to your ESP8266 development board using a serial cable.

    1. Navigate to the demo directory: cd esp8266_spi_flash
    2. Compile the project: make all
    3. Connect your board to your PC via serial cable.
    4. Grant permissions to the serial port: sudo chmod 777 /dev/ttyUSB*
    5. Flash the firmware: make flash
    6. Open the monitor to view logs: make monitor

    The demo starts a FreeRTOS task to run the FlashDB routine and will restart after 1000 seconds.

    cd esp8266_spi_flash
make all
sudo chmod 777 /dev/ttyUSB*
make flash
make monitor