ReductStore

The Lifecycle Module implements background lifecycle policies for buckets to automate data maintenance. It supports two primary actions:

• delete: Removes records older than a specified older_than duration.
• compress: Recompresses eligible blocks older than a specified older_than duration.

The module is organized into three layers:

1. Repository: Manages lifecycle definitions, validates settings, and controls the lifecycle of workers.
2. Task: A worker loop that runs a specific policy on a schedule and emits system events.
3. Action: The execution layer that performs the actual work (deleting or compressing) against storage and reports progress.

For primary nodes, definitions are persisted in .lifecycles within the data directory. For replica nodes, the module operates in a read-only mode where no workers are started.

The LifecycleTask runs an inner loop with specific retry logic to handle sparse data:

• If a run completes and returns affected_records == 0 but caught_up == false, the worker does not wait for the full scheduled interval. Instead, it retries after 100 ms.
• This allows the task to quickly advance through empty time windows until it reaches the current data cutoff.
• Once the task either finds records to process (affected_records > 0) or reaches the cutoff (caught_up == true), it returns to its normal scheduled interval.

ReductStore is a high-performance, time-indexed object storage system designed for robotics and industrial IoT.

Use ReductStore if:

• You have a continuous stream of binary data (images, video, sensor payloads, logs, ROS bags).
• You need to query this data by both time range and metadata (labels).
• You need to manage data lifecycles (e.g., compression or deletion after a certain period).
• You want to replicate only specific subsets of data (filtered by labels) to the cloud to save bandwidth.

Do NOT use ReductStore if:

• You only have numerical data (use a Time-Series Database instead).
• You only have blob data without a need for historical/time-indexed access (use standard object storage).
• You need a message broker (ReductStore is for storage and streaming, not message queueing).

The fastest way to run ReductStore is using Docker. You need to create a local data directory and ensure it has the correct permissions for the container user (UID 10001).

1. Create a data directory: mkdir -p ./data
2. Set permissions: sudo chown -R 10001:10001 ./data
3. Run the container, mapping port 8383 and providing an RS_API_TOKEN via environment variable.

mkdir -p ./data
sudo chown -R 10001:10001 ./data
docker run -p 8383:8383 -v ${PWD}/data:/data \
  -e RS_API_TOKEN=my-secret-token reduct/store:latest

You can run ReductStore directly on Linux by downloading the binary from the latest release. You must specify the data path using the RS_DATA_PATH environment variable and provide an RS_API_TOKEN.

curl -LO https://github.com/reductstore/reductstore/releases/latest/download/reductstore.x86_64-unknown-linux-gnu.tar.gz
tar -xzf reductstore.x86_64-unknown-linux-gnu.tar.gz
mkdir -p ./data
RS_DATA_PATH=./data RS_API_TOKEN=my-secret-token ./reductstore

The ReductStore HTTP API supports specialized operations by appending suffixes to the entry path. These suffixes allow you to perform batch operations or complex queries on an entry instead of standard single-entry CRUD operations.

Batch Operations

Append /batch to the entry path to trigger batch processing for reading, writing, updating, or removing records.

• Read Batch: GET /{bucket_name}/{entry_name}/batch
• Write Batch: POST /{bucket_name}/{entry_name}/batch
• Update Batch: PATCH /{bucket_name}/{entry_name}/batch
• Remove Batch: DELETE /{bucket_name}/{entry_name}/batch

Query Operations

Append /q to the entry path to perform queries or targeted removals using a JSON body.

• Query/Remove via Query: POST /{bucket_name}/{entry_name}/q
  • Uses a QueryEntry JSON body to specify the QueryType (e.g., Query or Remove).

ReductStore can automatically attach metadata from system meta entries (e.g., entry/$meta) to your extension queries. If a meta entry exists for a matching pattern, its contents are parsed as JSON and injected into the extension's configuration under an attachments key.

To use this, ensure your meta entries have a label key that matches the extension name or the desired attachment key. The system will collect up to MAX_META_ATTACHMENTS (100) records from the meta entry to build the attachment object.

The processing_interval is an optional setting used to bound the maximum span of data time a single lifecycle run processes. This is useful for keeping individual runs computationally cheap on buckets with large backlogs.

• Purpose: Limits the data window per run.
• Requirement: Must be a positive duration.
• Behavior: When set, it replaces the default 24 * interval window and also bounds the full-range window used when system events are disabled.