TinyFlux

TinyFlux provides a structured API for managing time-series data through several specialized modules. To get started with writing and querying data, refer to the getting-started guide. The API is organized into the following functional areas:

• TinyFlux Database API: Core database management and lifecycle.
• Point API: Handling individual data points.
• Queries API: Executing searches and data retrieval.
• Measurement API: Managing measurements (the logical grouping of data).
• Index API: Managing indices for optimized querying.
• Storages API: Managing the underlying storage backends.
• Utils API: General utility functions.

TinyFlux is designed to maintain a small memory footprint by avoiding loading entire datasets into memory.

• Indexing: It uses an internal index that is typically ~80% smaller than the full dataset, providing high query performance without the memory cost of an in-memory database.
• Updates and Removals: When performing updates or removals, TinyFlux evaluates items one at a time. It writes to temporary storage and then replaces the original storage with the updated version. It does not rewrite data in time-ascending order to avoid expensive memory or disk operations.
• Time-based Searches: Queries are optimized by searching a sorted container of timestamps held in memory, which allows for fast retrieval of candidate Points without exhaustive scans.

TinyFlux separates data into tags and fields to optimize performance and enforce type consistency:

• Tags (dict[str, str]): Use these for attributes with low cardinality (values that don't change frequently, like city or country). TinyFlux uses tags to index data efficiently.
• Fields (dict[str, int|float]): Use these for attributes with high cardinality (values that vary significantly, like temperature or pressure). Fields are not indexed.

Note: TinyFlux does not support complex types (like nested dictionaries or lists) as values. For document-based storage with complex types, use TinyDB instead.

A field set (or "fields") is a collection of field key and field value pairs belonging to a Point.

• Schema: TinyFlux is schemaless; a Point can have zero or more fields.
• Types: Field keys are strings, while field values are numeric (float or int in Python).
• Indexing: Unlike tags, field keys and values do not map to Points in a TinyFlux index. This is because the number of distinct field values is typically too high for efficient indexing.
• On Disk: Field sets occupy side-by-side columns. By default, field keys are prefixed with _field_.

# Example of adding a field set to a Point
Point(
    fields={
        "num_restaurants": 12,
        "num_boba_shops": 3,
    }
)

A measurement in TinyFlux is an abstraction analogous to a "table" in a traditional Relational Database Management System (RDBMS). It allows you to partition data within the same database file. When you access TinyFlux through a measurement, you use the same API as the main database, but all operations are automatically filtered to that specific measurement.

By default, TinyFlux uses a measurement named _default if no specific measurement is targeted.

TinyFlux builds an index upon initialization. If you insert a Point that is out of chronological order (i.e., its timestamp is earlier than existing points), the index becomes invalid (db.index.valid will return False).

If auto-index is set to True (the default), TinyFlux will attempt to rebuild the index during the next read operation, which can cause significant latency. To avoid this, always attempt to insert Points into TinyFlux in time-order.

from tinyflux import TinyFlux, Point
from datetime import datetime, timedelta, timezone

db = TinyFlux("my_db.csv")
t = datetime.now(timezone.utc)

# Valid insertion (in order)
db.insert(Point(time=t))
print(db.index.valid)  # True

# Invalid insertion (out of order)
db.insert(Point(time=t - timedelta(hours=1)))
print(db.index.valid)  # False

TinyFlux is optimized for high-speed, write-heavy workloads. It uses a default CSV store designed for nearly instantaneous appends, regardless of the file size.

Important Indexing Note: TinyFlux maintains an index for performance, but this index is invalidated if an insert occurs with a timestamp that precedes the most recent insert (out-of-order data). TinyFlux will not automatically rebuild the index during a write operation if invalidation occurs.

A tag set (or "tags") is a collection of tag key and tag value pairs belonging to a Point.

• Schema: TinyFlux is schemaless; a Point can have zero or more tags.
• Types: Both tag keys and tag values are strings.
• Indexing: Tag keys and values are mapped to Points using a hashmap in a TinyFlux index, which allows for highly efficient retrieval.
• Best Practice: In a well-designed database, the number of distinct tag values should be significantly lower than the number of distinct field values.
• On Disk: Tag sets occupy side-by-side columns (one for the key, one for the value). By default, tag keys are prefixed with _tag_.

# Example of adding a tag set to a Point
Point(
    tags={
        "city": "LA",
        "neighborhood": "Chinatown",
        "food": "good",
    }
)

When updating tags or fields, you can pass a Callable to perform complex logic. The callable receives the current dictionary of tags or fields and must return the updated dictionary.

Note on Deletion: Using a Callable to update tags or fields will not remove keys from the dictionary, even if you omit them from the returned dictionary. This follows the behavior of the Python dict.update() method. To actually remove keys, you must use the unset_tags or unset_fields arguments.

>>> # Update a specific tag value within the tag set
>>> Measurement = MeasurementQuery()
>>> def california_updater(tags):
...     if "state" in tags and tags["state"] == "CA":
...         return {**tags, "state": "California"}
...     else:
...         return tags
>>> db.update(Measurement == "US Metros", tags=california_updater)

A timestamp is the time associated with a Point.

• In Python: It is represented as a datetime object.
• Storage: When inserted into a TinyFlux database, it is automatically cast to a timezone-aware object in the UTC timezone.
• On Disk: It is serialized as an ISO 8601 formatted string and occupies the first column of the default CSV storage class.

# Example of a Point with a timestamp
from tinyflux import Point
from datetime import datetime, timezone

p = Point(time=datetime.now(timezone.utc), measurement="test")