Flatdata

Performance and Storage Principles

flatdata achieves efficiency through several core technical strategies:

• Bit-Level Storage: It allows operating at the bit level to minimize footprint. For example, boolean attributes use single bits rather than full bytes, and small integer ranges can be packed into spare bits of existing fields.
• Zero-Copy Access: No data is accessed or copied until the moment it is explicitly used. When you access a field, only the necessary bytes are read.
• Memory-Mapped Files: It leverages memory-mapped files and the OS page cache to manage data loading and access efficiently.
• Platform Independence: Data is stored in little-endian order using a custom platform-independent structure alignment (one byte).
• Efficient Implementation: The core reading and writing logic is implemented in C++ using templates and inlining to ensure minimal instructions per data access.

The performance of flatdata-py depends heavily on the access pattern used.

Iteration (Slow for bulk operations)

Iterating over a vector (e.g., for x in archive.links) yields one Python object per element. This incurs significant per-element overhead because each field access must unpack bits from the memory-mapped data.

Vectorized Access (Fast for bulk operations)

For high-performance bulk operations, use vectorized methods that read fields directly into NumPy arrays or Pandas DataFrames. This avoids the per-element Python object overhead.

Best Practices:

• Single column access: Use vector.field_name to get a Pandas DataFrame of a specific field. This is most efficient when you only need one or a few fields.
• Full row access: Use vector.to_numpy() or vector.to_data_frame() when you need all fields in a vector at once.
• Slicing: Slices work with vectorized methods (e.g., vector[1000:2000].to_numpy()).
• Random access: Use vector[i].field for accessing individual elements by index.

Note: The underlying data is memory-mapped; the OS pages it from disk on demand. Vectorized results are materialized as NumPy arrays in RAM.

# SLOW: Iterating over a vector
count = sum(1 for x in archive.links if x.speed_limit > 100)

# FAST: Single column access (returns a pandas DataFrame)
df = archive.links.speed_limit
count = len(df[df['speed_limit'] > 100])

# FAST: Full NumPy structured array
arr = archive.links.to_numpy()
count = int(np.sum(arr['speed_limit'] > 100))

# FAST: Slices work too
arr = archive.links[1000:2000].to_numpy()
df = archive.links[::10].to_data_frame()

Decorations are first-class citizens in the schema and are validated during archive opening. They declare additional properties for entities.

Common Decorations:

• @optional: Applied to resources. If the resource is missing, the archive can still be opened.
• @optional(<name>): Applied to fields. Marks a specific constant value as a sentinel (the none value), making the field optional in many language backends.
• @const(<name>): Applied to fields to indicate which constant can appear there.
• @explicit_reference(source_field, target_resource): Declares a relationship similar to a Foreign Key between a field and a resource.
• @bound_implicitly(resource_a, resource_b): Groups multiple resources together to represent a single entity (useful for data locality).
• @range(name_of_range_attribute): Simplifies retrieving slices of data from a vector based on an index attribute in a preceding vector.

When to use flatdata

flatdata is designed for high-performance data access patterns where data is read much more frequently than it is updated. It is most effective when:

• Infrequent Updates: Your data updates infrequently (e.g., every few hours), and the update rate is significantly lower than the time required for serialization.
• Full Rebuilds Allowed: You can afford to recreate the entire flatdata archive whenever a structural update is required.
• RAM-Compatible Data Size: Your data can be serialized into a mid-sized archive (e.g., 50GB to 200GB) that is comparable to the available RAM on your machine.
• High Read Volume: Your data is accessed many times, substantially more often than it is updated.
• Cache Optimization: You want to optimize for cache-friendliness by sorting data, minimizing index sizes, grouping related data, and separating infrequent data.

Ideal Data Types

While flatdata does not impose a specific structure, it is optimized for:

• Structures containing categorical and numerical data.
• Data with implicit or explicit references (one-to-one, one-to-many, many-to-many) implemented via associative vectors.
• Low-frequency attributes that only apply to a small subset of entities.
• Arbitrary metadata attached to entities.

When your schema uses import statements, you must generate each file separately. Imported types are referenced via include/import directives in the generated code rather than being re-emitted in the main file.

For C++:

1. Generate the shared types first: flatdata-generator -s schema/types.flatdata -g cpp -O schema/types.h
2. Generate the main schema (which will #include "types.h"): flatdata-generator -s schema/main.flatdata -g cpp -O schema/main.h

For Rust: Follow the same approach. Each imported file becomes its own module with pub use re-exports connecting the namespaces.

Note for Python and Dot: Python and Dot generators emit all types monolithically. You do not need to generate imported files separately; all definitions are included in the single output file.

# Generate shared types
flatdata-generator -s schema/types.flatdata -g cpp -O schema/types.h

# Generate main schema (will #include "types.h")
flatdata-generator -s schema/main.flatdata -g cpp -O schema/main.h