JLD2.jl

JLD2 is a Julia package designed for saving and loading data. It provides a simple API for serializing complex nested structures and is optimized for performance by using the Julia compiler to generate efficient serialization code.

Key features include:

• HDF5 Compatibility: JLD2 files adhere to the HDF5 format specification, allowing them to be compatible with HDF5 tooling and H5 libraries in other languages. It can also read existing HDF5 files.
• Custom Serialization: Users can provide custom serialization procedures to control how specific data is stored.
• Data Upgrades: JLD2 includes mechanisms to handle data structure upgrades, allowing for post-processing during the load phase (e.g., when Julia types have changed since the data was saved).

JLD2 is built upon the HDF5 Format Specification and produces files compatible with the official HDF5 C library. This allows JLD2 files to be accessed by other HDF5-compatible tools and libraries, such as HDF5.jl in Julia or h5py in Python. You can also use standard HDF5 introspection tools like h5dump and h5debug to inspect JLD2 files.

Warning on Compatibility: General compatibility is only guaranteed for a specific list of basic types:

• Numbers (FloatXX, IntXX, and UIntXX)
• Booleans
• Strings
• Arrays of the types listed above

Other structures may be decodable but may require additional work to interface with external HDF5 tools.

To transform old data into a new struct format during loading, use JLD2.Upgrade(NewType) within the typemap dictionary.

When using Upgrade, JLD2 loads the fields of the old struct into a NamedTuple and then calls rconvert(::Type{NewType}, nt) to perform the conversion. You must implement a corresponding rconvert method for the new type to handle the data transformation.

# The new version of your struct
struct UpdatedStruct
    x::Float64
    y::Float64
    z::Float64
end

# Implement conversion from the old fields (as a NamedTuple)
JLD2.rconvert(::Type{UpdatedStruct}, nt::NamedTuple) = UpdatedStruct(Float64(nt.x), nt.y, nt.x*nt.y)

# Use Upgrade in the typemap to trigger rconvert
load("test.jld2", "data"; typemap=Dict("Main.OldStructVersion" => JLD2.Upgrade(UpdatedStruct)))

For convenience, JLD2 provides variants of the legacy macros that operate on entire scopes without requiring explicit variable names:

• @save <filename>: Writes all variables in the current module's global scope to the specified file.
• @load <filename>: Loads all variables from the specified file into the current scope. Note: When using @load without variable arguments, the filename must be provided as a string literal; you cannot select the file dynamically at runtime.

@save "example.jld2"
@load "example.jld2"

The JLDFile object is designed to mimic the Base.Dict API in Julia. When working with a JLDFile instance, you can interact with the stored data using standard dictionary methods. This allows you to treat a file on disk as if it were an in-memory dictionary of keys and values.

# Example of expected JLDFile behavior (mimicking Base.Dict)
keys(jldfile)
length(jldfile)
haskey(jldfile, "key_name")
isempty(jldfile)
get(jldfile, "key_name", default_value)
get!(jldfile, "key_name", value)

When saving non-default types (like Julia structs), JLD2 uses HDF5 compound datatypes. To ensure efficiency, JLD2 creates and commits these type definitions to a special _types/ group within the file. This allows the type definition to be written only once, while all subsequent instances of that struct simply reference the existing definition.

When inspecting a file with h5dump, you will typically see:

1. A _types group containing the compound datatype definitions (including metadata like julia_type).
2. The actual datasets containing your data, which reference the datatypes defined in _types.

using JLD2

struct MyCustomStruct
    x::Int64
    y::Float64
end

@save "test.jld2" a=MyCustomStruct(42, π)

While JLD2 is designed to allow files to be loaded across different operating systems and between 32-bit and 64-bit systems, many Julia struct types are inherently architecture-dependent.

Limitation: Moving data from a 64-bit system to a 32-bit system is only guaranteed to work for basic datatypes. Complex Julia structs may fail to load if their memory layout differs between architectures.

You can combine multiple filters by providing a vector of filter instances. This is useful for combining preprocessing filters (like Shuffle) with compression filters.

Note: Filters are applied in the order provided during compression, and in reverse order during decompression. Preprocessing filters like Shuffle() should typically come before compression filters.

using JLD2

# Combine Shuffle preprocessing with Deflate compression
filters = [Shuffle(), Deflate()]

jldopen("example.jld2", "w"; compress = filters) do f
    # Shuffle() reorders bytes to improve compression efficiency
    f["numeric_data"] = UInt.(rand(UInt8, 10000))
end