FileIO.jl

FileIO uses lazy loading to improve startup performance. Instead of loading all IO backends (like PNGFiles.jl or ImageIO.jl) when you run using FileIO, it only loads the specific backend required when you actually call load() or save().

While this makes using FileIO very fast, it can trigger a world age issue if you call load inside a function. This happens because the backend packages (and their dependencies, like IndirectArrays) are loaded into a 'newer' Julia world than the one where your function was compiled. When your function tries to interact with the newly loaded types, Julia may throw a MethodError stating that the applicable method is too new.

julia> using FileIO

julia> f() = size(load("indexed_image.png"))

# This might fail with a MethodError if the backend
# packages weren't loaded when f() was compiled:
# ERROR: MethodError: ... The applicable method may be too new ...

Streaming I/O allows users to obtain a decoded stream object that they can interact with using standard read or write calls.

Implementation Steps:

1. Define a Reader/Writer type: Create a struct (e.g., WAVReader) that wraps the underlying IO.
2. Implement Base.read or Base.write: Define how to process data through your wrapper.
3. Implement Base.close: This is critical for resource management.
   • If the user provided a File, your close method must close the underlying stream.
   • If the user provided a Stream, your close method must only perform internal cleanup and must not close the stream.
4. Implement loadstreaming and savestreaming: Provide implementations for both File and Stream types.

Shortcut: Implementing load/save via streaming

If you have already implemented loadstreaming and savestreaming, you can implement the standard load and save methods easily by using the Formatted abstract supertype (which covers both File and Stream).

struct WAVReader
    io::IO
    ownstream::Bool
end

function Base.read(reader::WAVReader, frames::Int)
    # read and decode audio samples from reader.io
end

function Base.close(reader::WAVReader)
    # do whatever cleanup the reader needs
    reader.ownstream && close(reader.io)
end

# Implementations for FileIO to call
loadstreaming(f::File{format"WAV"}) = WAVReader(open(f), true)
loadstreaming(s::Stream{format"WAV"}) = WAVReader(s, false)

# Shortcut to implement standard load/save using the streaming logic
function save(q::Formatted{format"WAV"}, data, args...; kwargs...)
    savestreaming(q, args...; kwargs...) do stream
        write(stream, data)
    end
end

function load(q::Formatted{format"WAV"}, args...; kwargs...)
    loadstreaming(q, args...; kwargs...) do stream
        read(stream)
    end
end

To implement standard file loading and saving, you should provide methods for both File and Stream types.

Implementation Requirements:

• load(f::File{format"FORMAT"}): Should open the file, handle magic bytes (e.g., by skipping them), and then call the load(s::Stream{format"FORMAT"}) method. This method must close the stream it opens (using do blocks is recommended).
• load(s::Stream{format"FORMAT"}; keywords...): Should perform the actual reading from the stream. The stream is assumed to be positioned after the magic bytes. This method must not close the stream.
• save(f::File{format"FORMAT"}, data): Should open the file for writing, write the necessary magic bytes, and then perform the save. This method must close the stream it opens.
• save(s::Stream{format"FORMAT"}; keywords...): Should perform the actual writing to the stream. This method must not close the stream.

module MyFileFormat

using FileIO

# Private load function for Files
function load(f::File{format"PNG"})
    open(f) do s
        skipmagic(s)  # skip over the magic bytes
        load(s)       # call the Stream-based load
    end
end

# Private load function for Streams
function load(s::Stream{format"PNG"}; keywords...)
    # s is already positioned after the magic bytes
    chunklength = read(s, UInt32)
    # ... read the rest
end

# Private save function for Files
function save(f::File{format"PNG"}, data)
    open(f, "w") do s
        write(s, magic(format"PNG"))
        # ... write the rest
    end
end

end # module MyFileFormat