ratarmount

Ratarmount is a tool that provides random access to archived resources (like TAR files) without requiring full extraction. It achieves this by collecting file positions and building indices containing seek points. It then mounts the archive using FUSE (via mfusepy), allowing you to browse and read files as if they were on a standard filesystem.

Key capabilities include:

• Random Access: Fast seeking inside compressed streams (bzip2, gzip, xz, zstd).
• Highly Parallelized: Uses all available cores for decompression by default (controllable via -P <cores>).
• Recursive Mounting: Automatically mounts nested archives (TARs inside TARs) into subdirectories.
• Union/Bind Mounting: Supports merging multiple TARs, folders, or compressed files under a single mountpoint.
• Write Overlay: Allows redirecting changes to a specific folder so deletions and modifications can be tracked and potentially applied back to the archive.
• Remote Support: Mounts archives via URIs using fsspec (e.g., HTTP, SFTP, S3, Dropbox, GitHub).

Ratarmount is designed for high-performance random access to large TAR archives using FUSE. Key performance characteristics include:

• Subsequent Mounts: Extremely fast (under 1 second) if a preexisting index sidecar file is available. Subsequent mounts avoid the heavy cost of initial index creation.
• File Access: Reading file contents is significantly faster than alternatives like archivemount and does not scale poorly with archive size.
• Memory Usage: Generally low and does not grow linearly with archive size. Note that the zstd backend uses mmap, and the gzip backend's memory usage grows with archive size.
• Parallelism: Using the -P flag enables parallel decoding for bzip2 and xz archives, which can be significantly faster on multi-core processors.
• Metadata Operations: Operations like find on the mount point may be slower than direct file reads due to the overhead of high-level FUSE interfaces.
• Recursive Mounting: Unlike some alternatives, ratarmount supports mounting TAR files that are contained within other TAR files (recursive mounting).

The ratarmountcore library provides a MountSource interface designed for FUSE integration. This interface allows for listing paths, retrieving file metadata, and accessing file contents.

There are several implementations of MountSource categorized by their source type:

Archive Implementations

• SQLiteIndexedTar: The most powerful implementation; provides fast access to files inside (even compressed) TAR archives using an index.
• RarMountSource: Uses rarfile to handle RAR archives.
• ZipMountSource: Uses the built-in zipfile module for ZIP archives.
• FolderMountSource: Treats an existing local folder as a mount source.

Functional/Layered Implementations

• UnionMountSource: Merges multiple MountSource implementations into a single, unified file hierarchy.
• FileVersionLayer: A wrapper that accepts <file>.version/<number> paths and calls the underlying MountSource with the specified version.
• AutoMountLayer: Recursively mounts archives found within a MountSource (similar to UnionMountSource).

To achieve high-performance metadata insertion, Ratarmount uses an intermediary unsorted table strategy. Instead of inserting directly into a table with a complex primary key (which incurs high overhead), it follows these steps:

1. Create a temporary table (files_tmp) with a simple integer primary key. This allows for constant-time insertions.
2. Insert all file metadata into this temporary table.
3. Create the final files table with the required (path, name) primary key.
4. Move data from the temporary table to the final table using INSERT INTO ... SELECT ... ORDER BY. Using ORDER BY during the transfer significantly improves performance by ensuring data is written to the final table in a sorted manner.
5. Drop the temporary table.

CREATE TABLE "files_tmp" (
    "id" INTEGER PRIMARY KEY,
    "path" VARCHAR(65535),
    "name" VARCHAR(65535)
);
INSERT INTO files VALUES
    (0,"abcdef", "ghijklmn"),
    (1,"opqrst","uvwxyz"),
    (2,"abcdef", "ghijklmn");
CREATE TABLE "files" (
    "path" VARCHAR(65535),
    "name" VARCHAR(65535),
    PRIMARY KEY (path,name)
);
INSERT INTO "files" (path,name)
    SELECT path,name FROM "files_tmp"
    ORDER BY path,name;
DROP TABLE "files_tmp";

When choosing between ratarmount and alternatives like archivemount, consider the following performance behaviors:

• File Seek Time: ratarmount provides true seeking. Unlike some archive mounters that emulate seeking by reading all preceding data (scaling linearly with TAR size), ratarmount access time is constant regardless of file position. For bzip2 compressed TARs, it seeks directly to the bzip2 block using index information.
• Memory Footprint: For TAR files with more than 20k files, ratarmount maintains a relatively constant memory footprint (approx. 30MB on some systems) because the index is written to disk during creation. In contrast, some alternatives keep the entire index in memory.
• Mounting Time: ratarmount creates an index file next to the TAR file. Subsequent mounts are extremely fast because the index is reused. While initial mounting for very small files might be slower than some competitors, ratarmount scales much better for massive archives (e.g., 4M+ files).
• Metadata Retrieval: Currently, listing files (e.g., using find) may be slower in ratarmount compared to C-based alternatives due to the underlying use of Python and SQLite.