Cloud Storage FUSE

The tool expects a specific subdirectory named gcsfuse-shared-chunk-cache within your provided -cache-dir. It automatically detects this structure, which uses SHA256-based hashing for organization.

Structure Format: <cache-dir>/gcsfuse-shared-chunk-cache/<2-char>/<2-char>/<full-hash>/<start>_<end>.bin

Note: This is distinct from the standard GCSFuse file cache, which uses the gcsfuse-file-cache subdirectory.

The tool uses a two-phase eviction process to ensure that concurrent reads of chunked files are not disrupted: files are renamed to .bak during the current run and are only physically deleted during the next run.

Execution Workflow:

1. Cleans up leftover .bak files from previous runs.
2. Scans the cache directory for .bin files, collecting their atime and size.
3. If the total size is below the -target-size-mb, the tool exits.
4. Sorts files by atime and selects the oldest files for expiration.
5. Renames selected files to .bak.
6. Removes .tmp files older than 1 hour.
7. Cleans up empty directories.

The garbage collector requires a file system (such as NFS or most POSIX-compliant systems) that supports:

1. Atomic rename: Used to safely expire cache files by renaming .bin to .bak without disrupting concurrent reads.
2. Access time (atime) tracking: The LRU eviction algorithm relies on file access times to identify the least recently used chunks.

Cloud Storage allows an object named foo and an object named foo/ to exist simultaneously. Because traditional Linux filesystems do not allow this, Cloud Storage FUSE resolves the conflict by renaming the file/symlink to foo (where is the U+000A line feed character).

When a conflict occurs:

• The directory foo will appear in listings.
• The file or symlink will appear as `foo

`.

• Cloud Storage FUSE uses the line feed character because it is illegal in GCS object names, ensuring the mapping is unambiguous.

Cloud Storage FUSE does not support all standard POSIX filesystem features. Key limitations include:

• Directory Renaming: Only supported atomically in Hierarchical Namespace Buckets. In Flat buckets, it is unsupported unless --rename-dir-limit is used (see guide).
• Permissions/Ownership: File and directory permissions and ownership cannot be changed.
• Timestamps:
  • Modification times (mtime) are tracked for files, but not for directories.
  • Other timestamps like ctime and atime are not tracked. Requests to change them will appear to succeed, but the results are unspecified.

GCSFuse implements custom retry logic for Folder APIs and GetStorageLayout calls to mitigate stall issues. This is applied to all buckets for GetStorageLayout and to all control client operations for rapid buckets.

Default parameters:

• Retry Deadline: 30 seconds
• Total Budget: 5 minutes
• Initial Backoff: 1 second

This strategy uses exponential backoff with jitter and includes stall detection with a deadline per attempt.

Certain path segments like //, /./, or /../ are valid object names in Google Cloud Storage but are not supported by the Linux filesystem. From v3.6.0 onwards, Cloud Storage FUSE handles these as follows:

1. Listing: These objects are hidden from file listings to prevent system errors or crashes.
2. Rename/Delete: Directory-level operations (like deleting a parent directory) still apply to all contained objects, ensuring that these unsupported objects are not orphaned or left behind.

Cloud Storage FUSE provides close-to-open and fsync-to-open consistency.

• Close/fsync: Once a file is closed or synced, a new generation of the object is created in Cloud Storage (provided the object hasn't changed since it was last observed).
• Open: An open call guarantees to observe a generation at least as recent as all generations created before the open was called.

Example of Conflict

1. Machine A and B both open a file containing 'ABC'.
2. Machine A modifies it to 'ABC-123' and closes/syncs it. The cloud now has 'ABC-123'.
3. Machine B (still holding the old version) modifies its local copy to 'ABC-XYZ' and tries to close/sync.
4. Because Machine A's write won, Machine B's file descriptor will receive an ESTALE error to prevent silent data loss.

Cloud Storage FUSE supports concurrent reads and writes to different objects in the same bucket. Concurrent writes to the same object from the same mount behave like a native file system.

Warning: Multiple Mounts When different mounts (e.g., different machines) attempt to write to the same object, the first mount to flush its changes wins. Other mounts that have not updated their local file descriptors will encounter a syscall.ESTALE error when attempting to save edits due to precondition checks.

Best Practice To ensure data consistency and avoid ESTALE errors, do not allow multiple sources to modify the same object simultaneously.

Cloud Storage FUSE v3 automatically optimizes its configuration when running on specific high-performance Google Cloud machine types. This maximizes performance for demanding workloads by effectively utilizing the machine's capabilities.

Note: Any values manually set at the time of mount will override these automatic defaults.