DOCSEARCHbyAlgolia
Submit Docs

JuiceFS Documentation

website·Indexed 34 days ago

https://juicefs.com/docs/

A high-performance cloud native file system. Documentation covers installation, command-line interface reference (including `juicefs auth`, `gc`, and `fsck`), Python SDK, and deployment options for Kubernetes, Windows, Samba, NFS, and WebDAV. Includes guides on multi-level caching, file import and conversion, and performance benchmarks against HDFS and OSS.

Tokens
686.8K
Snippets
2.5K
Records
4.8K
Agent score
65%

What's inside JuiceFS

  1. Overview of JuiceFS Trash functionality

    JuiceFS provides a trash feature where deleted files are preserved for a specific period before permanent deletion. These files continue to incur billing costs as data remains in object storage.

    Key Details:

    • Default Retention: Free edition is 1 day; Professional edition is 7 days. Retention can be modified in the JuiceFS Console settings.
    • Scope: Trash protects files and data modifications (including overwrite writes).
    • Limitations:
      • Directories are deleted immediately (though files within them are trashed and original directory structures are recreated upon restoration).
      • Empty directories cannot be recovered.
      • Symbolic links (symlinks) cannot be moved to the trash and must be rebuilt.
      • Files created via juicefs import are not eligible for the trash.
    • Requirement for Cleanup: At least one JuiceFS client (FUSE, Hadoop Java SDK, or S3 Gateway) must remain mounted and running for the background tasks to expire and release trashed data from object storage.

  2. Overview of JuiceFS Trash functionality

    JuiceFS automatically enables Trash upon file system creation to protect against accidental deletion and data modification (including content overwrites). Deleted files are kept in object storage for a specified period before permanent deletion.

    Key Constraints:

    • Billing: Trash files are billable as they remain in object storage.
    • Expiration: Default expiration is 1 day for the free plan and 7 days for the professional plan (configurable in the JuiceFS Console).
    • Garbage Collection: At least one JuiceFS Client (FUSE mount, Hadoop Java SDK, or S3 Gateway) must be running for trash expiration and garbage collection to function; otherwise, files will not be expired.

  3. Overview of JuiceFS architecture for AI workloads

    JuiceFS is a distributed file system designed for high-throughput, high-concurrency AI training and inference. It uses a decoupled architecture that separates data from metadata to avoid the tight coupling between throughput and capacity found in traditional parallel file systems.

    • Data Storage: Stored in object storage (e.g., AWS S3). Data is divided into 4 MB blocks to reduce write amplification and improve efficiency.
    • Metadata Storage:
      • Enterprise Edition: Uses a Raft-based clustered metadata engine.
      • Community Edition: Supports databases such as Redis and TiKV.
    • Client-Side Processing: Protocol conversion and cache acceleration are handled by clients on each application node (via JuiceFS CSI for Kubernetes or direct host access), eliminating single points of failure and improving resource utilization.

  4. Overview of JuiceFS Cloud Service

    JuiceFS Cloud Service is a managed, high-performance file storage service designed for public clouds. It is specifically tailored for large-scale AI model training and managing tens of billions of files using an in-house distributed metadata storage engine. Key service guarantees include a 99.95% service availability SLA and 99.99999999% (10 nines) data reliability SLA.

  5. Overview of JuiceFS quota implementation and scope

    JuiceFS implements a distributed quota system that balances performance and consistency using a model of pre-write incremental estimation, local delta accumulation, and periodic background synchronization. This eventual consistency model minimizes overhead on the write path. Quota control is supported across multiple dimensions to enable multi-tenant isolation and resource management, including:

    • Total file system capacity
    • Directory-level quotas
    • User-level quotas
    • User group-level quotas

  6. Overview of JuiceFS Data Replication

    JuiceFS supports asynchronous data replication to synchronize data from a 'primary region' (the object storage region specified during filesystem creation) to a 'target region'. The target region can be within the same cloud provider or a different one.

    Key use cases include:

    • Cross-region data sharing: Improving object storage performance for clients mounting JuiceFS in different regions.
    • Object storage disaster recovery: Manually switching to the target region (by modifying the --bucket option and remounting) if the primary region fails.
    • Smooth migration: Writing to two buckets simultaneously to migrate underlying object storage services.
    • Mirror Filesystem support: Used in conjunction with mirror filesystems to optimize metadata and data access.

    Constraint: Data replication currently only supports 'one-to-one' replication. For 'one-to-many' synchronization, use juicefs sync or the Mirror feature.

  7. Overview of JuiceFS S3 Gateway

    JuiceFS S3 Gateway (based on MinIO Gateway) implements the S3 API, allowing users to access and manage data in a JuiceFS file system using any S3-compatible client. Unlike standard object storage, JuiceFS provides a distributed file system with a unique data format, multi-protocol access, caching capabilities, and capacity quotas.

  8. Overview of JuiceFS for AI Model Training

    JuiceFS is an open-source, cloud-native distributed file system compatible with POSIX, HDFS, and S3 APIs. It uses a decoupled architecture where metadata is stored in a dedicated metadata engine (e.g., Redis, MySQL) and file data is stored in object storage. This design allows AI teams to scale storage independently of compute, maintain POSIX compatibility for training frameworks, and efficiently manage large datasets and unstructured data (small files).

  9. Overview of JuiceFS Console API

    The JuiceFS Web Console API allows developers to perform any file system interaction available in the browser, such as creating file systems, browsing file lists, viewing the recycle bin, and setting ACLs.

    Important Limitation: The Console API cannot manipulate or modify files within the file system. To programmatically access or modify files, use the S3 Gateway instead.

  10. Overview of JuiceFS Enterprise Edition

    JuiceFS Enterprise is a high-performance distributed file system designed for independent deployment across cloud and on-premises data centers. It is optimized for scenarios requiring massive scalability, low latency, and multi-cloud data management.

  11. Overview of JuiceFS Mirror file systems

    The Mirror file system feature allows you to create one or more complete mirrors of a JuiceFS file system across different regions. Each mirror has the same content as the source but uses a different volume name and access credentials. This is used to maintain high performance across regions. Mirrors are billed separately. Starting from version 5.0, mirror file systems can be configured for both read-only and read-write access.

  12. Overview of JuiceFS Architecture

    JuiceFS is a cloud-native distributed file system that separates metadata from data storage. Its architecture comprises:

    • Metadata Engine: Stores file system metadata and data indexes. Supports various backends (Redis, TiKV, MySQL, PostgreSQL, FoundationDB) in Community Edition, and a self-developed high-performance service in Enterprise Edition.
    • Data Storage: Uses object storage (public cloud or on-premises like MinIO/Ceph RADOS) for data reliability and consistency.
    • Clients: Supports multiple access protocols including POSIX (FUSE), Hadoop SDK, CSI Driver, and S3 Gateway.