DOCSEARCHbyAlgolia
Submit Docs

JuiceFS

repository·main·Indexed 12 days ago

https://github.com/juicedata/juicefs

A high-performance, cloud-native, POSIX-compatible distributed file system that separates data stored in object storage (e.g., Amazon S3, MinIO) from metadata stored in databases like Redis, MySQL, TiKV, or SQLite. It features HDFS and S3 compatibility, a Kubernetes CSI driver, and strong consistency, making it suitable for big data analytics, machine learning, and shared workspaces.

Tokens
250.6K
Snippets
735
Records
1.1K
Agent score
90%

What's inside JuiceFS

  1. Common JuiceFS Usage Scenarios

    main

    JuiceFS is optimized for the following use cases:

    • Big Data: Integrates with Spark, Presto, and Hive via HDFS compatibility.
    • Machine Learning: Provides shareable POSIX storage for ML/DL frameworks to improve team data management.
    • Kubernetes: Uses the CSI Driver to provide decoupled persistent storage for pods, enabling stateless applications and data sharing between containers.
    • Shared Workspace: Enables concurrent read/write operations across any host via POSIX mounting.
    • Data Backup: Acts as a scalable, aggregated storage target for data from multiple hosts.

  2. Compare JuiceFS and SeaweedFS features

    main

    JuiceFS and SeaweedFS are both high-performance distributed file systems using the Apache License 2.0, but they differ significantly in architecture and feature sets.

    Key Differences:

    • Metadata Engine: JuiceFS Community supports various databases; Enterprise uses a proprietary high-performance engine. SeaweedFS supports up to 24 databases.
    • Atomicity: JuiceFS strictly ensures operation atomicity via database transactions. SeaweedFS has weaker atomicity guarantees (e.g., rename operations might not use locks under high load).
    • Data Storage: JuiceFS relies on Object Storage (like S3) for data. SeaweedFS uses a self-contained Volume/Master server architecture.
    • POSIX/HDFS Compatibility: JuiceFS provides full POSIX and HDFS compatibility. SeaweedFS provides basic/partial support for both.
    • File Splitting: JuiceFS uses 64MB logical chunks split into 4MB physical blocks. SeaweedFS uses 8MB blocks.
    • Data Compression: JuiceFS supports global settings (LZ4 or Zstandard). SeaweedFS compresses based on file extensions/types.

  3. Key features of JuiceFS

    main

    JuiceFS provides several core capabilities for high-performance distributed storage:

    • POSIX Compatibility: Use it like a local file system with no changes required to existing applications.
    • HDFS Compatibility: Full support for the HDFS API, offering high metadata performance.
    • S3 Compatibility: Provides an S3 Gateway for S3-protocol access.
    • Cloud Native: Easily deployable in Kubernetes via the Kubernetes CSI driver.
    • Distributed Design: Supports simultaneous mounting on thousands of servers with high-concurrency read/write capabilities.
    • Strong Consistency: File modifications are immediately visible across all servers.
    • High Performance: Low latency and near-infinite throughput (dependent on object storage scale).
    • Data Security: Supports both encryption in transit and encryption at rest.
    • File Locking: Supports BSD locks (flock) and POSIX locks (fcntl).
    • Data Compression: Supports LZ4 and Zstandard algorithms to save storage space.

  4. Data Backup, Migration, and Recovery

    main

    JuiceFS provides tools and patterns for managing large-scale data lifecycle and movement:

    • Juicesync: A tool for cross-cloud data migration.
    • Backup & Recovery: Optimizing metadata backup/recovery performance and performing high-speed backups for large file counts (e.g., 100 million files).
    • Database Backups: Using JuiceFS to optimize MySQL (XtraBackup) and NGINX log archiving.
    • Elasticsearch: Implementing low-cost cloud backup storage for Elasticsearch.

  5. Navigate the JuiceFS source code structure

    main

    The JuiceFS repository is organized into several key directories that define its functionality:

    • cmd/: The entry point for all CLI commands. For example, the juicefs format command is implemented in cmd/format.go.
    • pkg/: The core implementation logic:
      • pkg/fuse/: The entry point for FUSE implementation, providing the abstract FUSE interface.
      • pkg/vfs/: Concrete FUSE interface implementations. Metadata requests are handled by pkg/meta, read requests by pkg/vfs/reader.go, and write requests by pkg/vfs/writer.go.
      • pkg/meta/: Implementations for all metadata engines:
        • pkg/meta/interface.go: Defines the interface for all metadata engine types.
        • pkg/meta/redis.go: Redis implementation.
        • pkg/meta/sql.go: Relational database interface and generic implementation (e.g., pkg/meta/sql_mysql.go for MySQL).
        • pkg/meta/tkv.go: KV-type database interface and generic implementation (e.g., pkg/meta/tkv_tikv.go for TiKV).
      • pkg/object/: Implementations for interfacing with various object storage providers.
    • sdk/java/: Implementation of the Hadoop Java SDK, which relies on the sdk/java/libjfs library via JNI calls.

  6. Use JuiceFS on AWS services

    main

    JuiceFS can be integrated with various AWS products depending on your use case:

    • Amazon EC2: Mount the JuiceFS file system directly onto EC2 instances.
    • Amazon EKS: Use the JuiceFS CSI driver to provide storage to Kubernetes pods.
    • Amazon EMR: Use the JuiceFS Hadoop Java SDK for big data workloads.

    Note: For Amazon EKS, the JuiceFS CSI driver is not supported on Fargate nodes. Use EKS Managed Node Groups or Self-managed nodes instead.

  7. Big Data Use Cases and Architectures

    main

    JuiceFS is widely used in big data environments to achieve storage-compute separation, reduce costs (up to 85%), and provide HDFS-compatible performance using object storage. Key application areas include:

    • HDFS Replacement: Migrating from HDFS to JuiceFS on object storage to handle small files more efficiently and reduce costs.
    • Big Data Platforms: Building cloud-native big data platforms with storage-compute separation (e.g., for Spark, Flink, and Kylin).
    • Database/Search Engine Storage: Managing hot/cold data for Elasticsearch, ClickHouse, and Apache HBase.
    • Data Lakes: Exploring and implementing JuiceFS within data lake storage architectures.
    • Performance Optimization: Accelerating Spark Shuffle and optimizing Kylin 4.0 storage performance.

  8. Deploy JuiceFS on AWS (Overview)

    main

    JuiceFS can be integrated into the AWS ecosystem using three primary methods depending on your compute service:

    1. Amazon EC2: Mount the JuiceFS file system directly onto the instance.
    2. Amazon EKS: Use the JuiceFS CSI Driver to provide storage to Kubernetes pods.
    3. Amazon EMR: Use the JuiceFS Hadoop Java SDK for integration with the Hadoop ecosystem.

    A JuiceFS file system requires two components:

    • Object Storage: For data (e.g., Amazon S3).
    • Metadata Engine: A database for metadata (e.g., Amazon MemoryDB or Amazon RDS).

    When using AWS managed services, it is recommended to use Amazon MemoryDB for Redis as the metadata engine because it provides the strong consistency guarantees required by JuiceFS, whereas Amazon ElastiCache may not.

  9. JuiceFS Architecture Overview

    main

    JuiceFS is a distributed file system composed of three primary components:

    1. JuiceFS Client: Handles all file I/O, background tasks (like chunk compaction and trash expiration), and interacts with both the Data Storage and the Metadata Engine. The client supports multiple access methods:
      • FUSE: Mounts the file system as a POSIX-compatible local drive.
      • Python SDK: Direct file system access within Python processes (includes native fsspec support for frameworks like Ray).
      • Windows Client: Provides a local file system experience on Windows.
      • Hadoop Java SDK: Replaces HDFS for Hadoop workloads.
      • Kubernetes CSI Driver: Provides scalable storage directly to Kubernetes clusters.
      • S3 Gateway: Allows access via S3-compatible tools (AWS CLI, s3cmd, MinIO client).
      • WebDAV Service: Access via HTTP using a RESTful-like API.
    2. Data Storage: Files are split into chunks and uploaded to object storage. Supported backends include public cloud object storage and private solutions like OpenStack Swift, Ceph, and MinIO.
    3. Metadata Engine: Stores file metadata (filenames, sizes, permissions, directory structures, locks, etc.) and data indices (allocation, reference counts). Supported engines include Redis, TiKV, MySQL/MariaDB, PostgreSQL, and SQLite.

  10. Compare JuiceFS and Alluxio features

    main

    JuiceFS and Alluxio are both data access layers, but they differ significantly in architecture and capabilities. JuiceFS uses a block-based storage format with a 4 MiB cache granularity, making it highly efficient for random reads (e.g., Parquet/ORC). Alluxio uses an object-based format with a 64 MiB cache granularity.

    Key advantages of JuiceFS include:

    • Full POSIX Compatibility: Supports symbolic links, truncate, fallocate, append, xattr, mkfifo, mknod, and utimes.
    • Atomic Metadata Operations: Uses Redis transactions to ensure operations like renaming and deleting files are atomic.
    • Strong Consistency: The metadata service is the single source of truth, preventing inconsistencies between the metadata engine and object storage.
    • Data Security & Efficiency: Supports LZ4/Zstandard compression and encryption both in transit and at rest.
    • Operational Simplicity: Can use managed cloud databases as a metadata engine to reduce operational overhead.

  11. Data Sharing and AI Environment Optimization

    main

    JuiceFS enables efficient data sharing across different environments and tools, particularly for AI development:

    • Ollama + JuiceFS: Implementing a 'pull once, run anywhere' model for model weights.
    • Conda + JuiceFS: Enhancing the sharing capabilities of AI development environments.
    • Stable Diffusion: Sharing and storing model data in the cloud.
    • Milvus: Building distributed Milvus clusters on top of JuiceFS.

  12. Alternative ways to access JuiceFS data

    main

    Beyond standard mounting, JuiceFS supports several integration methods for different environments:

    • Kubernetes CSI Driver: Use JuiceFS as the storage layer for Kubernetes clusters.
    • Hadoop Java SDK: Access JuiceFS via a Java client that is highly compatible with the HDFS interface.
    • S3 Gateway: Access JuiceFS data using the S3 protocol.
    • Docker Volume Plugin: Use JuiceFS as a volume within Docker containers.
    • WebDAV Gateway: Access JuiceFS via the WebDAV protocol.