DOCSEARCHbyAlgolia
Submit Docs

Fluid Documentation

repository·master·Indexed 24 days ago

https://github.com/fluid-cloudnative/fluid

A Kubernetes-native distributed dataset orchestrator and accelerator designed to speed up data-intensive applications like AI and Big Data through dataset abstraction and elastic caching. It provides integration with various storage systems via Dataset, ThinRuntime, and ThinRuntimeProfile resources, including support for Ceph, CubeFS, Curvine, GlusterFS, and NFS.

Tokens
118K
Snippets
271
Records
516
Agent score
83%

What's inside Fluid

  1. Overview of JindoCache acceleration capabilities

    master

    JindoCache (formerly JindoFSx) is a cloud-native data lake acceleration product provided by Alibaba Cloud. It provides acceleration for both data and metadata through configurable caching strategies.

    JindoCache is designed to support various data lake workloads by applying different read/write policies to specific CacheSet configurations, allowing for fine-grained control over which data is cached and how.

  2. What is Fluid?

    master

    Fluid is an open-source, Kubernetes-native distributed dataset orchestration and acceleration engine. It is designed for data-intensive applications in cloud-native scenarios, such as Big Data (e.g., Spark) and AI/Machine Learning (e.g., TensorFlow) workloads.

    Key capabilities include:

    • Native Dataset Abstraction: Provides functional support for data-intensive applications to enable efficient data access and reduce management costs.
    • Extensible Data Engine Plugins: Offers a unified access interface to integrate third-party storage via different Runtimes.
    • Automated Data Operations: Provides multiple operation modes integrated with automated operations systems.
    • Data Elasticity and Scheduling: Combines data caching technology with elastic scaling and data affinity scheduling to improve performance.
    • Runtime Agnostic: Supports native, edge, Serverless Kubernetes, and multi-cluster Kubernetes environments, making it suitable for hybrid cloud scenarios.

  3. Overview of Fluid's data access and Runtime Plugins

    master

    Fluid provides a cloud data access scheme for Kubernetes users, enabling efficient interaction with underlying storage systems through distributed cache engines.

    Fluid uses a Runtime Plugin architecture to support different cache engines. Currently, it supports:

    • JindoFS (Alibaba Cloud EMR)
    • Alluxio (open-source)
    • JuiceFS
    • Other cache engines

    For users who need to access custom or general storage systems without writing a full CSI plug-in or a Runtime Controller, Fluid provides ThinRuntime. ThinRuntime is designed to facilitate rapid access to other storage systems by reducing the development overhead required to interface with Kubernetes.

  4. Overview of JindoFS

    master

    JindoFS is a self-developed big data storage service designed for cloud-native environments, specifically optimized for Alibaba Cloud EMR. It provides a decoupled compute and storage architecture that is fully compatible with the Hadoop FileSystem interface. It supports major big data engines including Spark, Flink, Hive, MapReduce, Presto, and Impala.

    JindoFS operates in two primary modes:

    1. Block Storage Mode: Provides a cloud-native solution using JindoNameService for high-performance metadata management (comparable to HDFS NameNode) and JindoStorageService to ensure data persistence on OSS. It supports various storage strategies (e.g., local replicas + OSS, or local only) to balance cost and performance.
    2. Cache Mode: Builds a distributed cache service on local clusters to make remote data (from OSS or remote HDFS) feel "local." This optimizes network bandwidth limitations by caching remote data locally. This mode is transparent to upper-layer jobs; you can use existing oss:// or hdfs:// paths without modification.

  6. Overview of Fluid Dashboard

    master

    Fluid Dashboard is a visual interface designed for managing data-intensive applications. It provides a GUI to lower the barrier for CLI operations and improve management efficiency for Fluid core resources.

    Key capabilities include:

    • Dataset Management: Create, view, edit, and delete Datasets.
    • Data Loading Tasks: Visual management of Dataload tasks with support for multiple loading strategies.
    • Runtime Management: Monitor the status and events of runtimes like Alluxio, JuiceFS, and GooseFS.
    • Multi-cluster and Namespace Support: Ability to switch between different clusters and namespaces for multi-tenant environments.

  7. Overview of Fluid for Kubernetes Data Access

    master

    Fluid provides a simple and efficient cloud-native data access solution for Kubernetes users. It functions by using Runtime Plugins to extend compatibility with various distributed caching engines.

    Fluid acts as an abstraction layer that allows users to interact with underlying storage systems through a caching layer, effectively bridging the data access path.

    Key features include:

    • Integrated Caching Engines: Support for open-source engines like Alluxio and JuiceFS, as well as cloud-specific engines like Alibaba Cloud EMR's JindoFS.
    • ThinRuntime: A specialized runtime developed by the Fluid community to allow users to quickly integrate other general-purpose storage systems without needing to write custom CSI plugins or complex Runtime Controllers.

  9. Review Fluid's security compliance and best practices

    master

    Fluid adheres to several industry standards and best practices:

    • OpenSSF Best Practices: Fluid has a passing page in CII/OpenSSF Best Practices and is working towards a Silver badge.
    • CI/CD Security: The development pipeline includes Trivy scans for image vulnerabilities, unit/e2e tests, and automatic code coverage via codecov.io. Container images and release artifacts are immutable or signed at the time of build.

  10. Key features of Fluid

    master

    Fluid provides several capabilities for managing data in Kubernetes:

    • Unified DataSet Abstraction: Consolidates multiple storage sources and provides observability (volume, cache size, hit rate) to help users decide when to scale cache resources.
    • Extensible Runtime Plugins: Supports various storage backends through specialized CacheRuntime or ThinRuntime implementations.
    • Automated Data Operations: Enables data prefetch, migration, and backup via Custom Resource Definitions (CRDs). Operations can be triggered via one-time, scheduled, or event-driven modes.
    • Data Elasticity and Scheduling: Combines distributed caching with autoscaling and data-affinity scheduling to improve performance.
    • Platform Agnostic: Runs on native, edge, and Serverless Kubernetes, as well as multi-cluster environments. It adapts to environment constraints (like the lack of CSI support in Serverless K8s) by choosing between CSI Plugins and sidecar modes.

  11. What is ThinRuntime and how does it work?

    master

    The ThinRuntime CRD allows users to integrate any custom storage system into Fluid by describing it via a ThinRuntimeProfile. This is useful for storage systems that are not natively integrated with Fluid.

    ThinRuntime supports two usage modes:

    1. Normal Mode: By specifying the profileName field, you can mount external storage systems.
    2. Reference Dataset Mode: By omitting the profileName field, you can reference other existing Datasets.

    To use ThinRuntime with a Fuse-based storage system, you must provide a container image that contains both a Fuse client program (e.g., goofys, s3fs) and a runtime parameter parsing script. This script reads configuration from /etc/fluid/config/config.json (or /etc/fluid/config.json in older versions) to extract mounting parameters like URLs, bucket names, and credentials, then executes the Fuse client to perform the mount.

  12. What is Fluid and how does it work?

    master

    Fluid is an application-oriented data abstraction layer for Kubernetes designed to solve data access latency and bandwidth overhead in cloud-native environments (like AI and Big Data workloads).

    Unlike traditional PVC-based storage which focuses on volume management, Fluid focuses on the process of using data. It introduces the DataSet as a first-class Kubernetes citizen, allowing users to perform CRUD operations, manage permissions, and accelerate access to datasets.

    Fluid works by converting distributed caching systems (such as Alluxio or JuiceFS) into observable, self-managing, and elastically scalable caching services. It also enables data-affinity scheduling, allowing Kubernetes to schedule applications closer to where the cached data resides.