DOCSEARCHbyAlgolia
Submit Docs

Fluid

website·Indexed 30 days ago

https://fluid-cloudnative.github.io/docs

An open source Kubernetes-native Distributed Dataset Orchestrator and Accelerator. Fluid provides a unified dataset abstraction for heterogeneous data sources, enabling data acceleration and multidimensional management in Kubernetes and multi-cluster environments. It supports cache runtime auto-scaling, data preloading, and integration with storage interfaces like POSIX and HDFS, as well as environments like Knative.

Tokens
111.6K
Snippets
466
Records
718
Agent score
67%

What's inside Fluid

  2. Overview of JindoRuntime for Fluid

    JindoRuntime is a distributed cache runtime implementation for Fluid based on the JindoFS engine. It is designed to optimize big data storage and is fully compatible with the Hadoop file system interface. It provides acceleration for accessing and caching remote files from storage products including OSS, HDFS, and AWS S3. Key features include:

    • POSIX file system interface via FUSE, allowing deep learning tools like PyTorch to read remote data as if it were on a local disk.
    • Independent caching and prefetching of metadata and data to reduce OSS metadata operation overhead.
    • Support for on-cloud data security, including password-free access and checksum verification.
    • Integration with Fluid's data perception and scheduling to place tasks on nodes where data is already cached.

  3. Overview of Fluid CLI

    Fluid CLI (fluid) is a standalone Go binary designed to help operators and developers troubleshoot Fluid workloads on Kubernetes. It provides a focused experience for inspecting Datasets and collecting diagnostic support bundles (YAML, events, logs, storage objects) without requiring modifications to Fluid controllers. It uses standard Kubernetes client configuration (kubeconfig, --context, -n).

  4. Overview of Fluid Pod Scheduling Optimization

    Fluid provides scheduling plugins that use a webhook mechanism to automatically inject affinity-related information into Pods. This optimizes cluster efficiency by aligning Pod placement with dataset layouts. Supported functions include:

    1. Data affinity scheduling compatible with the native K8s scheduler, Volcano, and Yunikorn.
    2. Prioritizing scheduling of Pods to nodes with data caching capabilities.
    3. Forcing Pods to nodes with data caching via specific Pod Labels.
    4. Avoiding nodes with cache for Pods that do not utilize datasets.

  5. Overview of JindoRuntime for Fluid

    JindoRuntime is a distributed cache runtime implementation for Fluid based on the JindoFS engine. It optimizes big data storage by providing a POSIX file system interface (via FUSE), allowing deep learning tools like PyTorch to access remote files on OSS, HDFS, and AWS S3 as if they were on a local disk. Key features include distributed caching of metadata and data, independent metadata prefetching, and support for on-cloud security features like checksum verification.

  6. Overview of Fluid CLI

    Fluid CLI (fluid) is a standalone Go binary used to inspect and diagnose Fluid workloads on Kubernetes. It allows operators to view Dataset status, related Kubernetes resources, and collect support bundles for troubleshooting without needing to manually aggregate logs and YAMLs via kubectl. It uses standard kubeconfig and does not require modifications to Fluid controllers.

  7. Overview of Fluid Cache Runtime Auto Scaling

    Fluid provides cache elasticity scaling capabilities via a custom HPA mechanism. This allows the cache space to expand automatically when the existing cache data reaches a specific percentage of the total capacity (e.g., triggering a scale-out when cache usage exceeds 80%). This on-demand scaling is transparent to the user and improves data usage efficiency by avoiding manual resource estimation.

  8. Overview of Fluid Distributed Dataset Orchestrator

    Fluid is an open-source, Kubernetes-native Distributed Dataset Orchestrator and Accelerator designed for data-intensive applications (e.g., AI and Big Data). It abstracts the process of using data on Kubernetes by introducing 'Datasets' as first-class citizens, allowing for CRUD operations, permission control, and access acceleration. Fluid converts distributed caching systems (such as Alluxio, JuiceFS, Vineyard, and CubeFS) into observable, self-managing, elastic, and self-healing caching services. It enables data-affinity scheduling, allowing Kubernetes to place applications on nodes where the required datasets are already cached.

  9. Overview of Fluid for Cloud-Native Data Orchestration

    Fluid is a Kubernetes-based data orchestration system that abstracts the process of computing the data used by tasks. Unlike traditional storage-oriented PVCs, Fluid introduces the concept of an 'elastic Dataset'. This allows developers to define data features (e.g., small files, read-only, or read-write) and manage the data lifecycle and elasticity based on application demand. It focuses on the orchestration of data and the applications that use it, providing a scheduling system centered on the Dataset.

  10. Overview of Fluid and JindoRuntime components

    v0.9

    Fluid and JindoRuntime work together to provide distributed data orchestration and acceleration on Kubernetes:

    • Fluid: An extensible distributed data orchestration and acceleration system. It manages how data is orchestrated and how applications that depend on that data are scheduled to minimize access latency.
    • JindoRuntime: A distributed cache runtime implementation for Fluid based on the JindoFS engine. It enables access and cache acceleration for remote storage products including OSS, HDFS, and AWS S3. It is compatible with native Kubernetes environments and supports Hadoop file system interfaces.

  11. Overview of JindoRuntime for Fluid

    v0.9
    JindoRuntime is a distributed cache runtime implementation for Fluid based on the JindoFS engine. It is designed to optimize big data storage by providing a POSIX file system interface via FUSE, allowing deep learning tools like PyTorch to read remote files (from OSS, HDFS, and AWS S3) as if they were on a local disk. Key features include distributed caching of metadata and data, independent metadata prefetching, and compatibility with native Kubernetes environments.

  12. Overview of Fluid for data-intensive Kubernetes workloads

    v0.9
    Fluid is a data orchestration layer for Kubernetes that abstracts the process of computing data used by tasks. Unlike traditional storage-oriented PVCs, Fluid introduces the concept of an 'elastic Dataset'. This allows developers to define data features (e.g., small files, read-only, or read-write) and manage the data lifecycle and orchestration based on application demand rather than just storage volume.