DOCSEARCHbyAlgolia
Submit Docs

Vineyard Documentation

repository·main·Indexed 21 days ago

https://github.com/v6d-io/v6d

Vineyard is a high-performance data object management system that provides an Apache Airflow Provider for XCom backends and a Kubernetes Operator for managing vineyardd clusters. It includes a Go SDK, a CLI tool called vineyardctl, and support for Kubernetes resources such as Backup, Recover, Operation (assembly and repartition), and Sidecar. The system features memory spilling configurations and provides benchmarks for system, mimalloc, and Vineyard-specific allocators, as well as blob operation throughput and latency testing.

Tokens
131.1K
Snippets
409
Records
555
Agent score
75%

What's inside Vineyard

  1. Overview of vineyard-graph

    main
    The vineyard-graph module defines the core graph data structures designed to be shared across different graph computing engines. It provides a standardized way to represent graph data, including vertices and edges, to ensure interoperability between various processing tools.

  2. Overview of vineyardctl CLI

    main

    vineyardctl is the command-line interface used to interact with the Vineyard Operator on Kubernetes. It allows users to manage the lifecycle of Vineyard components (like vineyardd and pluggable drivers), create/delete jobs (backup, recover, operation), and manage cluster state.

    Key capabilities include:

    • Lifecycle Management: Deploy, delete, and check the status of the Vineyard Operator and components.
    • Job Creation: Create backup, recover, and operation jobs on Kubernetes.
    • Data & Storage: Manage CSI drivers, inject sidecars, and perform data operations like put or ls.
    • Workload Orchestration: Schedule workloads or workflows to existing Vineyard clusters.

  3. What is Vineyard (v6d)?

    main

    Vineyard (v6d) is an in-memory immutable data manager designed for big data tasks like graph analytics, numerical computing, and machine learning. It provides high-level abstractions and zero-copy in-memory sharing for distributed data, which helps eliminate the overhead of serialization/deserialization and I/O when exchanging data between different computing systems (e.g., moving data between SQL and Python environments).

    Key features include:

    • Efficient Data Sharing: Uses shared memory to share immutable data across systems without extra overhead.
    • Out-of-the-box Data Abstraction: Uses a metadata-payload separated data model. It utilizes VCDL (Vineyard Component Description Language) to annotate sharable members and methods, enabling automatic boilerplate code generation.
    • Pluggable I/O Routines: Provides common manipulation routines (drivers) for immutable data, such as IO adapters and data partition strategies, which can be reused across diverse computation jobs.
    • Kubernetes Orchestration: Supports cloud-native big data processing by leveraging Kubernetes for scaling and data-aware scheduling.

  4. Key Features of Vineyard

    main

    Efficient In-Memory Immutable Data Sharing

    Eliminates serialization/deserialization and I/O costs by using shared memory for data exchange between systems.

    Out-of-the-Box High-Level Data Abstractions

    Provides hierarchical metadata to describe complex objects (e.g., tensors, dataframes, graphs) so different computation systems can exchange data without manual transformation.

    Stream Pipelining

    Uses a special kind of immutable data called a stream. A producer can write data chunks to Vineyard while a successor job reads them from shared memory, allowing computation and I/O to overlap and reducing total processing time.

    Versatile Drivers

    Provides reusable routines (drivers) for common boilerplate tasks like I/O adapters, data partition strategies, and migration jobs.

  5. Extending vineyard with custom data types

    main

    Vineyard provides efficient data structures like tensors, data frames, tables, and graphs. You can extend vineyard by registering user-defined types in the vineyard type registry. Once registered, computing engines built on top of vineyard can automatically leverage these custom data structures.

    There are two primary ways to extend vineyard:

    1. Python: Craft builders and resolvers for custom Python data types.
    2. C++: Implement and register custom data types in C++ for deep integration with the vineyard ecosystem.

  6. Understand Vineyard key concepts

    main

    Vineyard is built around several core abstractions that govern how data is represented, integrated, and accessed. To use Vineyard effectively, you should understand the following conceptual pillars:

    • Vineyard Objects: The fundamental design space of objects within the system.
    • VCDL and Integration: How Vineyard uses VCDL (Vineyard Common Data Language) to integrate with various computing systems.
    • Accessing Objects: The different methodologies available for retrieving and interacting with objects stored in Vineyard.
    • Streams: The stream abstraction used for immutable data sharing and its various use cases.
    • I/O Drivers: The built-in drivers designed to ease the integration of computing engines with existing infrastructure.

    If you are new to the project, it is recommended to start with the Getting Started guide before diving into these specific technical concepts.

  7. Use the Vineyard Go SDK

    main
    The Vineyard Go SDK provides Go language bindings for the Vineyard client and its data structure abstractions. It allows Go developers to interact with Vineyard services and utilize its core data models directly within Go applications.

  8. Deploy vineyard components on Kubernetes

    main

    The vineyardctl deploy command is the primary entrypoint for deploying various vineyard components to Kubernetes. Available subcommands include:

    • backup-job: Deploy a backup job for the vineyard cluster.
    • csidriver: Deploy the vineyard CSI driver.
    • operator: Deploy the vineyard operator.
    • recover-job: Deploy a job to recover a backup.
    • vineyard-cluster: Deploy the vineyard cluster.
    • vineyard-deployment: Deploy vineyardd without the operator.
    • vineyardd: Deploy vineyardd on Kubernetes.
    # deploy the default vineyard cluster on kubernetes
vineyardctl --kubeconfig $HOME/.kube/config deploy vineyard-cluster

# deploy the vineyard operator on kubernetes
vineyardctl -n vineyard-system --kubeconfig $HOME/.kube/config deploy operator

# deploy the vineyardd on kubernetes
vineyardctl -n vineyard-system --kubeconfig $HOME/.kube/config deploy vineyardd

  9. Orchestrate workflows with Vineyard and Airflow or Kedro

    main

    Vineyard integrates with workflow orchestration engines like Apache Airflow and Kedro to enhance performance by optimizing how large data objects are handled during pipeline execution.

    Apache Airflow Integration

    Airflow uses Vineyard as an XCom backend. This allows users to work with large Python objects containing complex data types (such as pandas.DataFrame) at minimal cost. By using Vineyard, you eliminate the need for manual and expensive pickle.dump and pickle.loads operations when passing data between tasks.

    Kedro Integration

    Kedro uses Vineyard as a DataSet implementation. This enables efficient sharing of large data objects across different nodes in a pipeline. Compared to alternatives like AWS S3 or Minio, the Vineyard integration reduces the high cost of (de)serialization and I/O. It is designed to work seamlessly when scaling pipelines to Kubernetes without requiring intrusive modifications to your existing pipeline code.

  10. What is Vineyard and how does it work?

    main

    Vineyard (v6d) is an in-memory immutable data manager designed for zero-copy data sharing in distributed big data tasks (e.g., graph analytics, numerical computing, machine learning).

    Instead of saving intermediate results to a distributed file system (like HDFS) and reloading them—which incurs high I/O and transformation costs—Vineyard allows different systems (like Mars and PyTorch) to share data via a shared memory manager. This enables distributed workflows to behave similarly to single-machine workflows where data is shared via local variables.

  11. What is Vineyard and its core capabilities

    main

    Vineyard is an in-memory immutable data manager designed for big data tasks such as graph analytics, numerical computing, and machine learning. It provides high-level abstractions and zero-copy in-memory sharing for distributed data across different systems.

    Core Features:

    • Efficient Data Management: Zero-copy sharing of distributed data across systems.
    • High-Level Abstractions: Out-of-the-box support for distributed objects like tensors, tables, and graphs.
    • Polyglot Support: Native support for C++, Python, and Java.
    • Streaming Support: Built-in capabilities for data access and system pipelining.
    • Extensible Driver Framework: A set of efficient built-in drivers to eliminate boilerplate for I/O, serialization, and checkpointing.

    Cloud-Native Alignment:

    • Kubernetes Integration: Deploys and scales on Kubernetes; objects are observable via Custom Resource Definitions (CRDs).
    • Data-Aware Scheduling: Uses a Kubernetes scheduler plugin to enable efficient zero-copy sharing for data-intensive workflows.
    • Immutable Design: Uses an immutable object design consistent with cloud-native infrastructure principles.

  12. Understand data migration in Kubernetes

    main

    In a multi-host Kubernetes cluster, data migration occurs when a worker pod needs to access a local object that resides on a different host.

    Example Scenario:

    1. Workload A produces a global object O with local parts O1 (on host h1) and O2 (on host h2).
    2. Workload B starts with a pod B1 on h1 and B2 on h3.
    3. B1 accesses O1 locally via memory mapping.
    4. B2 cannot access O2 because it is on h2.
    5. A utility program in the initContainer of B2 triggers the migration of O2 from h2 to h3.

    Optimization: The Vineyard scheduler plugin attempts to minimize migration costs by prioritizing the placement of worker pods on hosts where the required data already resides.