DOCSEARCHbyAlgolia
Submit Docs

cuDF Documentation

repository·main·Indexed 27 days ago

https://github.com/rapidsai/cudf

cuDF is a GPU-accelerated DataFrame library for tabular data processing, part of the RAPIDS suite. It provides high-performance alternatives to pandas, Polars, and Dask by leveraging CUDA. The library includes a pandas-like API, a zero-code change accelerator via cudf.pandas, and integration with Polars Lazy API through cudf-polars. Additional components include cuStreamz for accelerated Kafka data ingestion and dask-cudf for multi-GPU clusters using LocalCUDACluster.

Tokens
98.5K
Snippets
207
Records
584
Agent score
95%

What's inside cuDF

  2. Overview of NVIDIA cuDF libraries

    main

    NVIDIA cuDF is a GPU-accelerated library for tabular data processing within the RAPIDS suite. It consists of several specialized libraries depending on your workflow:

    • cudf: A Python library providing a pandas-like DataFrame API. It includes cudf.pandas, a zero-code change accelerator for existing pandas code.
    • cudf-polars: A Python library that provides a GPU engine for Polars.
    • dask-cudf: A Python library providing a GPU backend for Dask DataFrames.
    • libcudf: A CUDA C++ library featuring Apache Arrow compliant data structures and fundamental tabular data algorithms.
    • pylibcudf: A Python library providing Cython bindings for libcudf.

  3. Overview of NDS-H Benchmarks for libcudf

    main

    NDS-H is a benchmarking suite for libcudf derived from the TPC-H Benchmarks.

    Important Disclaimer: Results obtained using NDS-H are not comparable to published TPC-H Benchmark results because NDS-H does not comply with the official TPC-H Benchmark specifications.

    Current Implementation Status: As of the current version, only the following queries are implemented:

    • Q1
    • Q5
    • Q6
    • Q9
    • Q10

  4. Overview of libcudf C++ library

    main
    libcudf is a C++ GPU DataFrame library designed for high-performance tabular data processing. It provides GPU-accelerated operations for loading, joining, aggregating, filtering, and manipulating data. The library is built around a column-oriented architecture.

  5. Use the cuDF DataFrame API

    main

    The cudf.DataFrame is the primary object for GPU-accelerated tabular data processing. It provides a comprehensive API similar to pandas, allowing for high-performance operations on data stored in GPU memory. Key functional areas include:

    • Data Access & Attributes: Inspect data using .dtypes, .shape, .columns, .index, and .values.
    • Indexing & Selection: Access data via .loc, .iloc, .at, .iat, or using .query().
    • Transformations: Convert types with .astype(), reshape with .pivot() or .melt(), and handle missing data with .fillna() or .dropna().
    • Computations: Perform descriptive statistics (.mean(), .sum(), .describe()), aggregations (.agg()), and window functions (.rolling()).
    • Joins & Merges: Combine datasets using .merge(), .join(), and .concat() (via pandas-like patterns).
    • IO & Serialization: Load/save data using .to_parquet(), .to_csv(), .to_json(), and interoperate with other formats via .from_arrow(), .to_pandas(), or .to_cupy().

  6. Use cuDF Index objects

    main

    cuDF provides several specialized Index objects for managing tabular data labels. While many index-related methods are available directly on Series or DataFrame objects (and should be preferred), you can interact with Index objects directly for specific manipulations.

    Key categories of Index operations include:

    • Properties: Access metadata like dtype, shape, size, is_unique, is_monotonic_increasing, and name.
    • Modifying/Computations: Perform operations like drop_duplicates, rename, unique, nunique, factorize, and copy.
    • Missing Values: Handle nulls using isna, isnull, notna, notnull, fillna, and dropna.
    • Conversion: Convert indices to other formats such as to_numpy, to_pandas, to_cupy, to_arrow, to_dlpack, or to_pylibcudf.
    • Sorting/Selection: Use sort_values, argsort, searchsorted, get_loc, and isin.
    • Set Operations: Combine or compare indices using append, union, intersection, join, and difference.
    • String Operations: Access string-specific methods via the .str accessor.

  8. Understand pylibcudf Design Principles

    main

    pylibcudf is a lightweight Cython wrapper around libcudf designed for near-zero overhead access to libcudf in Python. It aims for near-native C++ performance while maintaining interoperability with standard Python containers, the __cuda_array_interface__, and CuPy arrays.

    Key design principles include:

    • Public API: Every public function or method is cpdefed to allow usage in both Cython and Python.
    • Typing: Variables are strongly typed (primitives or cdef classes). C++ enums are mirrored using cpdef enum to provide both C-style enums in Cython and PEP 435-style Python enums.
    • Syntax: All typing uses Cython syntax rather than PEP 484.
    • Dependencies: pylibcudf should ideally depend only on rmm and have minimal runtime dependencies.
    • Type Stubs: Manual type stubs are provided and must be updated when adding new functionality.

  9. Understand libcudf Core Terminology

    main

    libcudf is a C++ library for GPU-accelerated data-parallel algorithms on column-oriented tabular data. Key concepts include:

    • Column: An array of data of a single type. It may include a validity mask for null values and support nested types. Equivalent to a cuDF Python Series.
    • Element: An individual data item within a column (also known as a row).
    • Scalar: A type representing a single element of a data type.
    • Table: A collection of columns with the same number of elements. Equivalent to a cuDF Python DataFrame.
    • View: A non-owning object providing zero-copy access (with potential slicing or offsets) to data owned by another object (e.g., column_view, table_view).

  11. Understand the cuDF architecture for developers

    main
    cuDF is a GPU-accelerated, pandas-like DataFrame library. Its architecture is designed to map pandas APIs to the underlying CUDA-accelerated libcudf C++ library. When developing for cuDF, it is important to understand that the Python-level functionality relies on efficient and robust mapping to libcudf functions.

  12. Numba Memory Management and the Numba Runtime (NRT)

    main

    cuDF UDFs rely on the Numba Runtime (NRT) for managing the lifecycle of complex objects like strings. The NRT uses NRT_MemInfo to track object lifetimes via reference counting (NRT_incref and NRT_decref).

    Key NRT components used in string UDFs:

    • NRT_MemInfo: A structure containing the reference count (refct) and a pointer to the object's destructor.
    • NRT_Allocate: Used to allocate memory for the MemInfo and the object itself.
    • NRT_decref: Decrements the reference count and triggers destruction if the count reaches zero.
    • NRT_MemInfo_call_dtor: Invokes the registered destructor for the object.