DOCSEARCHbyAlgolia
Submit Docs

HierarchicalKV

repository·master·Indexed 42 days ago

https://github.com/nvidia-merlin/hierarchicalkv

A high-performance key-value storage library for large-scale Recommender Systems (RecSys) and DLRMs. It optimizes memory by managing feature embeddings across GPU HBM and host memory using specialized CUDA kernels and eviction strategies such as LRU, LFU, and Epoch-based methods.

Tokens
3.2K
Snippets
7
Records
16
Agent score
51%

What's inside HierarchicalKV

  1. What is HierarchicalKV?

    master
    HierarchicalKV is a part of NVIDIA Merlin that provides hierarchical key-value storage designed for Recommender Systems (RecSys). It enables storing feature embeddings across both GPU High-Bandwidth Memory (HBM) and host memory simultaneously. This allows training large-scale models (DLRMs) that exceed the HBM capacity of a single GPU while bypassing CPUs to improve communication performance and reduce workload.

  3. Core concepts of HierarchicalKV

    master

    HierarchicalKV operates based on several key architectural principles:

    • Hybrid Storage: Stores keys in HBM and manages values across HBM and host memory.
    • Separation of Concerns: Keys and values are stored separately.
    • Local Ordering: Buckets are locally ordered.
    • Eviction Management: Uses built-in or customizable eviction strategies implemented via CUDA kernels to manage HBM consumption.
    • High Utilization: Designed to operate at a high working-status load factor close to 1.0.

  4. Configure EvictStrategy for key importance

    master

    Eviction occurs only when a bucket is full. The score (type uint64_t) defines the importance of a key: larger scores mean higher importance and a lower likelihood of eviction.

    Available strategies:

    • __Lru__: Uses device clock in nanoseconds.
    • __Lfu__: Uses frequency increments provided by the caller via the scores parameter in insert-like APIs.
    • __EpochLru__: Uses a combination of a caller-provided global_epoch (high 32 bits) and a device clock-based value (low 32 bits, granularity ~1ms).
    • __EpochLfu__: Uses a caller-provided global_epoch (high 32 bits) and a frequency (low 32 bits) that caps at 0xffffffff.
    • __Customized__: The caller provides the full score via the scores parameter in insert-like APIs.

    Note: insert-like APIs include insert_or_assign, insert_and_evict, find_or_insert, accum_or_assign, and find_or_insert. The caller is responsible for maintaining and providing the global_epoch for epoch-based strategies.

  5. Understand HierarchicalKV API Naming Conventions

    master

    When reviewing performance benchmarks or API documentation, note the following naming conventions for key operations:

    • find* and find_or_insert*: These represent APIs that directly return the addresses of values, rather than copying them. This is a critical distinction for high-performance usage.
    • insert_or_assign: Inserts a new key-value pair or updates the value if the key already exists.
    • insert_and_evict: Inserts a value and triggers the eviction strategy (e.g., LRU) if capacity is reached.
    • export_batch / export_batch_if: Batch operations for exporting data.
    • contains: Checks for the existence of a key.

  6. Add links to methods or headings with underscores

    master

    When creating links to a method or a heading that contains underscores, you must repeat the underscores in the link target, even though they are converted to hyphens in the final HTML output.

    Examples:

    • Linking to a heading with spaces and underscores: ../somefile.md#2heading-with-spaces-and_underscore_separated_words-too
    • Linking to a method with underscores: ./otherfile.md#save_params_to_files-method

  7. Build HierarchicalKV with Bazel

    master

    To build using Bazel, it is recommended to use a Docker container to ensure environment consistency.

    Important Notes

    • Do NOT use the --recursive flag when cloning the repository for Bazel builds.
    • If using customized Docker images, modify the environment variables in the .bazelrc file beforehand.
    • The nvcr.io/nvidia/tensorflow images are highly recommended.

    Build Steps

    1. Pull and run the recommended TensorFlow Docker image:
      docker pull nvcr.io/nvidia/tensorflow:22.09-tf2-py3
docker run --gpus all -it --rm nvcr.io/nvidia/tensorflow:22.09-tf2-py3
    2. Inside the container, clone the repository and run the provided build script:
      git clone https://github.com/NVIDIA-Merlin/HierarchicalKV.git
cd HierarchicalKV && bash bazel_build.sh

    Running Benchmarks

    • Benchmark: ./benchmark_util
    docker pull nvcr.io/nvidia/tensorflow:22.09-tf2-py3
docker run --gpus all -it --rm nvcr.io/nvidia/tensorflow:22.09-tf2-py3

# Inside container
git clone https://github.com/NVIDIA-Merlin/HierarchicalKV.git
cd HierarchicalKV && bash bazel_build.sh

  8. Build the HierarchicalKV documentation

    master

    To build the documentation locally, you must install doxygen and the Python dependencies specified in docs/requirements-doc.txt. The build process uses Doxygen, Breathe, and Exhale to generate API documentation from the C++ source code via Sphinx.

    1. Install system dependencies:
      sudo apt-get install doxygen
    2. Install Python requirements:
      pip install -r docs/requirements-doc.txt
    3. Run the build command from the root of the repository:
      make -C docs clean html

    The output will be located at docs/build/html/index.html.

    make -C docs clean html

  9. Build HierarchicalKV with CMake

    master

    HierarchicalKV is a header-only library. The build process primarily creates binaries for benchmarks and unit testing.

    Prerequisites

    • CUDA version >= 11.2
    • NVIDIA GPU with compute capability 8.0, 8.6, 8.7, or 9.0
    • GCC supporting C++17 or later
    • Bazel version >= 3.7.2 (if using Bazel)

    Build Steps

    1. Clone the repository recursively:
      git clone --recursive https://github.com/NVIDIA-Merlin/HierarchicalKV.git
    2. Create a build directory and run CMake. Note that you must specify the compute capability using the -Dsm flag (e.g., -Dsm=80 for Ampere architecture).

    Running Benchmarks and Tests

    After building, you can run the following binaries:

    • Benchmark: ./merlin_hashtable_benchmark
    • Unit Test: ./merlin_hashtable_test
    git clone --recursive https://github.com/NVIDIA-Merlin/HierarchicalKV.git
cd HierarchicalKV && mkdir -p build && cd build
cmake -DCMAKE_BUILD_TYPE=Release -Dsm=80 .. && make -j

  10. Link to files in the GitHub repository

    master

    If you need to link to a file that is not part of the published documentation (such as STYLE_GUIDE.md), provide the full URL to the file on GitHub so the reader knows they are leaving the documentation site.

    Example:

    ## Coding Style
Refer to the [Style Guide](http://github.com/NVIDIA-Merlin/HierarchicalKV/STYLE_GUIDE.md) in the GitHub repository for more details.

  11. Preview built documentation locally

    master

    After building the documentation, you can preview the generated HTML files by running a local HTTP server. Run the following command from the root of the repository:

    python -m http.server 8000 --directory docs/build/html

    Then, open your web browser and navigate to http://localhost:8000.

  12. Link between documentation files

    master

    When linking between files within the documentation (e.g., from README.md to another section), use relative paths. Sphinx will automatically convert the .md suffix to .html during rendering.

    Example:

    To build HierarchicalKV from scratch, refer to
[How to Build](./README.md#how-to-build) in the `README` file.