DOCSEARCHbyAlgolia
Submit Docs

Generative Recommenders

repository·main·Indexed 24 days ago

https://github.com/meta-recsys/generative-recommenders

Code for implementing and scaling trillion-parameter sequential transducers for recommendation systems. It features efficient algorithms like HSTU and M-FALCON, and includes a reference implementation for DLRMv3 inference, synthetic dataset generation, and performance benchmarking using LoadGen.

Tokens
8.6K
Snippets
26
Records
46
Agent score
83%

What's inside generative-recommenders

  1. Overview of LoadGen Over the Network (LON) architecture

    main

    The LoadGen Over the Network (LON) demo implements a distributed testing environment with two primary roles:

    • SUT (System Under Test): Implemented in sut_over_network_demo.py. It runs a Flask server that provides dummy interfaces for preprocessing, postprocessing, and model calling. In multi-node setups, each node runs an instance of the SUT.
    • LoadGen Node: Implemented in py_demo_server_lon.py. It runs the LoadGen, QSL, and QDL instances and acts as a Flask client to communicate with the SUT nodes.

    The test operates in MLPerf Server mode.

  2. Implement a Query Dispatch Library (QDL) for network testing

    main

    When running LoadGen over a network, you implement a Query Dispatch Library (QDL). The QDL acts as a proxy/load-balancer between LoadGen and the System Under Test (SUT).

    Implementation Details:

    • The QDL must implement the mlperf::QueryDispatchLibrary class, which inherits from mlperf::SystemUnderTest.
    • Query Issuance: The QDL receives queries via void IssueQuery(const std::vector<QuerySample>& samples) and dispatches them over the physical network.
    • Response Handling: The QDL receives responses from the SUT and notifies LoadGen using void QuerySamplesComplete(QuerySampleResponse* responses, size_t response_count).
    • Required Methods: The QDL must also implement const std::string& Name() (to identify it as a network benchmark) and void FlushQueries().

    Data Handling:

    • QSL may compress data to reduce network time; however, decompression must happen within the timed processing of the SUT.
    • Performing timed preprocessing or inference within the QDL is strictly disallowed.
    // QDL receives queries via:
void IssueQuery(const std::vector<QuerySample>& samples)

// QDL notifies LoadGen of completions via:
struct QuerySampleResponse {
  ResponseId id;
  uintptr_t data;
  size_t size;
};
void QuerySamplesComplete(QuerySampleResponse* responses, 
                          size_t response_count);

// Additional required methods:
const std::string& Name();
void FlushQueries();

  3. How LoadGen integrates into an inference system

    main

    LoadGen is a reusable module designed to measure the performance of inference systems by generating traffic. It is not model or dataset aware, meaning it relies on the user to provide the model logic and data.

    Integration Flow:

    1. Benchmark: Knows the model, dataset, and preprocessing logic.
    2. Dataset Handover: The Benchmark provides dataset sample IDs to LoadGen.
    3. Query Generation: LoadGen generates queries using those sample IDs.
    4. Request Creation: The Benchmark receives these queries and creates requests to the backend.
    5. Result Processing: Results are post-processed and forwarded back to LoadGen.
    6. Logging: LoadGen outputs logs for performance analysis.

  4. Install Generative Recommenders

    main

    You can install the project dependencies using requirements.txt. This setup has been tested on Ubuntu 22.04, CUDA 12.4, and Python 3.10.

    Alternatively, you can perform a manual installation by first installing PyTorch following official instructions, and then installing the following packages:

    • gin-config
    • pandas
    • fbgemm_gpu
    • torchrec
    • tensorboard

  5. Install LoadGen as a C++ Library

    main

    To build LoadGen as a static C++ library instead of a Python module, use CMake:

    1. Clone the repository:
      git clone https://github.com/mlcommons/inference.git mlperf_inference
cd mlperf_inference
    2. Build using CMake:
      mkdir loadgen/build/ && cd loadgen/build/
cmake .. && cmake --build .
    3. The resulting library libmlperf_loadgen.a will be located in the build directory. Copy it to your project directory as needed:
      cp libmlperf_loadgen.a ..
    mkdir loadgen/build/ && cd loadgen/build/
cmake .. && cmake --build .
cp libmlperf_loadgen.a ..

  6. Run DLRM-v3 Training and Inference

    main

    DLRM-v3 is a model built using HSTU. You can run benchmarks for both training and inference.

    Training: Run generative_recommenders/dlrm_v3/train/train_ranker.py with WORLD_SIZE and LOCAL_WORLD_SIZE environment variables set to the number of GPUs.

    Inference: Inference requires the mlperf_inference repository. You must clone it, install the loadgen component with specific CFLAGS, and then run generative_recommenders/dlrm_v3/inference/main.py.

    # Run training with 4 GPUs
LOCAL_WORLD_SIZE=4 WORLD_SIZE=4 python3 generative_recommenders/dlrm_v3/train/train_ranker.py --dataset debug --mode train

# Run inference with 4 GPUs
git clone --recurse-submodules https://github.com/mlcommons/inference.git mlperf_inference
cd mlperf_inference/loadgen
CFLAGS="-std=c++14 -O3" python -m pip install .
LOCAL_WORLD_SIZE=4 WORLD_SIZE=4 python3 generative_recommenders/dlrm_v3/inference/main.py --dataset debug

  7. Generate or download DLRMv3 datasets

    main

    DLRMv3 uses a synthetic dataset designed for large-scale sequential recommendation (large item sets and long sequence lengths).

    Generate synthetic dataset

    Run the streaming synthetic data script to generate a 2TB dataset containing 5 million users, a billion items, and 100 timestamps.

    Download sampled dataset for inference

    If you prefer to use the sampled dataset used in the inference benchmark, use the mlc-r2-downloader.sh script to download the metadata URI.

    # Generate synthetic dataset
cd generative_recommenders/dlrm_v3/
python streaming_synthetic_data.py

# Download sampled dataset for inference benchmark
bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) https://inference.mlcommons-storage.org/metadata/dlrm-v3-dataset.uri

  11. Install and build LoadGen for the LON demo

    main

    To run the LoadGen Over the Network (LON) demo, you must install the required Python dependencies, clone the MLPerf inference repository with submodules, and build the LoadGen wheel.

    Note: The build requires CFLAGS="-std=c++14 -O3" to be passed to setup.py.

    # Install python packages
pip install absl-py numpy wheel flask requests

# Clone the repository
git clone --recurse-submodules https://github.com/mlcommons/inference.git mlperf_inference

# Build and install LoadGen
cd mlperf_inference/loadgen
CFLAGS="-std=c++14 -O3" python setup.py bdist_wheel
cd ..; pip install --force-reinstall loadgen/dist/`ls -r loadgen/dist/ | head -n1` ; cd -

  12. Locate and view MLPerf trace logs

    main

    LoadGen generates an mlperf_log_trace.json file which uses the Trace Event Format to record a timeline of all involved threads.

    To view these logs:

    1. Open Chrome and type chrome://tracing in the address bar.
    2. Drag and drop the mlperf_log_trace.json file into the window.
    3. Alternatively, you can drag a zipped version of the trace file directly into the window.

    When filing bug reports, it is recommended to include zipped traces and other logs.