LLMPerf

The load test evaluates LLM performance by spawning concurrent requests and measuring inter-token latency and generation throughput. It uses a prompt based on randomly sampled lines from Shakespeare sonnets and counts tokens using the LlamaTokenizer for consistency across APIs.

To run a load test, use the token_benchmark_ray.py script. You must specify the --llm-api and provide necessary credentials via environment variables.

Example: OpenAI Compatible API

export OPENAI_API_KEY=secret_abcdefg
export OPENAI_API_BASE="https://api.endpoints.anyscale.com/v1"

python token_benchmark_ray.py \
--model "meta-llama/Llama-2-7b-chat-hf" \
--mean-input-tokens 550 \
--stddev-input-tokens 150 \
--mean-output-tokens 150 \
--stddev-output-tokens 10 \
--max-num-completed-requests 2 \
--timeout 600 \
--num-concurrent-requests 1 \
--results-dir "result_outputs" \
--llm-api openai \
--additional-sampling-params '{}'

When using Vertex AI, the --model flag is used for logging purposes only; the actual model is selected via the VERTEXAI_ENDPOINT_ID environment variable. Because Vertex AI does not return total generated token counts, LLMPerf uses the LlamaTokenizer to count them.

Note: The GCLOUD_ACCESS_TOKEN expires frequently (approx. 15 minutes) and must be refreshed.

gcloud auth application-default login
gcloud config set project YOUR_PROJECT_ID

export GCLOUD_ACCESS_TOKEN=$(gcloud auth print-access-token)
export GCLOUD_PROJECT_ID=YOUR_PROJECT_ID
export GCLOUD_REGION=YOUR_REGION
export VERTEXAI_ENDPOINT_ID=YOUR_ENDPOINT_ID

python token_benchmark_ray.py \
--model "meta-llama/Llama-2-7b-chat-hf" \
--mean-input-tokens 550 \
--stddev-input-tokens 150 \
--mean-output-tokens 150 \
--stddev-output-tokens 10 \
--max-num-completed-requests 2 \
--timeout 600 \
--num-concurrent-requests 1 \
--results-dir "result_outputs" \
--llm-api "vertexai" \
--additional-sampling-params '{}'

When running token_benchmark_ray.py with the --results-dir flag, LLMPerf saves individual response data to JSON files. You can use pandas to load these files and analyze performance metrics such as input/output token counts, Time To First Token (TTFT), end-to-end latency, and generation throughput.

To perform an analysis:

1. Load the JSON file using pd.read_json().
2. Filter for valid requests where error_code is not empty.
3. Map the raw fields to a cleaner DataFrame for plotting and statistical analysis.

Key fields in the response JSON include:

• number_input_tokens
• number_output_tokens
• ttft_s (Time To First Token in seconds)
• end_to_end_latency_s
• request_output_throughput_token_per_s (used for generation_throughput)

import pandas as pd

# Load the individual responses JSON file
df = pd.read_json('path/to/your_individual_responses.json')

# Filter for valid responses (where error_code is not empty)
valid_df = df[(df["error_code"] != "")]

# Create a summary DataFrame for analysis
final_df = pd.DataFrame()
final_df["number_input_tokens"] = valid_df["number_input_tokens"]
final_df["number_output_tokens"] = valid_df["number_output_tokens"]
final_df["ttft_s"] = valid_df["ttft_s"]
final_df["end_to_end_latency_s"] = valid_df["end_to_end_latency_s"]
final_df["generation_throughput"] = valid_df["request_output_throughput_token_per_s"]

# Example: Calculate means and plot TTFT vs Input Tokens
mean_tokens_in = final_df["number_input_tokens"].mean()
mean_tokens_out = final_df["number_output_tokens"].mean()
print(f"Mean number of input tokens: {mean_tokens_in}. Mean number of output tokens: {mean_tokens_out}")
final_df.plot.scatter(x="number_input_tokens", y="ttft_s", title="Number of Input Tokens vs. TTFT")

For programmatic control over LLM performance testing, use the RequestsLauncher and OpenAIChatCompletionsClient (or other client implementations) via Ray.

When using Ray clients, you must pass environment variables (like OPENAI_API_KEY) into ray.init(runtime_env=...) to ensure the remote actors have access to them.

import ray
from transformers import LlamaTokenizerFast
from llmperf.ray_clients.openai_chat_completions_client import OpenAIChatCompletionsClient
from llmperf.models import RequestConfig
from llmperf.requests_launcher import RequestsLauncher

# Pass environment variables to ray.init for remote client access
ray.init(runtime_env={"env_vars": {"OPENAI_API_BASE" : "https://api.endpoints.anyscale.com/v1",
                                   "OPENAI_API_KEY" : "YOUR_API_KEY"}})

base_prompt = "hello_world"
tokenizer = LlamaTokenizerFast.from_pretrained(
    "hf-internal-testing/llama-tokenizer"
)
base_prompt_len = len(tokenizer.encode(base_prompt))
prompt = (base_prompt, base_prompt_len)

# Create a client for spawning requests
clients = [OpenAIChatCompletionsClient.remote()]

req_launcher = RequestsLauncher(clients)

req_config = RequestConfig(
    model="meta-llama/Llama-2-7b-chat-hf",
    prompt=prompt
    )

req_launcher.launch_requests(req_config)
result = req_launcher.get_next_ready(block=True)
print(result)

import ray
from transformers import LlamaTokenizerFast

from llmperf.ray_clients.openai_chat_completions_client import (
    OpenAIChatCompletionsClient,
)
from llmperf.models import RequestConfig
from llmperf.requests_launcher import RequestsLauncher


# Copying the environment variables and passing them to ray.init() is necessary
# For making any clients work.
ray.init(runtime_env={"env_vars": {"OPENAI_API_BASE" : "https://api.endpoints.anyscale.com/v1",
                                   "OPENAI_API_KEY" : "YOUR_API_KEY"}})

base_prompt = "hello_world"
tokenizer = LlamaTokenizerFast.from_pretrained(
    "hf-internal-testing/llama-tokenizer"
)
base_prompt_len = len(tokenizer.encode(base_prompt))
prompt = (base_prompt, base_prompt_len)

# Create a client for spawning requests
clients = [OpenAIChatCompletionsClient.remote()]

req_launcher = RequestsLauncher(clients)

req_config = RequestConfig(
    model="meta-llama/Llama-2-7b-chat-hf",
    prompt=prompt
    )

req_launcher.launch_requests(req_config)
result = req_launcher.get_next_ready(block=True)
print(result)

To add support for a new LLM provider, implement the LLMClient base class from llmperf.ray_llm_client and decorate the class with @ray.remote to turn it into a Ray actor.

Your implementation must include the llm_request method, which handles a single completion request.

Method Signature: llm_request(self, request_config: RequestConfig) -> Tuple[Metrics, str, RequestConfig]

Returns:

• Metrics: Performance characteristics of the request.
• str: The text generated by the LLM API.
• RequestConfig: The configuration used for the request (primarily for logging).

from llmperf.ray_llm_client import LLMClient
import ray
from typing import Tuple
# Note: You will need to import RequestConfig and Metrics from the appropriate modules

@ray.remote
class CustomLLMClient(LLMClient):

    def llm_request(self, request_config: RequestConfig) -> Tuple[Metrics, str, RequestConfig]:
        """Make a single completion request to a LLM API

        Returns: Metrics, generated text, and the request_config.
        """
        # Implementation goes here
        ...

from llmperf.ray_llm_client import LLMClient
import ray


@ray.remote
class CustomLLMClient(LLMClient):

    def llm_request(self, request_config: RequestConfig) -> Tuple[Metrics, str, RequestConfig]:
        """Make a single completion request to a LLM API

        Returns: Metrics about the performance charateristics of the request.
        The text generated by the request to the LLM API.
        The request_config used to make the request. This is mainly for logging purposes.

        """
        ...

The RequestConfig class defines the schema for an individual LLM API request. It is used to specify the model, the prompt (including its length), and any additional parameters required by the target API.

Key fields:

• model: The identifier of the model to use.
• prompt: A tuple containing the prompt string and its integer length (prompt_text, prompt_length).
• sampling_params: An optional dictionary of additional sampling parameters (e.g., temperature, top_p) to be sent with the request. Refer to the specific LLM API's documentation for supported keys.
• llm_api: An optional string specifying the name of the LLM API provider.
• metadata: An optional dictionary for attaching arbitrary metadata for logging or validation.

from llmperf.models import RequestConfig

config = RequestConfig(
    model="gpt-4",
    prompt=("What is Ray?", 12),
    sampling_params={"temperature": 0.7},
    llm_api="openai",
    metadata={"test_run": "batch_01"}
)