DOCSEARCHbyAlgolia
Submit Docs

Arctic Inference

repository·main·Indexed 19 days ago

https://github.com/snowflakedb/arcticinference

An open-source vLLM plugin providing high-throughput and low-latency inference for LLMs and Embeddings. It utilizes Snowflake's proprietary optimizations, including Shift Parallelism, SwiftKV, and Forest Cascade Attention (FCA) to reduce redundant KV-cache reads. The library includes a Replica Manager for horizontal scaling, load balancing, and health monitoring via a unified gRPC endpoint.

Tokens
20.5K
Snippets
68
Records
92
Agent score
66%

What's inside arctic_inference

  1. What is Dynasor

    main
    Dynasor is a tool designed to accelerate LLM reasoning models without requiring training or fine-tuning. It achieves this through dynamic prompt execution, early stopping when the model has sufficient information, and adaptive compute based on model certainty. It acts as a general proxy server that is compatible with the standard OpenAI API and supports both batch and streaming responses.

  2. What is Arctic Ulysses and how does it work?

    main

    Arctic Ulysses is a form of sequence parallelism (SP) implemented as an extension to vLLM. It is designed to balance the low latency of tensor parallelism (TP) with the high throughput of data parallelism (DP) in transformer architectures.

    By allowing users to select specific degrees of both TP and SP, Arctic Ulysses enables optimized inference for different workloads:

    • Low Latency: Achieved by tuning TP and SP for faster time-to-first-token (TTFT).
    • High Throughput: Achieved by tuning TP and SP to maximize requests per second.

    Ulysses does not modify the underlying model architecture; it only modifies the parallelism strategy, meaning it maintains the same model quality as standard vLLM implementations. While it is compatible with any model supported by vLLM, it has been thoroughly validated with Llama and Qwen models.

  3. What is Shift Parallelism

    main

    Shift Parallelism is a dynamic inference parallelism strategy that automatically adapts between Tensor Parallelism (TP) and Arctic Sequence Parallelism (SP) in real time based on current traffic (batch size).

    Its goal is to optimize for different metrics depending on the workload:

    • Small Batches: Uses TP to minimize output token latency.
    • Large Batches: Uses SP to maximize throughput and minimize time-to-first-token.

    This transition is seamless because of KV cache invariance: the cache layout remains consistent between TP and SP as long as the product of the parallelism degrees equals the total number of GPUs (TP x SP = P).

  4. What is Arctic Ulysses and when to use it

    main

    Arctic Ulysses is a sequence parallelism technique designed to optimize LLM inference for long-context inputs. Unlike traditional Tensor Parallelism (TP) which partitions model computation, Arctic Ulysses partitions the input sequence itself.

    Key Benefits:

    • Reduces Time-to-First-Token (TTFT) latency.
    • Enhances throughput efficiency.
    • Minimizes communication overhead using all-to-all communication for attention computation.

    Ideal Use Cases:

    • Retrieval-Augmented Generation (RAG)
    • Summarization
    • Code generation

  5. What is Arctic Speculator

    main

    Arctic Speculator is a speculative decoding strategy designed to accelerate LLM inference. It combines three core components:

    1. Suffix Decoding: The underlying decoding algorithm.
    2. ArcticTraining: Provides easy-to-use recipes for training speculator models.
    3. ArcticInference: Provides an optimized speculative inference pipeline.

    When deployed on top of vLLM V1, Arctic Speculator can achieve up to 4x faster end-to-end task completion for LLM agents and up to 2.8x faster decoding for open-ended interactive workloads compared to standard vLLM without speculation.

  6. What is Suffix Decoding?

    main

    Suffix Decoding is a model-free speculative decoding method designed to accelerate inference for repetitive and predictable workloads, such as agentic tasks, self-refinement loops, and multi-agent pipelines.

    Unlike model-based speculation, it uses highly efficient suffix trees to predict draft tokens using CPU resources, avoiding GPU overhead. It maintains two types of trees:

    • Global Suffix Tree: Stores historical outputs from completed requests to capture cross-request patterns.
    • Per-Request Suffix Tree: Dynamically built from the current request's prompt and generated tokens.

    The method matches the most recent token sequence (pattern) against these trees and proposes speculative tokens based on frequency. The number of speculated tokens scales dynamically with the length of the matched pattern.

  8. What is SwiftKV and how does it work?

    main

    SwiftKV is an inference optimization technique designed to reduce compute overhead during the prefill phase of Large Language Model (LLM) inference, especially for long input prompts.

    It utilizes a method called SingleInputKV, which enables later transformer layers to reuse key-value (KV) pairs computed by earlier layers. This eliminates redundant computation, improving throughput and reducing latency without requiring modifications to the model weights. In benchmarks (e.g., Llama 3.1 70B), SwiftKV has been shown to reduce prefill computation by up to 50%.

  9. How Arctic Speculator architectures work

    main

    Arctic Speculator offers two optimized, lightweight architectures for generating speculative token candidates:

    1. MLP-based Speculator: A feed-forward neural network that uses final hidden states and recent token IDs from the main LLM to propose multiple candidate tokens simultaneously. It balances simplicity, low latency, and high acceptance rates.
    2. LSTM-based Speculator: An evolution of the MLP structure that incorporates LSTM gating mechanisms (forget, input, output, and cell gates) to capture sequential token dependencies more effectively, providing higher predictive accuracy with minimal extra cost.

    Pre-trained models for these architectures are available in the Snowflake speculators collection on Hugging Face.

  10. How the Replica Manager works

    main

    The Replica Manager (replica_manager.py) provides horizontal scaling for vLLM inference. Instead of managing individual model instances, you interact with the manager, which handles:

    • Multiple Model Replicas: Launching and managing several vLLM instances.
    • Load Balancing: Distributing incoming requests across replicas using round_robin, random, or least_loaded strategies.
    • Health Monitoring: Checking replica readiness.
    • Automatic Recovery: Retrying requests if a replica becomes unavailable.
    • Unified API: Providing a single gRPC endpoint that abstracts the underlying distribution logic.

  11. How the Arctic Inference Server architecture works

    main

    The Arctic Inference Server is a multi-model inference server built on Ray and vLLM. It is designed to serve multiple models simultaneously using automatic GPU sharing and dynamic rebalancing.

    Core Components:

    • Driver: The main server class. It tracks ReplicaPool instances, manages GPU allocation with even sharing, and routes requests based on model_id.
    • ReplicaPool: Manages a group of workers for a specific model. It handles scaling, health monitoring, weight synchronization, and request submission via a Scheduler.
    • Scheduler: Routes incoming requests to the least-loaded worker, utilizing concurrency adjustments based on worker utilization.
    • Worker: A lightweight wrapper around a vLLM AsyncLLM engine.
    • Pipeline: A high-level abstraction for dataset processing that supports both single-model and multi-model workflows.
    Client ──> FastAPI ──> Driver ──> ReplicaPool (model-a) ──> Scheduler ──> Worker 0
                         │                                            └──> Worker 1
                         └──────> ReplicaPool (model-b) ──> Scheduler ──> Worker 0

  12. How Forest Cascade Attention (FCA) works

    main

    Forest Cascade Attention (FCA) is an optimization for LLM inference that reduces redundant KV-cache reads when multiple concurrent requests share long prompt prefixes (such as system prompts or few-shot examples).

    Instead of performing standard attention, FCA splits each attention call into three stages:

    1. Prefix FA: A single grouped call over shared prefix blocks (causal=False), treating all requests in a group as one sequence.
    2. Suffix FA: A per-request causal call over the remaining unique KV blocks.
    3. Merge: The two partial outputs are combined using log-sum-exp weighted merging via merge_attn_states.

    To achieve this, the engine sorts requests by KV block tables, groups them by Longest Common Prefix (LCP), builds metadata for virtual sequences, and executes the split attention calls.

    1. **Prefix FA** — one grouped call over the shared prefix blocks (`causal=False`)
2. **Suffix FA** — a per-request causal call over the remaining (unique) KV blocks
3. **Merge** — the two partial outputs are combined via log-sum-exp weighted merging (`merge_attn_states`)