DOCSEARCHbyAlgolia
Submit Docs

FriendliAI Documentation

website·Indexed 35 days ago

https://friendli.ai/docs

Documentation for FriendliAI, featuring Friendli Container for deploying generative AI models on private infrastructure, the Friendli Python SDK, and guides for serving MoE, Multi-LoRA, and quantized models. Includes integration guides for Amazon EKS, SageMaker, and various coding agents, as well as support for models like GLM-5.2 and GLM-5.3-Flash.

Tokens
64.9K
Snippets
296
Records
447
Agent score
73%

What's inside FriendliAI

  1. Overview of Friendli Container for self-hosted AI deployment

    Friendli Container allows developers to deploy generative AI models on their own infrastructure by packaging the Friendli Engine as a Docker container. This is designed for users who require full control over GPU resources, networking, and scaling, or those with strict data privacy and compliance requirements that necessitate keeping data within their own environment.

  2. Overview of Friendli Dedicated Endpoints

    Friendli Dedicated Endpoints allow users to deploy custom or open-source generative AI models on GPU instances dedicated entirely to a single user. This ensures consistent performance and eliminates the need for infrastructure management or shared resources. Key capabilities include:

    • Custom Model Deployment: Support for bringing your own models or deploying models directly from Hugging Face.
    • Dedicated Hardware: Ability to select specific GPU types for workloads, ensuring no resource sharing.
    • Billing Model: A pay-per-second billing structure based on the time the model is running.

  3. Overview of Reasoning Models on FriendliAI

    Reasoning (or 'thinking') allows models to generate a chain of thought before responding, which improves quality for complex tasks but increases token usage and response time. FriendliAI supports two types of reasoning models:

    • Controllable reasoning models: Reasoning can be toggled on or off using the enable_thinking parameter.
    • Always-on reasoning models: Reasoning is always generated and cannot be disabled.

    Non-reasoning models cannot generate chains of thought.

  4. Overview of Friendli Model APIs

    Friendli Model APIs provide instant, serverless access to a curated set of frontier open-source AI models. The service is powered by Friendli Engine for high-performance inference and eliminates the need for users to handle infrastructure setup, GPU management, downloading, or hosting models. Key features include:

    • OpenAI Compatibility: Integration is possible using any OpenAI-compatible client.
    • Usage-Based Pricing: Users pay per token rather than paying for idle GPU time.
    • Zero Deployment: Models are available via ready-to-use API endpoints.

  5. Overview of Friendli Model APIs for Inference

    Friendli provides a suite of Model APIs for various inference tasks. Available capabilities include:

    • Chat Completions: Generate text through interactive conversations.
    • Responses (Beta): Generate model responses from text or image inputs, supporting tool calls, structured outputs, and reasoning.
    • Messages (Beta): Generate responses using Anthropic Messages-style payloads.
    • Completions: Standard text generation.
    • Tokenization & Detokenization: Break down text into tokens for processing and reconstruct tokens back into human-readable text.
    • Audio Transcriptions: Convert audio input into text.
    • Chat Render: Tools for rendering chat interfaces.

  6. Overview of Friendli Dedicated Endpoints API

    The Friendli Dedicated Endpoints API provides a comprehensive set of tools for managing dedicated model deployments and performing inference. The API is divided into two primary functional areas: Inference (for generating content, embeddings, and processing media) and Endpoint Management (for controlling the lifecycle and configuration of dedicated deployments).

  7. Overview of Friendli Container Inference Capabilities

    Friendli Container provides a self-hosted API for various machine learning inference tasks. Supported capabilities include:

    • Text Generation: Chat completions, standard completions, and Anthropic Messages-style payloads.
    • Advanced Responses: Tool calls, structured outputs, and reasoning (Beta).
    • Text Processing: Text classification, tokenization, and detokenization.
    • Multimodal: Image generation, image editing, and audio transcription.

  8. Understand Friendli Model API pricing structures

    Friendli Model APIs use different billing metrics based on the model type:

    • Text models are charged based on the number of processed tokens.
    • Audio models are charged based on the duration of the processed audio (calculated per second and aggregated into a per-minute rate).

  9. Understand the Friendli Container Completions Chunk Object schema

    When streaming the completions API in Friendli Container, the server returns a series of chunk objects. Each chunk represents a segment of the generated text or metadata about the request.

    Key fields include:

    • id: Unique identifier for the completion.
    • object: Always set to text_completion.
    • model: The model used to generate the completion.
    • created: Unix timestamp (seconds) of when the token was sampled.
    • choices: An array of choice objects containing the generated text, the token ID, and the finish_reason (stop for successful completion or length if it hit a token limit).
    • usage: Present in the final chunks to provide prompt_tokens, completion_tokens, and total_tokens.
    • logprobs: Optional detailed probability data including text_offset, token_logprobs, tokens, and top_logprobs.
    // Example of a text chunk
{
  "id": "cmpl-26a1e10db8544bc3adb488d2d205288b",
  "object": "text_completion",
  "choices": [
    {
      "index": 0,
      "text": " such",
      "token": 1778,
      "finish_reason": null,
      "logprobs": null
    }
  ],
  "created": 1733382157
}

// Example of a final usage chunk
{
  "id": "cmpl-26a1e10db8544bc3adb488d2d205288b",
  "object": "text_completion",
  "choices": [],
  "usage": {
    "prompt_tokens": 5,
    "completion_tokens": 10,
    "total_tokens": 15
  },
  "created": 1733382157
}

  10. Understand the Friendli Dedicated Messages streaming event sequence

    When streaming the messages API via Friendli Dedicated Endpoints, the response is delivered as a series of event chunks. A typical stream follows this sequence:

    1. message_start: Initializes the message envelope and usage tracking.
    2. content_block_start: Signals the beginning of a specific content piece (text, thinking, or tool use).
    3. content_block_delta: Sends incremental fragments of the content block.
    4. content_block_stop: Signals the end of that specific content block.
    5. message_delta: Provides final message-level metadata, such as the stop_reason and final token usage.
    6. message_stop: Signals the complete end of the stream.
    // Example of a standard text stream sequence
event: message_start
data: { "type": "message_start", "message": { ... } }

event: content_block_start
data: { "type": "content_block_start", "index": 0, "content_block": { "type": "text", "text": "" } }

event: content_block_delta
data: { "type": "content_block_delta", "index": 0, "delta": { "type": "text_delta", "text": "Hello" } }

event: content_block_stop
data: { "type": "content_block_stop", "index": 0 }

event: message_delta
data: { "type": "message_delta", "delta": { "stop_reason": "end_turn" }, "usage": { "output_tokens": 11 } }

event: message_stop
data: { "type": "message_stop" }

  11. Understand the Friendli Model APIs streaming message sequence

    When streaming the messages API, Friendli returns a series of events in a specific sequence to build the final response. The typical flow is: message_start $\rightarrow$ content_block_start $\rightarrow$ content_block_delta (repeated) $\rightarrow$ content_block_stop $\rightarrow$ message_delta $\rightarrow$ message_stop.
    // Example sequence for a text response
event: message_start
  data: { "type": "message_start", "message": { ... } }

event: content_block_start
  data: { "type": "content_block_start", "index": 0, "content_block": { "type": "text", ... } }

event: content_block_delta
  data: { "type": "content_block_delta", "index": 0, "delta": { "type": "text_delta", "text": "Hello" } }

event: content_block_stop
  data: { "type": "content_block_stop", "index": 0 }

event: message_delta
  data: { "type": "message_delta", "delta": { "stop_reason": "end_turn" }, "usage": { ... } }

event: message_stop
  data: { "type": "message_stop" }

  12. Optimize MoE and quantized models with Policy Search

    Policy search optimizes the execution policy of the Friendli Engine to boost inference throughput and reduce latency by 1.5x to 2x. This process is specifically effective for:

    1. Mixture of Experts (MoE) models (e.g., Mixtral).
    2. Quantized models using AWQ, FP8, or INT8.

    For simple model testing, policy search can be skipped, but it is recommended for production cost and latency analysis.