DOCSEARCHbyAlgolia
Submit Docs

Foundry Local Documentation

repository·main·Indexed 25 days ago

https://github.com/microsoft/foundry-local

An end-to-end local AI solution for running models on-device with complete data privacy. It provides SDKs for C#, JavaScript, Python, and Rust, featuring automatic hardware acceleration (NPU, GPU, or CPU) and an OpenAI-compatible API. Supported capabilities include text generation (e.g., phi-3.5-mini, qwen2.5-0.5b) and speech-to-text transcription (e.g., whisper-tiny).

Tokens
123.3K
Snippets
247
Records
592
Agent score
81%

What's inside Foundry Local

  1. Overview of Foundry Local Python Samples

    main

    The following Python samples demonstrate various capabilities of the Foundry Local SDK:

    • native-chat-completions: Initialize the SDK, start the local service, and run streaming chat completions.
    • embeddings: Generate single and batch text embeddings.
    • audio-transcription: Transcribe audio files using the Whisper model.
    • web-server: Start a local OpenAI-compatible web server and call it with the OpenAI Python SDK.
    • web-server-responses: Call a running local OpenAI-compatible web server with the Responses API, including streaming and tool calling.
    • tool-calling: Implement tool calling with custom function definitions (e.g., get_weather, calculate).
    • langchain-integration: Use LangChain for building translation and text generation chains.
    • tutorial-chat-assistant: Build an interactive multi-turn chat assistant.
    • tutorial-document-summarizer: Summarize documents with AI.
    • tutorial-tool-calling: Create a tool-calling assistant.
    • tutorial-voice-to-text: Transcribe and summarize audio.

  2. Overview of Foundry Local JavaScript samples

    main

    The samples/js directory contains various demonstrations of the foundry-local-sdk using Node.js. Available samples include:

    SampleDescription
    native-chat-completionsInitialize the SDK, download a model, and run non-streaming and streaming chat completions.
    embeddingsGenerate single and batch text embeddings using the Foundry Local SDK.
    audio-transcription-exampleTranscribe audio files using the Whisper model with streaming output.
    chat-and-audio-foundry-localUnified sample demonstrating both chat and audio transcription in one application.
    electron-chat-applicationFull-featured Electron desktop chat app with voice transcription and model management.
    copilot-sdk-foundry-localGitHub Copilot SDK integration with Foundry Local for agentic AI workflows.
    langchain-integration-exampleLangChain.js integration for building text generation chains.
    tool-calling-foundry-localTool calling with custom function definitions and streaming responses.
    web-server-exampleStart a local OpenAI-compatible web server and call it with the OpenAI SDK.
    web-server-responses-vision-exampleStream a vision (image understanding) response from the local web server using the Responses API.
    tutorial-chat-assistantBuild an interactive multi-turn chat assistant (tutorial).
    tutorial-document-summarizerSummarize documents with AI (tutorial).
    tutorial-tool-callingCreate a tool-calling assistant (tutorial).
    tutorial-voice-to-textTranscribe and summarize audio (tutorial).

  3. Overview of Foundry Local Rust samples

    main

    The following Rust samples demonstrate various capabilities of the Foundry Local Rust bindings:

    SampleDescription
    native-chat-completionsNon-streaming and streaming chat completions using the native chat client.
    embeddingsGenerate single and batch text embeddings using the native embedding client.
    audio-transcription-exampleAudio transcription (non-streaming and streaming) using the Whisper model.
    foundry-local-webserverStart a local OpenAI-compatible web server and call it with a standard HTTP client.
    foundry-local-webserver-responses-visionStream a vision (image understanding) response from the local web server using the Responses API.
    tool-calling-foundry-localTool calling with streaming responses, multi-turn conversation, and local tool execution.
    tutorial-chat-assistantBuild an interactive multi-turn chat assistant (tutorial).
    tutorial-document-summarizerSummarize documents with AI (tutorial).
    tutorial-tool-callingCreate a tool-calling assistant (tutorial).
    tutorial-voice-to-textTranscribe and summarize audio (tutorial).
    live-audio-transcription-exampleReal-time microphone transcription using the nemotron model. (Requires SDK live-transcription API — not yet available.)

  4. Verify WinML 2.0 Execution Providers

    main

    This sample is used to verify that WinML 2.0 execution providers (EPs) are correctly discovered, downloaded, and registered on a Windows machine. It validates the process by running inference on a model variant backed by a registered WinML EP and performing a native streaming chat check.

    Prerequisites

    • Windows with a compatible GPU
    • Python 3.11+

  5. Explore Foundry Local samples by language

    main

    Foundry Local provides complete working examples for various AI tasks including chat completions, embeddings, audio transcription, tool calling, and LangChain integration. You can find specialized samples categorized by programming language:

    • C#: 14 samples using the .NET SDK. Includes native chat, embeddings, audio transcription, tool calling, model management, web server, vision via Responses API, tutorials, and WinML EP verification (using WinML on Windows for hardware acceleration).
    • JavaScript: 16 samples using the Node.js SDK. Includes native chat, embeddings, audio transcription, Electron desktop app, Copilot SDK integration, LangChain, tool calling, web server, vision via Responses API, tutorials, and WinML EP verification.
    • Python: 14 samples using the OpenAI-compatible API. Includes chat, embeddings, audio transcription, LangChain integration, tool calling, web server, Responses API, tutorials, and WinML EP verification.
    • Rust: 12 samples using the Rust SDK. Includes native chat, embeddings, audio transcription, tool calling, web server, vision via Responses API, tutorials, and WinML EP verification.
    • C++: 1 sample for live audio transcription.

  8. Overview of Foundry Local capabilities

    main

    Foundry Local is a unified local AI runtime that provides a single SDK for multiple AI capabilities. It primarily supports text generation and speech-to-text through specific model aliases and API methods.

    Supported Capabilities

    CapabilityModel AliasesSDK API
    Chat Completions (Text Generation)phi-3.5-mini, qwen2.5-0.5b, etc.model.createChatClient()
    Audio Transcription (Speech-to-Text)whisper-tinymodel.createAudioClient()

  9. How to use std::span and std::string_view in the C++ SDK

    main

    To ensure container independence and efficient string handling, follow these patterns:

    Using std::span

    Instead of passing a const reference to a specific container like std::vector, pass a std::span by value. This allows the function to accept std::vector, std::array, or raw memory spans.

    // Instead of
void foo(const std::vector<int64_t>&);

// Use — accepts std::vector, std::array, raw spans, etc.
void foo(std::span<const int64_t>);

    When returning data from a class, return a std::span<const T> by value rather than a const reference to a member container.

    // Instead of
const std::vector<int64_t>& foo();

// Return a span by value
std::span<const int64_t> foo();

    Using std::string_view

    Prefer passing std::string_view by value instead of const std::string&. Ensure the underlying std::string instance outlives the std::string_view.

    // Instead of
void foo(const std::vector<int64_t>&);

// Use — accepts std::vector, std::array, raw spans, etc.
void foo(std::span<const int64_t>);

// Instead of
const std::vector<int64_t>& foo();

// Return a span by value
std::span<const int64_t> foo();

  10. Understand the SessionManager architecture and lifecycle

    main

    The SessionManager provides unified session tracking and caching to ensure shutdown safety and enable efficient multi-turn conversations via the Responses API.

    Core Concepts

    • Shutdown Safety: The manager tracks all active sessions. This prevents use-after-free errors by ensuring that ModelLoadManager does not destroy a GenAIModelInstance while a session is still performing inference.
    • Session Caching: For the Responses API, sessions can be cached using a key (e.g., a response ID). This allows the generator to retain its KV cache state across multiple HTTP requests, significantly reducing latency for multi-turn conversations.
    • Tracking vs. Caching: All sessions are tracked via the ISessionManager interface (Register/Deregister). Only specific sessions are additionally cached in a keyed map within the concrete SessionManager.

    Ownership and Lifetime

    Sessions are automatically deregistered when they are destroyed, regardless of how they are owned:

    • C API: Owned by the caller (delete or Session_Release).
    • Web Handlers: Owned by the handler scope (stack) or thread scope.
    • Cached Sessions: Owned by the SessionManager cache map.
    • Checked-out Sessions: Owned by the handler via std::unique_ptr during a request.

  11. Understand the Foundry Local JavaScript SDK (v2) Architecture

    main

    The v2 JavaScript/TypeScript SDK (foundry-local-sdk@2.x) is a native Node.js binding for the Foundry Local C++ SDK. It is designed to be a lightweight, high-performance replacement for the legacy v1 SDK.

    Key Architectural Layers

    1. v2 Public Surface: The primary TypeScript API containing Manager, Catalog, Model, Session (and specialized sessions like ChatSession), Request, Response, and Item.
    2. TypeScript Detail Layer: Handles native pointer ownership, AsyncIterable stream adapters, AbortSignal plumbing, and error mapping.
    3. node-addon-api C++ Addon: A C++20 layer using node-addon-api to bridge TypeScript to the C++ wrapper via Napi::AsyncWorker and ThreadSafeFunction.
    4. C++ Wrapper (foundry_local_cpp.h): Provides RAII handles, exception-based error handling, and typed accessors.
    5. C ABI (foundry_local_c.h): The foundational versioned vtable and opaque handles used by all SDKs (C#, Python, etc.).

    Important Constraints

    • Node.js Only: This SDK loads a native binary and does not support browser environments.
    • Async-First: All entry points wrapping C ABI calls are asynchronous and return Promises. Only read-only accessors that perform simple memory copies (e.g., Model.getInfo) are synchronous.
    • ESM Only: The package is ESM-only; it does not support CommonJS.