DOCSEARCHbyAlgolia
Submit Docs

Microsoft Foundry Toolkit

repository·main·Indexed 24 days ago

https://github.com/microsoft/foundry-toolkit

A Visual Studio Code extension for accelerating the development of AI agents. It provides tools for discovering and fine-tuning models via a Model Catalog and Playground, designing prompt-based or code-based agents in the Agent Builder, and managing the end-to-end lifecycle including local debugging with Agent Inspector and deployment to the Microsoft Foundry Agent Service.

Tokens
24.9K
Snippets
25
Records
121
Agent score
84%

What's inside Microsoft Foundry Toolkit

  1. Overview of Microsoft Foundry Toolkit features

    main

    Microsoft Foundry Toolkit (formerly AI Toolkit) is a Visual Studio Code extension for building AI agents. Features are organized into three main workflows:

    Models

    Tools for discovering, testing, and optimizing AI models:

    • Model Catalog: Access models from Microsoft Foundry, Foundry Local, GitHub, ONNX, Ollama, OpenAI, Anthropic, Google, and more.
    • Playground: An interactive chat environment for real-time testing with prompts, parameters, and multi-modal inputs (images/attachments).
    • Fine-tuning: Customize models locally with GPU support or in the cloud using Azure Container Apps.
    • Model Conversion: Convert and quantize models (e.g., from Hugging Face) for efficient local execution on Windows (CPU, GPU, or NPU).
    • Profiling (Windows ML): Diagnose hardware usage (CPU, GPU, NPU) for ONNX models and Windows Machine Learning events.

    Agents

    Tools for the end-to-end agent lifecycle:

    • Agent Builder: A no-code environment to design prompt agents, integrate MCP tools, and define structured outputs.
    • Tool Catalog: Connect Microsoft Foundry tools and local MCP servers to your agents.
    • Agent Inspector: Debug and visualize local and hosted agents.
    • Deployment: Push hosted agents to the Microsoft Foundry Agent Service by building container images for Azure Container Registry or uploading ZIP packages. Supports Python, C#, or YAML projects (including Microsoft Agent Framework and LangGraph).
    • Hosted Agent Playground: Test deployed agents with features like thread management, live container logs, OpenTelemetry traces, and tool-call inspection.
    • Evaluation & Tracing: Assess performance using datasets and metrics (F1 score, relevance, etc.) and monitor application behavior via trace data.

    Microsoft Foundry Resources

    Direct integration to browse Foundry projects, deploy models from providers (Microsoft, OpenAI, Meta, DeepSeek), and manage the Microsoft Foundry Agent Service.

  2. Key features of AI Toolkit

    main

    AI Toolkit for Visual Studio Code is designed for developers and AI engineers to build AI applications by developing and testing generative AI models locally or in the cloud.

    Core capabilities include:

    • Model Catalog: Access various generative AI model sources including GitHub, ONNX, OpenAI, Anthropic, and Google.
    • Model Flexibility: Bring your own models from remote hosts or use Ollama models running locally.
    • Playground: Test models via inference or chat interfaces.
    • Multi-modal Support: Attachment support for Multi-modal Language Models.
    • Roadmap: Upcoming features include evaluation, dataset management, batch runs, fine-tuning, and prompt engineering tools.

  3. Automate agents with Foundry Routines

    main

    Foundry Routines (preview) allow you to run agents automatically based on specific triggers. A routine pairs a trigger (a one-time timer or a recurring schedule) with an action that invokes a prompt or a hosted agent. This keeps triggers, permissions, connections, and run history unified within your Foundry project.

    Key capabilities in VS Code:

    • Create routines: Directly from the Agents view, hosted-agent detail, or the routine list.
    • Routine list: A dedicated view to browse all project routines.
    • Routine detail view: Inspect full run history, including inputs, outputs, status, and links to agent responses and trace details.
    • Management: Edit, rename, duplicate, resume, pause, and delete routines.

  4. Integrate and build MCP tools for agents

    main

    You can extend agent capabilities by integrating Model Context Protocol (MCP) tools. Foundry Toolkit allows you to:

    • Connect to existing servers: Connect to an MCP server via stdio or sse.
    • Build custom servers: Create your own MCP server using the Weather MCP Server scaffold in either TypeScript or Python.
    • Test and Debug: Use the AI Toolkit Agent Builder or the MCP Inspector to validate your server.
    • Discover Tools: Access featured MCP servers provided by Microsoft, community members, or MCP Reference implementations.

  5. Use the My Resources tree view

    main

    The AI Toolkit and Foundry extension sidebar panels are unified into a single My Resources view. This view organizes your assets as follows:

    • Local Resources: A node containing your local models, agents, and tools.
    • Foundry Resources: Remote Foundry resources appear alongside local ones.
    • Visual Cues: Foundry agents and Foundry models use dedicated icons to distinguish them from local resources.

    If Foundry is installed but not yet configured, an inline setup prompt will appear in the panel.

  6. Trace AI application execution and observability

    main
    AI Toolkit provides local observability for AI applications and agents. You can trace the execution of your applications, including interactions with generative AI models, and use the AI Toolkit tracing visualization tool to analyze behavior and performance.

  7. How tracing works in Microsoft Foundry Toolkit

    main

    AI Toolkit hosts a local HTTP and gRPC server that acts as an OTLP (OpenTelemetry Protocol) collector. This server collects trace data from your AI applications. To visualize the collected data, the toolkit provides a built-in UI within Visual Studio Code.

    Any framework or SDK that supports OTLP and follows the semantic conventions for generative AI systems is compatible. The toolkit supports various AI SDKs across Python and TS/JS, often utilizing third-party instrumentation libraries (like traceloop, LangSmith, or opentelemetry-python-contrib) when the official SDKs do not natively support OTLP.

  8. Use AIAgentExpert for agent development

    main

    The AIAgentExpert is a specialized custom agent introduced in version 0.26.0. It provides expert-level assistance specifically for the lifecycle of AI agents, including:

    • Creating: Scaffolding new agents.
    • Iterating: Refining existing agent logic.
    • Tracing: Debugging agent execution flows.
    • Evaluating: Testing agent performance against metrics.

  9. Build collaborative agents with Connected Agents

    main
    In the Agent Builder (v0.26.0), you can use Connected Agents to build collaborative systems. This allows multiple agents to connect and interact with one another to accomplish complex, multi-step tasks that a single agent might struggle to complete.

  11. Configure VNet-restricted projects

    main
    If your Foundry project has public network access disabled, it requires an approved Azure Virtual Network (VNet). In the resources tree, these projects will display a VNet required label when you are not connected to an approved VNet. This label provides a link to the private network configuration guide and reminds you to refresh the project once you have connected to the correct VNet.

  12. How Agent Inspector works internally

    main

    The Agent Inspector operates by wrapping your agent in a local server environment that facilitates communication between the VS Code UI and your Python code.

    Execution Flow

    1. Agent Server: When you press F5, the agentdev CLI starts an HTTP server on port 8087 and attaches debugpy on port 5679.
    2. Discovery: The Inspector UI fetches available agents and workflows via the /agentdev/entities endpoint.
    3. Streaming: Chat inputs are sent to /v1/responses, which uses Server-Sent Events (SSE) to stream execution events back to the UI for real-time visualization.
    4. Code Navigation: The UI uses the /agentdev/.../location endpoint to map workflow nodes to source files.

    Local Server Endpoints

    The TestToolServer provides the following internal endpoints:

    • /agentdev/entities: Lists available agents.
    • /agentdev/entities/{id}/info: Retrieves details for a specific agent.
    • /agentdev/.../location: Facilitates code navigation.
    • /agentdev/ws/health: WebSocket health check.
    • /v1/responses: SSE stream for execution events.

    An EventMapper component converts Agent Framework events into the OpenAI SSE format used by the Inspector UI.