DOCSEARCHbyAlgolia
Submit Docs

Azure MCP

repository·main·Indexed 23 days ago

https://github.com/azure/azure-mcp

An archived project providing Model Context Protocol (MCP) integration and tooling for Azure in Visual Studio Code via the vscode-azure-mcp-server package (v0.1.0-alpha.1). It includes documentation for running the Azure MCP server in STDIO and SSE modes, authoring Azure SDK MCP servers in C#, and managing test recording assets using tools like test-proxy and the generate-assets-json.ps1 script.

Tokens
96.4K
Snippets
163
Records
544
Agent score
76%

What's inside azure-mcp

  1. Analyze Tool Selection Results for Azure MCP

    main

    The ToolDescriptionEvaluator generates results for testing how well an LLM can select the correct Azure MCP tool based on a natural language prompt. Each test case evaluates a specific Prompt against an Expected Tool and provides a ranked list of tools with their corresponding semantic similarity Score.

    Key metrics in the results include:

    • Rank: The position of the tool in the selection list.
    • Score: The confidence/similarity score assigned to the tool.
    • Tool: The name of the azmcp_* tool identified.
    • Status: Indicates if the top-ranked tool matches the EXPECTED tool (✅) or not (❌).

    This data is used to validate the accuracy of tool descriptions and the effectiveness of the tool selection mechanism within the Azure MCP ecosystem.

  2. Migrate to the new Azure MCP repository

    main

    This repository (azure/azure/azure-mcp) is archived as of August 25, 2025. For all current development, code, and issues related to Azure MCP, use the new repository at https://github.com/microsoft/mcp.

    Specifically, for information regarding the Azure MCP Server, refer to the documentation located at: https://github.com/microsoft/mcp/blob/main/servers/Azure.Mcp.Server/README.md.

  3. Understand analysis metrics and performance ratings

    main

    The tool evaluates tool-to-prompt matching using the following metrics:

    • Confidence Scores: Cosine similarity scores ranging from 0.0 to 1.0.
    • Success Rate: The percentage of prompts where the expected tool achieved the highest rank.
    • Top-N Accuracy: Frequency of the expected tool appearing in the top 3, 5, or 10 results.

    Performance Ratings

    RatingCriteria
    🟢 Excellent>90%
    🟡 Good75-90%
    🟠 Fair50-75%
    🔴 Poor<50%

  4. Use pre- and post-scripts to manage test resource parameters

    main

    If resource creation requires external artifacts (like certificates) or specific setup steps, you can use test-resources-pre.ps1 or test-resources-post.ps1 located in the same directory as your test-resources.json file.

    You can pass data from a pre-script to the main resource template by adding entries to the $templateFileParameters hash table. These entries will map to parameters of the same name in your JSON/Bicep template.

    Example Workflow:

    1. In test-resources-pre.ps1, generate a certificate and add it to $templateFileParameters.
    2. In test-resources.json, define a parameter with the matching name to receive the value.
    # Snippet from test-resources-pre.ps1
$templateFileParameters['ConfidentialLedgerPrincipalPEM'] = Format-X509Certificate2 -Certificate $cert

# Corresponding snippet from test-resources.json
"parameters": {
  "ConfidentialLedgerPrincipalPEM": {
    "type": "string"
  }
}

  5. Understand the Command Hierarchy and Architecture

    main

    Commands in Azure MCP follow a strict inheritance hierarchy to ensure consistent behavior across the CLI. All commands must implement the IBaseCommand interface and follow this pattern:

    Hierarchy Pattern:

    1. IBaseCommand (Root interface)
    2. BaseCommand (Implements core capabilities)
    3. GlobalCommand<TOptions>
    4. SubscriptionCommand<TOptions> (Handles subscription parameters)
    5. Service-specific base commands (e.g., BaseSqlCommand for adding service-wide options)
    6. Resource-specific commands (e.g., SqlIndexRecommendCommand)

    Core Interface (IBaseCommand) Members:

    • Name: Command name for CLI display.
    • Description: Detailed command description.
    • Title: Human-readable command title.
    • GetCommand(): Returns the System.CommandLine command definition.
    • ExecuteAsync(): Executes the command logic.
    • Validate(): Validates command inputs.
    • ToolMetadata: Property defining behavioral characteristics.

  6. Use the ObjectVerb naming convention for files and classes

    main

    To ensure discoverability and consistency, all command-related files and classes must follow the ObjectVerb naming pattern: {Resource}{SubResource}{Operation}.

    Examples:

    • ServerListCommand (Resource: Server, Operation: List)
    • ServerConfigGetCommand (Resource: Server, SubResource: Config, Operation: Get)
    • TableSchemaGetCommand (Resource: Table, SubResource: Schema, Operation: Get)

    Apply this pattern to:

    • Command classes: ServerConfigGetCommand
    • Options classes: ServerConfigGetOptions
    • Test classes: ServerConfigGetCommandTests
    • File names: ServerConfigGetCommand.cs, ServerConfigGetOptions.cs

    Anti-patterns to avoid:

    • ❌ GetConfigCommand (Missing resource prefix)

  7. Analyze Tool Selection Analysis Results

    main

    The ToolDescriptionEvaluator produces results for various test cases to evaluate how accurately a prompt maps to an expected Azure MCP tool. Each test result includes:

    • Expected Tool: The specific tool name that should be selected for the given prompt.
    • Prompt: The natural language input used for the test.
    • Results Table: A ranked list of tools identified by the evaluator, containing:
      • Rank: The order of selection.
      • Score: A numerical value representing the confidence/relevance of the tool to the prompt.
      • Tool: The name of the tool selected.
      • Status: Indicates if the top-ranked tool matches the Expected Tool (✅ EXPECTED) or not (❌).

    These results are used to benchmark the effectiveness of tool descriptions in guiding model selection.

  8. Understand Tool Description confidence scores

    main

    The Tool Description Evaluator uses Azure OpenAI embeddings to compare prompts against tool descriptions. It produces a confidence score for each tool-prompt combination:

    • Score Range: 0.00 to 1.00.
    • High Score: Indicates a high probability that the tool will be selected by a user given that specific prompt.
    • Low Score (< 0.40): Indicates a low chance of tool selection. If scores are below this threshold, you should improve the tool's description to better align with user intent.

  9. Organize new Azure MCP areas using the Area Pattern

    main

    To maintain consistency, all new Azure services and their commands must follow the Area pattern. This pattern groups all code, options, models, and tests for a specific service (an "Area") into a single directory structure.

    Directory Structure:

    • Area code: areas/{area-name}/src/AzureMcp.{AreaName} (e.g., areas/storage/src/AzureMcp.Storage)
    • Unit Tests: areas/{area-name}/tests/AzureMcp.{AreaName}.UnitTests
    • Live Tests: areas/{area-name}/tests/AzureMcp.{AreaName}.LiveTests
    areas/storage/src/AzureMcp.Storage

  10. Follow the MCP Command Naming Pattern

    main

    Commands follow the Model-Context-Protocol (MCP) pattern for the CLI interface. The structure is:

    azmcp <azure service> <resource> <operation>

    Example: azmcp storage container list

    Rules for Command Groups:

    • No Underscores: Command group names cannot contain underscores. Use camelCase, concatenated names, or dash separators.
      • ✅ new CommandGroup("entra-admin", ...)
      • ❌ new CommandGroup("entra_admin", ...)
    • Avoid Mixing Resource and Operation: Do not combine the resource name and operation into a single command. Use a hierarchy instead.
      • ✅ azmcp postgres server param set (Groups: server $\rightarrow$ param, Operation: set)
      • ❌ azmcp postgres server setparam

  11. Understand Tool Selection Analysis Results

    main

    The ToolDescriptionEvaluator generates analysis results to measure how accurately a prompt maps to the expected Azure MCP tool. Each test result includes:

    • Expected Tool: The correct tool name for the given prompt.
    • Prompt: The natural language input used for the test.
    • Rank: The position of the tool in the selection list.
    • Score: A numerical value representing the confidence/similarity score.
    • Tool: The name of the tool identified by the evaluator.
    • Status: Indicates if the top-ranked tool matches the Expected Tool (✅ EXPECTED) or not (❌).

  12. Interpret job suffixes (bX and ibY)

    main

    When viewing pull request jobs, you may see suffixes like b1 or ib1. These are added automatically during matrix creation when packages are batched to manage scale.

    • bX (e.g., b1): Indicates a batch of direct packages. Packages are grouped by their matrix configuration (from their ci.yml) and then split into batches based on a configurable batchSize (defaulting to 10).
    • ibX (e.g., ib1): Indicates a batch of indirect packages. These are batched similarly but, by default, use a single deterministically selected item from the resolved test matrix rather than the full matrix.