Azurite Documentation

repository·main·Indexed 24 days ago

https://github.com/azure/azurite

An open source, Node.js-based Azure Storage API compatible emulator for local testing and development. Azurite provides a cross-platform server that simulates Azure Blob, Queue, and Table storage services. It supports various authentication methods including SharedKey, SAS, and OAuth, and can be deployed via NPM, Docker, or as a Visual Studio Code extension.

Tokens
16K
Snippets
26
Records
82
Agent score
76%

What's inside Azurite

  1. What is Azurite?

    main

    Azurite is an open source, Node.js-based emulator that provides an API-compatible server for Azure Storage. It allows developers to simulate Azure Storage services in a local, cross-platform environment with minimal dependencies.

    Azurite V3 is built on a new architecture using TypeScript and a server code generator that leverages the same swagger definitions used by Azure Storage SDKs, ensuring better alignment with official Azure Storage APIs.

  2. Understand Azurite Workspace Structure and Data Cleanup

    main

    When Azurite is initialized in a workspace, it creates several metadata files and folders to persist data.

    Metadata and Data Files:

    • azurite_db_blob.json: Metadata for the blob service.
    • azurite_db_blob_extent.json: Extent metadata for the blob service.
    • blobstorage: Binary data for the blob service.
    • azurite_db_queue.json: Metadata for the queue service.
    • azurite_db_queue_extent.json: Extent metadata for the queue service.
    • queuestorage: Binary data for the queue service.
    • azurite_db_table.json: Metadata for the table service.

    Note: To perform a complete cleanup of all stored data, delete these files and folders and then restart Azurite.

  3. Queue Service: VisibilityTimeout and AccessPolicy behavior

    main

    When interacting with the Azurite Queue service, note the following deviations from standard Swagger definitions to align with actual server behavior:

    • VisibilityTimeout: There are no minimum or maximum limitations enforced for VisibilityTimeout or VisibilityTimeoutRequired.
    • AccessPolicy: The required section has been removed from the AccessPolicy definition to match how the server actually processes these policies.
  4. Understand Azurite's API Version Compatibility Strategy

    main

    Azurite V3 follows a "Try best to serve" strategy regarding Azure Storage API versions:

    • Baseline Version: Azurite has a baseline API version. It uses a Swagger definition (OpenAPI) of this version to generate its protocol layer.
    • Matching Version: If a request uses the same API version as Azurite's baseline, Azurite provides parity with Azure Storage.
    • Higher Version: If a request uses an API version higher than Azurite's baseline, Azurite returns an InvalidHeaderValue error for the x-ms-version header (HTTP 400 - Bad Request).
    • Lower Version: If a request uses an API version lower than Azurite's baseline, Azurite attempts to handle the request using its baseline version behavior.
    • Response Headers: Azurite returns its baseline API version in the response headers.
    • SAS: Shared Access Signature (SAS) patterns are accepted from API version 2015-04-05.
  5. The AST Node structure and evaluation

    main

    The Abstract Syntax Tree (AST) is composed of node classes found in ./QueryNodes/*.ts. Every node must implement the evaluate(context: IQueryContext) method.

    Nodes resolve values by recursively evaluating their children:

    • ConstantNode: Returns a literal value (e.g., a string or number).
    • Complex Nodes (e.g., EqualsNode): Evaluates child nodes (typically left and right) and performs the operation on the returned values. For example, an EqualsNode returns the result of left.evaluate(context) === right.evaluate(context).
  6. How the Table Query Interpreter works

    main

    The Azurite Table query interpreter implements the OData query API used for $filter-ing query results from a Table. It processes query strings through a multi-layered pipeline to resolve them into runtime values without dynamic code emission.

    The pipeline consists of four main stages:

    1. Lexer: Converts the raw query string into a flat sequence of tokens (e.g., identifiers, operators, strings).
    2. Parser: Uses a Recursive Descent strategy to transform the flat token stream into an Abstract Syntax Tree (AST) based on language rules (EBNF).
    3. Validator: Performs semantic validation on the AST (e.g., ensuring the query references at least one identifier like PartitionKey, RowKey, or TableName).
    4. Evaluation: The AST is traversed to resolve the query. Each node in the tree implements an evaluate(context: IQueryContext) method, which is called using a depth-first strategy to produce a final result.
  7. Understand the limitations of current Type-Aware Queries

    main

    Currently, the Azurite Table query interpreter evaluates queries without strict regard for data types. This can lead to discrepancies between Azurite and actual Azure Table Storage behavior.

    Example of current behavior: 123.0L eq '123.0' evaluates to true in Azurite, even though the data types (Int64 vs String) are different. This mismatch applies to datetime, binary, X, and guid types.

    Planned Improvement: A future breaking change aims to implement a TypedValue interface to ensure type-wise equality during comparison:

    export interface TypedValue {
      value: any;
      type: "Undefined" | "Null" | "Edm.Guid" | "Edm.DateTime" | "Edm.Int64" | "Edm.Binary" | "Edm.Boolean" | "Edm.String" | "Edm.Int32" | "Edm.Double";
    }
  8. Handle binary data changes in Version 3.36.0

    main
    In Version 3.36.0, Azurite standardized binary data handling using Uint8Array instead of Buffer. This affects areas such as MD5 hashes, Content-MD5 headers, and internal buffer conversions. If your integration relies on Buffer-specific behavior, you may need to adjust your code to handle Uint8Array.
  9. Use in-memory storage for Azurite

    main

    You can run Azurite entirely in-memory to avoid persisting data to disk. If the process terminates, all data is lost.

    Constraints:

    • This setting is rejected if AZURITE_DB (SQL-based metadata) is enabled or if the --location option is specified.
    • The in-memory extent store (for blob and queue content) is limited to 50% of the host's total memory by default.

    Memory Management:

    • Override the memory limit using --extentMemoryLimit <megabytes>.
    • Content is not freed immediately upon deletion. The blob extent GC runs every 10 minutes and the queue extent GC runs every 1 minute. Memory is released only after both the extent GC and the Node.js runtime GC have run.
    • If the limit is reached, write operations will fail with HTTP 409 and error code MemoryExtentStoreAtSizeLimit.
  10. Key features of Azurite V3

    main

    Azurite V3 supports the following services and authentication methods:

    Supported Services

    • Blob Storage: Supports Block Blobs and Page Blobs, including container management and service property operations.
    • Queue Storage: Supports queue management and message operations (Put, Get, Peek, Update, Delete, Clear).
    • Table Storage (Preview): Supports table and entity management (Create, List, Delete, Insert, Update, Query).

    Authentication Support

    • SharedKey
    • Account SAS
    • Service SAS
    • OAuth
    • Public Access

    Architectural Improvements

    • Built with TypeScript and native ECMA async/promise features.
    • New architecture based on a TypeScript server generator for protocol layers, models, and handlers.
    • Extensible structure allowing for custom handler implementations, persistency layers, and HTTP pipeline middleware injection.
    • Detailed debugging log support.
  11. Correct DNS name formatting for multi-block URLs

    main

    Starting from Version 3.14.2, Azurite no longer supports DNS names with multiple blocks that omit the account name in the first block (e.g., http://foo.bar.com:10000/devstoreaccount1/container).

    When using a DNS name with multiple blocks, the storage account name must be included in the first block.

    Correct format: http://devstoreaccount1.blob.localhost:10000/container

  12. Regenerate Protocol Layer from Swagger using Autorest

    main

    To regenerate the protocol layer from Swagger using Autorest, follow these steps:

    1. Install Autorest globally: npm install -g autorest.
    2. Clone the Autorest TypeScript server generator: git clone --recursive https://github.com/xiaoningliu/autorest.typescript.server.
    3. Build the generator:
      • Navigate to the cloned folder.
      • Run npm install.
      • Run npm install -g gulp.
      • Run npm run build.
    4. Update the package.json in the Azurite repository: modify build:autorest:blob and build:autorest:queue to point to your local path of the cloned generator.
    5. Run the generation commands from the Azurite root folder:
      • npm run build:autorest:queue
      • npm run build:autorest:blob
    npm install -g autorest
    git clone --recursive https://github.com/xiaoningliu/autorest.typescript.server
    # Inside autorest.typescript.server folder:
    npm install
    npm install -g gulp
    npm run build
    # Inside Azurite root folder:
    npm run build:autorest:queue
    npm run build:autorest:blob