Azure Pipelines YAML Documentation

repository·master·Indexed 23 days ago

https://github.com/microsoft/azure-pipelines-yaml

YAML templates, samples, and design documentation for Azure Pipelines. Includes guidance on architectural principles for artifacts, authentication tasks for Maven (MavenAuthenticate@0), npm (npmAuthenticate@1), NuGet/dotnet/MSBuild (NuGetAuthenticate@0), pip (PipAuthenticate@0), and twine (TwineAuthenticate@1), as well as configuration for custom checkout paths.

Tokens
27.2K
Snippets
81
Records
116
Agent score
79%

What's inside azure-pipelines-yaml

  1. Make pipeline variables read-only

    master

    To prevent steps from accidentally or maliciously overwriting important data, you can mark variables as read-only. Once a variable is marked as read-only, any attempt to change its value using the ##vso[task.setVariable] logging command will generate a warning: VariableName is read-only and can't be changed.

    The following types of variables are treated as read-only:

    • System variables: All variables prefixed with System., Agent., Build., etc.
    • Output variables: Variables set with isOutput=true.
    • Queue-time variables: Variables provided by the server at queue time.
    • Script-created variables: Variables set via logging commands with the isReadonly=true property.
    • YAML-defined variables: Variables defined in the YAML pipeline using the readonly: true property.
    variables:
    - name: first
      value: one
      readonly: true  # new syntax marking this variable readonly
    
    steps:
    - script: echo "##vso[task.setVariable variable=second;isReadonly=true]two"
      displayName: Set a readonly variable from a script/task
    - script: echo "##vso[task.setVariable variable=third;isOutput=true]three"
      displayName: Output variables are automatically readonly
    - script: echo Another readonly variable is $(Build.SourcesDirectory)
      displayName: System variables are readonly
  2. Principles for using Artifacts in Azure Pipelines YAML

    master

    When using Azure Pipelines with a YAML-based configuration, the system is designed to support package operations (via tools like nuget.exe, dotnet, npm, mvn, gradle, pip, twine, etc.) through a script-first approach.

    Key architectural principles include:

    • Tool Agnosticism: You can invoke supported tools in any way you prefer (e.g., via - script:, - powershell:, or custom scripts) and they should work regardless of whether you are running in a container job.
    • Script-First Preference: The system prefers the use of - script: steps over specialized tasks (like NuGet@2 or npm) for most operations. Specialized tasks should be reserved for complex operations that are difficult to script.
    • Authentication Hierarchy: To manage authentication to private feeds, the system follows a specific preference order for providing credentials:
      1. Credential Providers: The preferred method for most ecosystems.
      2. Environment Variables: Used if credential providers are unavailable.
      3. Workspace Configuration Files: Tokens are written to a configuration file in the build's workspace directory (above the sources directory) if the above are not possible.
      4. Specific Tool Configuration: If a tool requires a file in a specific location, the system will write to the user-provided file or create a new one.
    • Security & Cleanup: All implementations must ensure build/commit provenance for published packages, support corporate proxies, and use post-tasks to clean up credentials from the workspace.
  3. Understand the difference between Pipeline variables and Environment variables

    master
    In Azure Pipelines, 'variables' is an overloaded term. Most pipeline variables are automatically converted into environment variables on the agent. However, there is a critical distinction regarding secrets: secrets are NOT automatically injected into the environment for security reasons. You must manually map secrets into the environment using the env statement within an individual step. Additionally, some variables are only available in the YAML expression context and are not propagated to the environment by default.
  4. Use lifecycle hooks for safe deployments

    master

    To enable safe deployments (initialization, deployment, testing, and rollback), you can use lifecycle hooks within a deployment strategy. These hooks allow you to run specific steps at different stages of the deployment lifecycle.

    If a lifecycle hook event fails, it can trigger an automatic rollback. The type of agent used for these hooks is determined by the pool attribute; by default, they inherit the pool from the deployment job, but can be overridden (e.g., using pool: server).

    Lifecycle Hooks:

    • preDeploy: Run tasks before the deploy step.
    • deploy: Run the actual deployment tasks.
    • routeTraffic: Run tasks to serve traffic to the updated version.
    • postRouteTraffic: Run tasks after traffic is routed, typically used for monitoring health for a defined interval.
    • on: failure: Run tasks to perform rollback actions (does not trigger further rollback).
    • on: success: Run tasks to perform cleanup or notifications (does not trigger further rollback).
  5. Use the `target` property to specify step execution context

    master

    By default, steps in a job run in a single context: either on the agent host or inside the job's specified container. You can use the target property on a step to override this default and choose where that specific step executes.

    Available Targets:

    • Service Containers: You can target a specific service container defined in the services section of a job (e.g., target: postgres).
    • host (or self): Targets the agent host machine where the worker is running.
    • Default (No target specified): If the target property is omitted, the step runs in the job's default context (the job container if one is defined, otherwise the host).

    Constraints:

    • All steps in a job run in one context; separate steps cannot target different contexts throughout the job (though the target property allows switching between the host, job container, and services).
    • Steps cannot target service containers if they are not explicitly defined in the job's services section.
    steps:
    - script: ...
      displayName: Configure postgres
      target: postgres  # targets a service container
    - script: ...
      displayName: Run tests
      # no target, so targets the default (the job container)
    - script: ...
      target: host  # targets the host machine
  6. Understand Azure Pipelines YAML versioning

    master

    Azure Pipelines uses a version keyword at the root level of the pipeline YAML to manage breaking changes and syntax evolutions.

    • Version 1 (Default): The current syntax with existing backward compatibility. If no version is specified, the pipeline is assumed to be Version 1. No new features will be added to this version.
    • Version 2: The active development version which introduces breaking changes to clean up the schema (e.g., removing queue in favor of pool, and restructuring triggers).

    Note: The project has officially decided not to proceed with YAMLv2. This document is maintained for historical context regarding the design intent.

  7. Use relative paths for the `checkout: self` path property

    master

    When using the path property in a checkout step, use relative paths. These are supported cross-platform (Windows and Linux).

    For example, path: foo/src will resolve to $(Pipeline.Workspace)/foo/src.

    Note: Absolute paths are not supported to ensure cross-platform compatibility, security, and proper agent cleanup.

    steps:
    - checkout: self
      path: foo/src
  8. Accessing downloaded artifact paths via variables

    master

    When using pipeline resources, you can access the disk location of checked-out resources using specific variable syntaxes. This is useful for scripts that need to know exactly where an artifact was placed.

    • Pipeline Variable Syntax: $(Pipeline.Resources.<resource_name>)
    • Environment Variable Syntax: $PIPELINE_RESOURCES_<RESOURCE_NAME_UPPERCASE>

    Example: If you have a pipeline resource named foo, you can access its location via $(Pipeline.Resources.foo) or $PIPELINE_RESOURCES_FOO.

    resources:
      pipelines:
      - pipeline: foo
    
    jobs:
    - job: consumer
      steps:
      - bash: |
          echo $(Pipeline.Resources.foo)   # pipeline variable syntax
          echo $PIPELINE_RESOURCES_FOO     # environment variable syntax
  9. Use the 'each' template expression for iteration

    master

    The ${{ each <item> in <collection> }}: expression allows you to iterate over a sequence or a mapping within an Azure Pipelines YAML template. This is primarily used to dynamically generate steps, jobs, or other YAML structures based on input parameters.

    Iterative sequence insertion

    You can use each to inject a sequence of items into an existing list. This is useful for wrapping user-provided items with mandatory pre-steps or post-steps.

    mySequence:
    - outer pre
    - ${{ each myItem in parameters.myCollection }}:
      - nested pre
      - ${{ myItem }}
      - nested post
    - outer post
  10. Use deployment jobs for environment targeting

    master

    To leverage environment-specific features like deployment strategies and traceability, use the deployment job type instead of a standard job. A deployment job is a collection of steps run against a specific environment. It supports specialized strategies such as runOnce, canary, blueGreen, and rolling (note: some strategies may be subject to future support availability).

    - deployment: deployWeb
      displayName: Deploy web pkg
      pool:
        vmImage: 'Ubuntu 16.04'
      environment: production    # create environment and/or record deployments
      strategy:
        runOnce:                 # default strategy
          deploy:
            steps:       
            - script: echo deploy web pkg
  11. Configure deployment strategies in Azure Pipelines YAML

    master

    Azure Pipelines uses deployment jobs to manage application updates through specific orchestration strategies. The strategy defines how updates are rolled out and how the system responds to health checks or failures.

    Available strategies include:

    • runOnce: The default strategy if none is specified. It executes a single deployment sequence.
    • canary: Reduces risk by rolling out changes to a small subset of users incrementally (e.g., 10%, then 20%).
    • rolling: Replaces instances of the previous version with the new version on a fixed set of machines in iterations, often used to ensure availability by waiting for readiness checks.

    Each strategy is defined under the strategy key within a deployment job.

    jobs:
    - deployment:
      environment: musicCarnivalProd
      pool:
        name: musicCarnivalProdPool  
      strategy:                 
        runOnce:              
          deploy:    
            steps:             
            - script: echo deploy web app...   
  12. Use the new Pipeline.* variable namespace

    master

    To support unified pipelines (merging Build and Release models), a new Pipeline.* namespace has been introduced. These variables replace the deprecated Build.* and Release.* namespaces.

    Note that many new variables are not in the environment by default and must be explicitly mapped in a step's env section if you need them in a script. Some variables are only available via YAML expressions (e.g., for template expansion or conditional logic).