ModularPipelines

repository·main·Indexed 19 days ago

https://github.com/thomhurst/modularpipelines

A C# framework for writing CI/CD pipelines as strongly-typed, debuggable code instead of YAML. It enables developers to use IDE features like IntelliSense and breakpoints, providing automatic parallel execution based on module dependencies, dependency collision detection, and a wide range of strongly-typed wrappers for cloud providers (AWS, Azure, GCP), containerization tools (Docker, Kubernetes, Helm), and CI/CD services.

Tokens
571.3K
Snippets
657
Records
857
Agent score
67%

What's inside ModularPipelines

  1. Overview of ModularPipelines

    main
    ModularPipelines is a .NET framework for defining pipelines using strong types, IntelliSense, and the full .NET ecosystem. Unlike YAML-based pipelines or scripted C# frameworks (like Cake), ModularPipelines organizes work into separate, strongly-typed module classes. This approach promotes the Single Responsibility Principle (SRP), enables automatic parallelization of independent tasks, and allows for easy debugging and local execution using standard dotnet run commands.
  2. Overview of Modular Pipelines

    main
    Modular Pipelines is a C# framework designed to replace YAML-based pipeline definitions with strongly-typed C# code. It allows developers to build complex workflows using the full power of the .NET ecosystem, providing features like IntelliSense, parallel execution, and dependency management. Instead of learning a new domain-specific language (DSL) like YAML, you write familiar C# code that is source-controllable, debuggable, and build-agent agnostic.
  3. Overview of ModularPipelines.OptionsGenerator

    main

    ModularPipelines.OptionsGenerator is a tool that converts CLI metadata into strongly typed C# code (options, services, extension methods, enums) and Markdown documentation. It is used to power ModularPipelines tool integrations.

    Generation Paths

    • First-party integrations: Generated by scraping installed CLI help (preferred) or public HTML documentation (fallback).
    • Private/External integrations: Generated from a versioned JSON definition using the --input <path> flag.

    Note: Generated files are outputs. Do not edit them manually; instead, modify the scraper, generator, type override, or JSON definition and regenerate.

  4. Use Sub-Modules to organize execution blocks

    main

    Sub-modules allow you to group and track specific blocks of execution within a larger module. They are particularly useful when iterating through data (e.g., in a loop) where you want to isolate failures and monitor progress for each individual item.

    Key benefits:

    • Error Tracking: If a submodule fails, it throws an exception containing the provided submodule name, allowing you to identify exactly which item in a collection caused the failure.
    • Progress Monitoring: The console progress dialog displays the time taken for each individual submodule execution.

    To create a submodule, use the context.SubModule method, which takes two arguments:

    1. A string name (used for identification in errors and progress logs).
    2. A Func<Task<T>> (or similar delegate) containing the code to execute.
    // Example pattern for using submodules in a loop
    foreach (var item in items)
    {
        yield return await context.SubModule(item.Name, () => context.Tools.SomeTool.ExecuteAsync(item, cancellationToken));
    }
  5. Access required values vs safe accessors

    main

    Depending on whether a dependency's success is mandatory, you have two ways to access data:

    1. Required Values (.Value)

    Use the .Value property when the dependency must have produced a non-null value. If the module failed, was skipped, or returned null, accessing .Value will throw an InvalidOperationException identifying the specific module and outcome.

    2. Safe Accessors

    If an absent value is expected, avoid .Value and instead inspect the union using non-throwing accessors or pattern matching:

    • Use if (result is ModuleResult<T>.Success success) to access the value.
    • Use result.ExceptionOrDefault to check for errors.
    • Use result.SkipDecisionOrDefault to check for skip reasons.
    var result = await context.GetModule<MyModule>();
    
    // Safe pattern matching
    if (result is ModuleResult<MyResult>.Success success)
    {
        var value = success.Value;
        // Process value
    }
    
    // Safe property access
    if (result.ExceptionOrDefault is { } exception)
    {
        // Handle error
    }
    
    if (result.SkipDecisionOrDefault is { } skipDecision)
    {
        // Handle skip
    }
  6. How Run Identifier Resolution works

    main

    Distributed coordination requires an invocation-scoped identifier to isolate Redis namespaces. The system resolves the RunIdentifier using the following priority:

    1. Explicit Configuration: The value set in RedisDistributedOptions.RunIdentifier.
    2. Environment Variable: The RUN_IDENTIFIER environment variable.

    Important Requirements:

    • Isolation: Rerunning the same commit must receive a fresh Redis namespace. Do not use commit identifiers as run identifiers.
    • Orchestration: For local multi-process runs, export a unique RUN_IDENTIFIER before starting the master and workers. CI workflows must similarly generate and export a unique RUN_IDENTIFIER for every invocation.
  7. Handle Failed Modules

    main

    When a module execution fails (after all retry attempts are exhausted), the following failure sequence occurs:

    1. Module<T>.OnFailedAsync (Virtual Hook)
    2. Module<T>.OnAfterExecuteAsync (Virtual Hook), receiving a failed ModuleResult<T>.
    3. IModuleFailureHandler (Attribute Handler)
    4. IModuleEventReceiver.OnModuleFailureAsync (Global Event Receiver)

    If a configured failure condition is set to ignore the failure, the resulting module status will reflect that policy instead of a failure state.

  8. Mozilla Public License (MPL) 2.0 Overview

    main

    The Mozilla Public License 2.0 is a weak copyleft license. It allows you to combine Covered Software with other material to create a Larger Work, provided you comply with the license requirements for the Covered Software portion.

    Key Terms

    • Covered Software: The source code form to which the initial contributor has attached the notice in Exhibit A, the executable form, and modifications.
    • Larger Work: A work that combines Covered Software with other material in separate files that is not Covered Software.
    • Modifications: Any file in Source Code Form that results from an addition, deletion, or modification of the contents of Covered Software, or any new file containing Covered Software.
    • Secondary License: Includes the GNU General Public License (GPL) versions 2.0, 2.1, and Affero GPL 3.0.

    Distribution Requirements

    • Source Code Form: All distribution of Covered Software in Source Code Form (including Modifications) must be under the terms of the MPL. You must inform recipients how they can obtain a copy of the license.
    • Executable Form: If you distribute the software in Executable Form, you must also make the Source Code Form available via reasonable means at a charge no more than the cost of distribution.
    • Larger Works: You may distribute a Larger Work under terms of your choice, but you must comply with the MPL for the Covered Software component.
  9. Configure module dependencies and parallelization

    main

    ModularPipelines attempts to run modules in parallel by default. To ensure a module runs only after another has completed, use the [DependsOn<T>] attribute on your module class. This creates an explicit dependency that the pipeline engine respects.

    [DependsOn<MyOtherModule>]
    public class MyModule : Module<string>
    {
        protected override async Task<string> ExecuteAsync(IModuleContext context, CancellationToken cancellationToken)
        {
            // MyOtherModule is guaranteed to have completed before this runs
            return "result";
        }
    }
  10. How OS capability auto-detection works

    main

    By default, AutoDetectOsCapability is true. The framework automatically adds the current operating system as a capability to the worker:

    • Windows runners advertise "windows"
    • Linux runners advertise "linux"
    • macOS runners advertise "macos"

    Additionally, if a module uses [RunIfAll<OnLinux>], [RunIfAll<OnWindows>], or [RunIfAll<OnMacOS>], the framework automatically adds the corresponding OS capability requirement to that module. This prevents the need to manually add [RequiresCapability("linux")] when using platform-specific attributes.

    // The "linux" capability is auto-detected — no [RequiresCapability] needed
    [RunIfAll<OnLinux>]
    public class LinuxBuildModule : Module<string>
    {
        protected override async Task<string> ExecuteAsync(
            IModuleContext context, CancellationToken cancellationToken)
        {
            // Only executes on workers that have the "linux" capability
            return "built on linux";
        }
    }
  11. Handle Module Results in V3

    main

    V3 provides three primary patterns for handling ModuleResult<T>:

    1. Pattern Matching (Recommended): Use C# switch expressions to handle Success, Skipped, and Failure states explicitly.
    2. Match Helper: Use the .Match() extension method to provide callbacks for success, failure, and skipped states.
    3. Simple Property Access: Use .ValueOrDefault and check if the result is ModuleResult<T>.Success for quick migrations.
    // Pattern 1: Pattern matching (recommended)
    var result = await context.GetModule<BuildModule>();
    return result switch
    {
        ModuleResult<BuildOutput>.Success { Value: var output } => Process(output),
        ModuleResult<BuildOutput>.Skipped => null,
        ModuleResult<BuildOutput>.Failure { Exception: var ex } => throw ex,
        _ => null
    };
    
    // Pattern 2: Match helper
    var result = await context.GetModule<BuildModule>();
    return result.Match(
        onSuccess: output => Process(output),
        onFailure: ex => throw ex,
        onSkipped: skip => null
    );
    
    // Pattern 3: Simple property access
    var result = await context.GetModule<BuildModule>();
    if (result is ModuleResult<BuildOutput>.Success)
    {
        var value = result.ValueOrDefault;
    }
  12. Benefits of using ModularPipelines for .NET developers

    main

    ModularPipelines allows you to define build and deployment logic using C# and .NET instead of YAML, PowerShell, or Bash. This provides several advantages:

    • Strong Typing: Modules are structured as objects, ensuring data availability and type safety throughout the pipeline.
    • Ecosystem Integration: You can leverage existing .NET libraries and features instead of reinventing logic in a domain-specific language.
    • Reduced Context Switching: .NET developers can manage pipelines using familiar syntax and tools without learning unfamiliar build system UIs or scripting languages.