Overview of ModularPipelines
maindotnet run commands.repository·main·Indexed 19 days ago
https://github.com/thomhurst/modularpipelinesA C# framework for writing CI/CD pipelines as strongly-typed, debuggable code instead of YAML. It enables developers to use IDE features like IntelliSense and breakpoints, providing automatic parallel execution based on module dependencies, dependency collision detection, and a wide range of strongly-typed wrappers for cloud providers (AWS, Azure, GCP), containerization tools (Docker, Kubernetes, Helm), and CI/CD services.
dotnet run commands.ModularPipelines.OptionsGenerator is a tool that converts CLI metadata into strongly typed C# code (options, services, extension methods, enums) and Markdown documentation. It is used to power ModularPipelines tool integrations.
--input <path> flag.Note: Generated files are outputs. Do not edit them manually; instead, modify the scraper, generator, type override, or JSON definition and regenerate.
Sub-modules allow you to group and track specific blocks of execution within a larger module. They are particularly useful when iterating through data (e.g., in a loop) where you want to isolate failures and monitor progress for each individual item.
Key benefits:
To create a submodule, use the context.SubModule method, which takes two arguments:
string name (used for identification in errors and progress logs).Func<Task<T>> (or similar delegate) containing the code to execute.// Example pattern for using submodules in a loop
foreach (var item in items)
{
yield return await context.SubModule(item.Name, () => context.Tools.SomeTool.ExecuteAsync(item, cancellationToken));
}Depending on whether a dependency's success is mandatory, you have two ways to access data:
.Value)Use the .Value property when the dependency must have produced a non-null value. If the module failed, was skipped, or returned null, accessing .Value will throw an InvalidOperationException identifying the specific module and outcome.
If an absent value is expected, avoid .Value and instead inspect the union using non-throwing accessors or pattern matching:
if (result is ModuleResult<T>.Success success) to access the value.result.ExceptionOrDefault to check for errors.result.SkipDecisionOrDefault to check for skip reasons.var result = await context.GetModule<MyModule>();
// Safe pattern matching
if (result is ModuleResult<MyResult>.Success success)
{
var value = success.Value;
// Process value
}
// Safe property access
if (result.ExceptionOrDefault is { } exception)
{
// Handle error
}
if (result.SkipDecisionOrDefault is { } skipDecision)
{
// Handle skip
}Distributed coordination requires an invocation-scoped identifier to isolate Redis namespaces. The system resolves the RunIdentifier using the following priority:
RedisDistributedOptions.RunIdentifier.RUN_IDENTIFIER environment variable.Important Requirements:
RUN_IDENTIFIER before starting the master and workers. CI workflows must similarly generate and export a unique RUN_IDENTIFIER for every invocation.When a module execution fails (after all retry attempts are exhausted), the following failure sequence occurs:
Module<T>.OnFailedAsync (Virtual Hook)Module<T>.OnAfterExecuteAsync (Virtual Hook), receiving a failed ModuleResult<T>.IModuleFailureHandler (Attribute Handler)IModuleEventReceiver.OnModuleFailureAsync (Global Event Receiver)If a configured failure condition is set to ignore the failure, the resulting module status will reflect that policy instead of a failure state.
The Mozilla Public License 2.0 is a weak copyleft license. It allows you to combine Covered Software with other material to create a Larger Work, provided you comply with the license requirements for the Covered Software portion.
ModularPipelines attempts to run modules in parallel by default. To ensure a module runs only after another has completed, use the [DependsOn<T>] attribute on your module class. This creates an explicit dependency that the pipeline engine respects.
[DependsOn<MyOtherModule>]
public class MyModule : Module<string>
{
protected override async Task<string> ExecuteAsync(IModuleContext context, CancellationToken cancellationToken)
{
// MyOtherModule is guaranteed to have completed before this runs
return "result";
}
}By default, AutoDetectOsCapability is true. The framework automatically adds the current operating system as a capability to the worker:
"windows""linux""macos"Additionally, if a module uses [RunIfAll<OnLinux>], [RunIfAll<OnWindows>], or [RunIfAll<OnMacOS>], the framework automatically adds the corresponding OS capability requirement to that module. This prevents the need to manually add [RequiresCapability("linux")] when using platform-specific attributes.
// The "linux" capability is auto-detected — no [RequiresCapability] needed
[RunIfAll<OnLinux>]
public class LinuxBuildModule : Module<string>
{
protected override async Task<string> ExecuteAsync(
IModuleContext context, CancellationToken cancellationToken)
{
// Only executes on workers that have the "linux" capability
return "built on linux";
}
}V3 provides three primary patterns for handling ModuleResult<T>:
Success, Skipped, and Failure states explicitly..Match() extension method to provide callbacks for success, failure, and skipped states..ValueOrDefault and check if the result is ModuleResult<T>.Success for quick migrations.// Pattern 1: Pattern matching (recommended)
var result = await context.GetModule<BuildModule>();
return result switch
{
ModuleResult<BuildOutput>.Success { Value: var output } => Process(output),
ModuleResult<BuildOutput>.Skipped => null,
ModuleResult<BuildOutput>.Failure { Exception: var ex } => throw ex,
_ => null
};
// Pattern 2: Match helper
var result = await context.GetModule<BuildModule>();
return result.Match(
onSuccess: output => Process(output),
onFailure: ex => throw ex,
onSkipped: skip => null
);
// Pattern 3: Simple property access
var result = await context.GetModule<BuildModule>();
if (result is ModuleResult<BuildOutput>.Success)
{
var value = result.ValueOrDefault;
}ModularPipelines allows you to define build and deployment logic using C# and .NET instead of YAML, PowerShell, or Bash. This provides several advantages: