terraform-aws-gitlab-runner

This Terraform module automates the deployment of GitLab Runners on AWS, specifically optimized for using Spot Instances to reduce EC2 costs. It utilizes the docker+machine executor to scale builds automatically.

Key features include:

• Spot Instance Support: Uses spot instances by default for build executors.
• Shared S3 Cache: Provides an S3 bucket for shared caching across runners with configurable lifecycle management to clean up old objects.
• CloudWatch Logging: Automatically streams runner logs to AWS CloudWatch.
• Automatic Registration: Runner agents are automatically registered with GitLab.

Note: If you are upgrading from older versions, refer to the migration guides for v7 (issue 819) and v8 (pr 1204).

The GitLab runner EC2 instances require two specific AWS service linked roles to function correctly:

1. AWSServiceRoleForAutoScaling
2. AWSServiceRoleForEC2Spot

By default, the module attempts to create these roles automatically. If you set allow_iam_service_linked_role_creation = false, you must ensure these roles exist beforehand. You can create them manually or via Terraform using the following configuration:

resource "aws_iam_service_linked_role" "spot" {
  aws_service_name = "spot.amazonaws.com"
}

resource "aws_iam_service_linked_role" "autoscaling" {
  aws_service_name = "autoscaling.amazonaws.com"
}

The module supports three distinct architectural patterns for deploying GitLab Runners:

1. Single GitLab CI docker-machine runner

In this setup, a single runner agent runs on one EC2 node. It uses docker machine to provision additional spot instances for builds. The module creates a default S3 cache that is shared across all spawned spot instances.

2. Multiple GitLab CI docker-machine runners

You can instantiate the module multiple times to create multiple runner agents, each with different configurations. To share a cache across these multiple agents, you must manage the S3 cache externally (outside the individual module instances).

3. GitLab CI docker runner

This scenario uses the standard docker executor instead of docker-machine. Builds run directly on the same EC2 instance as the agent. Note: Auto-scaling is not supported in this mode.

The terminate-instances module provides a Lambda function that acts as an AWS Auto Scaling Group (ASG) lifecycle hook.

Core Logic:

1. Instance Tracking: The Lambda evaluates EC2 instances for a specific tag: gitlab-runner-parent-id. This tag is applied to worker instances by the parent module's user data, linking them to the parent runner's instance ID.
2. Cleanup Trigger: When a parent runner instance in the ASG is terminated, the lifecycle hook triggers this Lambda to:
   • Terminate all spawned runner instances that share the matching gitlab-runner-parent-id tag.
   • Terminate "orphaned" instances that have no running parent runner.
3. Spot Request Management: The Lambda also cancels open spot requests. This prevents costs from being incurred by spot requests that might be fulfilled later for instances that no longer have a running parent.

The module implements a structured zero-downtime deployment process using AWS Auto Scaling Group (ASG) lifecycle hooks:

1. Provisioning: A new instance is set to pending. It has up to 5 minutes to install the GitLab Runner and provision capacity.
2. In-Service: Once ready, the new instance is set to InService.
3. Graceful Shutdown: The old (current) instance is moved to a terminating:wait state. This triggers the monitor_runner.sh systemd service, which sends a SIGQUIT to the GitLab Runner process. The runner stops accepting new jobs and finishes existing ones.
4. Termination: Once the runner finishes jobs, it sends a complete-lifecycle-action to the ASG. The ASG then moves the instance to terminating:proceed and terminates it.

The maximum time allowed for the shutdown process is controlled by the runner_terminate_ec2_lifecycle_timeout_duration variable.

The following versions are required for this deployment:

Terraform Versioning It is recommended to use tfenv to manage the Terraform version. Check the .terraform-version file in the repository for the exact tested version.

Provider and Module Requirements

• Terraform: >= 1.3
• AWS Provider: >= 5.78.0
• Local Provider: >= 2.5.2
• Null Provider: >= 3.2.3
• Random Provider: >= 3.6.3
• TLS Provider: >= 4.0.6

Modules Used

• runner: ../../ (the core module)
• vpc: terraform-aws-modules/vpc/aws >= 5.16.0
• vpc_endpoints: terraform-aws-modules/vpc/aws//modules/vpc-endpoints >= 5.16.0

This example demonstrates a deployment where a single EC2 instance acts as the GitLab Runner agent. The agent uses docker+machine to automatically scale runners using AWS Spot Instances.

Key Features:

• Auto-scaling: Uses the docker+machine executor to scale based on demand.
• Shared Cache: Automatically creates an S3 bucket to serve as a shared cache across all spot instance runners.
• Connectivity: Supports both public and private VPC configurations.
• Access: Allows instance access via AWS SSM (Session Manager).
• Multi-region: Can be deployed across multiple regions by instantiating the module multiple times with different AWS providers (ensure each region has its own S3 cache).

Configuration Note: You can configure a Docker registry mirror by uncommenting the runners.docker.services section in the configuration.

The cache module creates an S3 bucket specifically designed for GitLab runner build caches. It automatically manages a lifecycle policy to expire old cache objects and generates an IAM policy that can be passed to GitLab runners to grant them access to the bucket.

To use this module with the main GitLab runner module, instantiate the cache module and pass its bucket and policy_arn outputs into the cache_bucket configuration block of the runner module.

module "cache" {
  source      = "https://github.com/cattle-ops/terraform-aws-gitlab-runner/tree/move-cache-to-moudle/cache"
  environment = "cache"
}

module "runner" {
  source  = "cattle-ops/gitlab-runner/aws"

  # ... other runner configuration ...

  cache_bucket = {
    create = false
    policy = "${module.cache.policy_arn}"
    bucket = "${module.cache.bucket}"
  }
}