Nomad Autoscaler

The Nomad Autoscaler is an autoscaling daemon for Nomad designed with a plugin architecture for extensibility. It supports three primary scaling modes:

1. Horizontal Application Autoscaling: Automatically adjusts the number of task group allocations based on metrics (e.g., CPU, memory, or connection counts) to maintain service-level agreements (SLAs).
2. Horizontal Cluster Autoscaling: Manages the size of the Nomad cluster by interacting with remote providers to add or remove Nomad clients based on available cluster resources (e.g., free CPU or memory).
3. Dynamic Application Sizing (Enterprise): Optimizes application resource consumption by analyzing historical usage data to provide CPU and Memory resource recommendations.

To add a new test case for the Nomad policy source, follow these steps:

1. Create a new Nomad jobspec file inside the src directory.
2. Generate the corresponding golden file from that jobspec using the make commands described in the regeneration guides.

If you only need to update a single golden file, you can specify the target file directly with make.

make -B full-scaling.json.golden

You can build the Nomad Autoscaler binary for local testing or development using the make command. The compiled binary will be located at ./bin/nomad-autoscaler.

make dev

Use make to regenerate all golden files from the source HCL jobspecs. You can use the -j flag to run the generation in parallel to improve speed.

# Regenerate all files sequentially
make -B

# Regenerate all files in parallel (replace <NUM_OF_CORES> with your core count)
make -B -j <NUM_OF_CORES>

Golden files are generated from the source HCL jobspecs located in the src directory. The generation process relies on the Nomad API to parse HCL into JSON.

Prerequisite: You must have a Nomad agent running in development mode to facilitate the parsing:

nomad agent -dev

For testing the latest features and bug fixes, you can use the nightly release or preview Docker images:

• Nightly Releases: Binaries built from the latest main branch code are available on the GitHub releases page.
• Docker Preview Images: Every commit to main generates a preview image available on Docker Hub at hashicorppreview/nomad-autoscaler.

Choosing whether to enable readiness override options depends on your cluster's priority between stability and responsiveness.

Use these options when:

• You run spot-heavy clusters with frequent node replacement events.
• Your environment has long drain windows (e.g., due to long-running jobs).
• Your priority is restoring headroom quickly and meeting SLAs over strict stability.

Do NOT use these options when:

• You require strict stability and conservative scaling behavior.
• Capacity accounting accuracy is more important than scaling speed.
• Your workloads are highly sensitive to short-term over/under scaling during transitions.

To run multiple instances of the autoscaler in HA mode, use the -high-availability-enabled flag. This triggers leader election using a lock in Nomad. All instances must use the same lock-namespace, lock-path, lock-ttl, and lock-delay to participate in the same election.

nomad-autoscaler agent -high-availability-enabled -high-availability-lock-path="/autoscaler/lock" -high-availability-lock-namespace="default"

To create a custom plugin for the Nomad Autoscaler, you must implement the Base interface provided by the sdk package. The base.pluginServer (an internal gRPC implementation) wraps your implementation of Base to expose it to the Nomad Autoscaler via gRPC.

Your implementation of Base must satisfy the following methods which are mapped to the gRPC server:

• PluginInfo() (*sdk.PluginInfo, error): Returns metadata about the plugin, including its Name and PluginType (e.g., sdk.PluginTypeAPM, sdk.PluginTypeStrategy, or sdk.PluginTypeTarget).
• SetConfig(config map[string]interface{}) error: Receives the configuration provided by the Nomad Autoscaler and applies it to the plugin.

The nomad-autoscaler agent command starts the Autoscaler agent and runs until an interrupt is received. Configuration is primarily managed via config files, but can be partially overridden using CLI arguments or environment variables.

Usage:

nomad-autoscaler agent [options] [args]

nomad-autoscaler agent -config=/path/to/config.hcl