repository·master·Indexed 12 days ago
https://github.com/netflix/chaosmonkeyA tool that randomly terminates virtual machine instances and containers in production environments to incentivize the development of resilient services. Designed to work with Spinnaker, it supports backends including AWS, Google Compute Engine (GCE), Azure, Kubernetes, and Cloud Foundry. It includes a CLI for managing terminations, scheduling via cron jobs, and identifying eligible instances.
Chaos Monkey is a tool designed to randomly terminate instances in production environments. This process forces engineers to build services that are resilient to unexpected instance failures.
To begin using Chaos Monkey, follow the deployment guide to get the service running, then use Spinnaker to configure specific behaviors for your applications.
Chaos Monkey operates on groups of instances. You can define the scope of a group using one of three options:
Region Independence: By default, Chaos Monkey treats regions separately. If you uncheck the regions are independent option, Chaos Monkey will avoid terminating instances that belong to the same group but reside in different regions. This is useful for protecting cross-region replicated databases.
Chaos Monkey determines how often to terminate instances for an application based on two parameters defined at the app level. Every weekday, Chaos Monkey performs a weighted coin flip for each instance group to decide if a termination should be scheduled between 9AM and 3PM.
To control this behavior, configure:
μ (mean time between terminations in work days): This determines the probability p of a termination occurring on any given day. The relationship is approximately p = 1/μ.ε (min time between terminations in work days): This sets a minimum threshold of work days that must pass before a termination can occur.Key Behaviors:
μ=1, then p=1, guaranteeing a termination every day.ε > 1, the distribution is no longer strictly geometric, and the actual mean time between terminations will be larger than μ.Chaos Monkey supports dynamic configuration for specific properties using etcd or Consul. This allows operators to change behavior without redeploying.
Supported dynamic properties:
chaosmonkey.enabledchaosmonkey.leashedchaosmonkey.schedule_enabledchaosmonkey.accountsImplementation details:
[dynamic] section to your configuration file containing the service endpoint and a path to a JSON file containing the properties.chaosmonkey.toml configuration file takes precedence over the dynamic provider. To allow dynamic updates to work, do not specify these properties in your static config file.isDisabled boolean flag to filter groups, ensuring that only active groups are targeted for termination.Constrainer interface, to integrate with their organization's specific systems.Outage interface is the contract required to implement a custom outage checker plugin. Any type implementing this interface can be used by Chaos Monkey to determine if it should automatically disable itself during an ongoing outage.Chaos Monkey is designed to work with Spinnaker, the continuous delivery platform. To use Chaos Monkey for terminating instances, you must be managing your applications with Spinnaker.
Chaos Monkey supports any backend that Spinnaker supports, including:
It has been explicitly tested with AWS, GCE, and Kubernetes.
If you want to avoid the overhead of the tests starting and stopping the MySQL container every time (for example, to run tests faster or to use a native MySQL instance), use both the docker and dockerup tags. This allows the tests to interact with an existing environment rather than managing the container lifecycle.
go test -tags "docker dockerup" ./...Chaos Monkey supports encrypted passwords for the MySQL database and Spinnaker p12 certificates. Because Chaos Monkey does not ship with built-in decryptor implementations, you must implement the Decryptor interface in Go to use encrypted credentials at runtime.
To implement a custom decryptor, follow these steps:
"gpg").Decryptor interface.decryptor.go in the Chaos Monkey source code to ensure the application recognizes and can instantiate your new decryptor type.Chaos Monkey uses a configuration file in TOML format. It searches for a file named chaosmonkey.toml in the following locations in order:
. (the current working directory)/apps/chaosmonkey/etc/etc/chaosmonkeyTo add custom functionality (like a custom Constrainer) to Chaos Monkey, you must implement the required interface, import the implementation into a custom main.go using a blank import to trigger its init() function, and then build a new binary.
This guide assumes your code resides in $GOPATH/src/example.com/chaosmonkey (replace example.com with your organization's domain).
### 1. Grab the open source Chaos Monkey source
```bash
go get github.com/netflix/chaosmonkey/cmd/chaosmonkeyFile: $GOPATH/src/example.com/chaosmonkey/constrainer.go
File: $GOPATH/src/example.com/chasmonkey/cmd/chaosmonkey/main.go
package main
import (
"github.com/Netflix/chaosmonkey/command"
_ "example.com/chaosmonkey/constrainer"
)
func main() {
command.Execute()
}go build example.com/chaosmonkey/cmd/chaosmonkey