Toxiproxy

Toxiproxy is a framework for simulating network conditions, designed specifically for testing, CI, and development environments. It allows for deterministic tampering with connections, as well as randomized chaos and customization.

Its architecture consists of two parts:

1. A TCP proxy: Written in Go, this handles the actual network traffic.
2. A client: Communicates with the proxy over HTTP to manipulate connection health and simulate various network failures (toxics).

By routing application connections through Toxiproxy, you can prove that your application is resilient to single points of failure by simulating latency, bandwidth limits, packet loss, and connection outages.

You can store configuration values directly in your toxic struct. Because the Toxiproxy API encodes and decodes these structs using JSON, all public fields in your struct will be accessible via the API.

Note on State: A separate instance of the toxic exists for every connection. Do not rely on struct fields to persist state across interrupts; instead, use local variables within Pipe() or implement the StatefulToxic interface for connection-specific state.

type LatencyToxic struct {
    Latency int64 `json:"latency"` 
    Jitter  int64 `json:"jitter"`  
}

To collect metrics related to Toxiproxy's internal proxy operations (such as bytes sent and received), start Toxiproxy with the -proxy-metrics flag.

toxiproxy -proxy-metrics

To use the Go client, import github.com/Shopify/toxiproxy/v2/client and initialize a new client by providing the address of your Toxiproxy server (e.g., localhost:8474).

import toxiproxy "github.com/Shopify/toxiproxy/v2/client"

client := toxiproxy.NewClient("localhost:8474")

To collect metrics related to the Go runtime state, build version, and process information, start Toxiproxy with the -runtime-metrics flag. These metrics follow the standard Prometheus collectors for Go, including NewGoCollector, NewBuildInfoCollector, and NewProcessCollector.

toxiproxy -runtime-metrics

To use your custom toxics, you must compile a custom Toxiproxy binary. Instead of forking the entire repository, you can:

1. Copy the server implementation from cmd/server/server.go into a new project.
2. Register your custom toxic using an init() function.
3. Compile your new project as a standalone binary.

If your toxic needs to modify the data stream, you can use the ChanReader and ChanWriter interfaces from the stream package. These allow you to treat the input and output channels as standard io.Reader and io.Writer objects.

When using these, ensure you call reader.SetInterrupt(stub.Interrupt) so that the standard Read calls can return stream.ErrInterrupted when the toxic is stopped.

func (t *NoopToxic) Pipe(stub *toxics.ToxicStub) {
    buf := make([]byte, 32*1024)
    writer := stream.NewChanWriter(stub.Output)
    reader := stream.NewChanReader(stub.Input)
    reader.SetInterrupt(stub.Interrupt)
    for {
        n, err := reader.Read(buf)
        if err == stream.ErrInterrupted {
            writer.Write(buf[:n])
            return
        } else if err == io.EOF {
            stub.Close()
            return
        }
        writer.Write(buf[:n])
    }
}

The ProxyCollection type is used to manage a set of Proxy objects. It ensures the integrity of the proxy set by preventing duplicate names and provides thread-safe operations for adding, retrieving, removing, and clearing proxies.

Key behaviors:

• Thread Safety: Uses a sync.RWMutex to allow concurrent reads but exclusive writes.
• Lifecycle Management: Methods like Add and Remove can trigger the Start() or Stop() methods on the underlying Proxy objects.
• Integrity: Prevents adding multiple proxies with the same name via the Add method.

collection := toxiproxy.NewProxyCollection()

// Adding a proxy and starting it immediately
err := collection.Add(myProxy, true)

// Retrieving all proxies as a map
allProxies := collection.Proxies()

A Proxy manages its own lifecycle using a tomb.Tomb for graceful shutdowns.

When Start() is called:

1. A background server() goroutine is launched.
2. The proxy begins listening on the Listen address.
3. For every new client connection accepted, the proxy dials the Upstream address.
4. It creates two 'Links' (upstream and downstream) and registers them in the Toxics collection to allow for chaos injection.
5. Connections are tracked in a ConnectionList.

When Stop() is called:

1. The proxy enters a dying state via tomb.
2. The listener is closed to stop accepting new connections.
3. All existing connections in the ConnectionList are closed.

The ApiServer uses a mux.Router to handle incoming requests. It applies several layers of middleware to every request:

1. Logging: Uses hlog to integrate zerolog with the router, providing request IDs and access logs (including client IP, method, URL, status, and duration).
2. Browser Protection: The stopBrowsersMiddleware rejects any request where the User-Agent starts with Mozilla/ with a 403 Forbidden error.
3. Timeout: The timeoutMiddleware ensures no request hangs longer than 25 seconds.

If metrics are enabled in the metricsContainer, a /metrics endpoint is also exposed.