DOCSEARCHbyAlgolia
Submit Docs

DeUnhealth

repository·main·Indexed 42 days ago

https://github.com/qdm12/deunhealth

A tool designed to safely restart unhealthy Docker containers by monitoring Docker events via a stream. It identifies containers to manage using the `deunhealth.restart.on.unhealthy=true` label and includes a specialized Client Mode for ephemeral execution, such as within Docker healthchecks.

Tokens
2.2K
Snippets
9
Records
13
Agent score
54%

What's inside DeUnhealth

  1. Install and run DeUnhealth via Docker

    main

    To run DeUnhealth, use the following Docker command. It is recommended to run it with --network none to minimize the attack surface, as it only requires access to the Docker socket to function.

    docker run -d --network none -v /var/run/docker.sock:/var/run/docker.sock qmcgaw/deunhealth

    Alternatively, you can use docker-compose.yml by running:

    docker-compose up -d

    Note: It is recommended to use specific version tags instead of :latest to avoid breaking changes before the v1.0.0 release.

  2. Configure containers to restart on unhealthy status

    main

    DeUnhealth identifies which containers it should manage by looking for specific Docker labels. To enable automatic restarts for a container when it becomes unhealthy, add the following label to that container:

    deunhealth.restart.on.unhealthy=true

  4. DeUnhealth Service Architecture

    main

    When running in its standard long-running mode, DeUnhealth operates as a collection of concurrent services managed by a service group. The core components include:

    • Unhealthy Loop: Monitors Docker for unhealthy containers.
    • Unhealthy Info Loop: Provides information/metrics regarding unhealthy containers.
    • Health Server: An HTTP server that listens on a configured address to respond to health queries (used by the Client Mode).

    These services are coordinated using a goservices.Group, ensuring that they all start together and shut down gracefully when a termination signal (like SIGINT or SIGTERM) is received.

  5. Run DeUnhealth in Client Mode

    main

    DeUnhealth can be run in a specialized 'Client Mode' designed for ephemeral execution, such as within a Docker healthcheck. In this mode, the application does not start its full service suite (loops, servers, etc.) but instead acts as a client to query the status of a long-running DeUnhealth instance via its health server.

    To trigger this mode, the application must be invoked with arguments that satisfy the health.IsClientMode(args) check. This is typically used when you want to use DeUnhealth to monitor itself or other services through a single query.

  6. Configure DeUnhealth via Docker Compose

    main

    You can run DeUnhealth using Docker Compose. The service requires access to the host's Docker socket to function and is configured to run with no network access (network_mode: "none") for security.

    Key configuration points:

    • Docker Socket: You must mount /var/run/docker.sock:/var/run/docker.sock to allow the container to interact with the Docker daemon.
    • Network: The service uses network_mode: "none".
    • Restart Policy: Set to always to ensure the service recovers from failures.
    version: "3.7"
services:
  deunhealth:
    build: .
    image: qmcgaw/deunhealth
    container_name: deunhealth
    network_mode: "none"
    environment:
      - LOG_LEVEL=info
      - HEALTH_SERVER_ADDRESS=127.0.0.1:9999
      - TZ=America/Montreal
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  7. Available development commands

    main

    Use these commands to build, test, and lint the project locally:

    # Build the binary
go build cmd/app/main.go

# Test the code
go test ./...

# Lint the code
golangci-lint run

# Build the Docker image
docker build -t qmcgaw/deunhealth .

  8. Configure DeUnhealth via environment variables and flags

    main

    You can customize DeUnhealth's behavior using the following environment variables or command-line flags:

    | Environment variable | Flag | Default | Possible values | Description |
| --- | --- | --- | --- | --- |
| `DOCKER_HOST` | `--docker-host` | Default Docker socket location | Docker host value | Docker host value such as `unix:///var/run/docker.sock` or `tcp://socket-proxy:2375` |
| `LOG_LEVEL` | `--log-level` | `info` | `debug`, `info`, `warning`, `error` | Logging level |
| `HEALTH_SERVER_ADDRESS` | `--health-server-address` | `127.0.0.1:9999` | Valid address | Internal health http server listening address. Nothing to do with restarting unhealthy containers. |
| `TZ` | N/A | `America/Montreal` | *string* | Timezone |

  9. Use the health.Client to query container health

    main

    The health.Client is used to query the internal healthcheck server of another instance of the program via HTTP.

    To use it:

    1. Initialize a client using NewClient(). This client comes with a default timeout of 5 seconds.
    2. Call Query(ctx, address) where address is the host and port of the target instance (e.g., "127.0.0.1:8080").

    If the target instance returns a non-200 status code, Query returns an error wrapping health.ErrUnhealthy along with the response body as the error message.

    package main

import (
	"context"
	"errors"
	"fmt"
	"github.com/qdm12/qdm12/deunhealth/internal/health"
)

func main() {
	client := health.NewClient()
	ctx := context.Background()
	address := "127.0.0.1:8080"

	err := client.Query(ctx, address)
	if err != nil {
		if errors.Is(err, health.ErrUnhealthy) {
			fmt.Printf("Instance is unhealthy: %v\n", err)
		} else {
			fmt.Printf("Error querying health: %v\n", err)
		}
		return
	}
	fmt.Println("Instance is healthy!")
}

  10. DeUnhealth Docker Environment Variables

    main

    When running DeUnhealth in a Docker container, you can configure its behavior using the following environment variables:

    VariableDescription
    LOG_LEVELSets the logging verbosity (e.g., info).
    HEALTH_SERVER_ADDRESSThe address and port where the health server listens (e.g., 127.0.0.1:9999).
    TZSets the timezone for the container (e.g., America/Montreal).
    environment:
  - LOG_LEVEL=info
  - HEALTH_SERVER_ADDRESS=127.0.0.1:9999
  - TZ=America/Montreal

  12. Check if running in client mode

    main

    The IsClientMode function determines if the program is being executed in healthcheck mode by inspecting the command-line arguments. It returns true if the second argument (args[1]) is exactly "healthcheck".

    // Example usage with os.Args
if health.IsClientMode(os.Args) {
    // Perform client-specific logic
}