Ember Documentation

repository·main·Indexed 18 days ago

https://github.com/alexandre-daubois/ember

A real-time monitoring tool for Caddy servers providing a zero-config terminal dashboard for traffic, latency, and status codes. Ember includes deep introspection for FrankenPHP thread and worker management, a TUI for visualizing per-host HTTP metrics, and a headless daemon mode for Prometheus export. It supports remote Caddy instances via the admin API, mTLS, and an experimental plugin system for custom metric visualization.

Tokens
24.2K
Snippets
76
Records
117
Agent score
62%

What's inside Ember

  1. Analyze memory usage with Memory columns

    main

    If you are running a FrankenPHP server (version 1.12.2 or later), the By Route view can display Avg Mem and Max Mem columns. These columns aggregate per-thread PHP memory usage keyed by (method, pattern).

    Requirements and Behavior:

    • Visibility: Columns only appear if the terminal is wide enough (roughly 153+ columns) and at least one route in view has a memory sample.
    • Sampling: Values are sampled when Ember polls a busy thread, not measured per request. A request completing between polls will show .
    • Worker Mode Note: In worker mode, the sample represents the thread's memory usage. Since threads are reused, the value reflects how much memory a thread holds while serving that specific route. Compare routes over many samples rather than relying on a single peak.
    • Data Staleness: Memory samples arrive via the admin-API poll. If the poll is paused, memory figures may become stale even if Count, Avg, and Max continue to update from the live log stream.
  2. Plugin Lifecycle

    main

    The Ember plugin lifecycle follows these stages:

    1. Registration: plugin.Register() is called during init() at import time.
    2. Provisioning: Provision(ctx, cfg) is called before the TUI or daemon starts. If this returns an error, the plugin is logged as a warning and disabled, but Ember continues running.
    3. Runtime:
      • TUI Mode: Fetch is called on every tick. Update, View, HandleKey, StatusCount, and HelpBindings are called from the event loop.
      • Daemon Mode (--daemon): Only Fetch and WriteMetrics are called.
    4. Shutdown: Close() is called on plugins implementing the Closer interface, in reverse registration order.
  3. Configure multi-instance JSON output

    main

    When you provide multiple --addr flags, Ember polls every instance and emits one JSONL line per instance. Each line is prefixed with an instance field containing the alias provided in the --addr flag.

    Behavioral notes:

    • The first emission is sorted alphabetically by instance name.
    • Subsequent lines interleave based on when each instance's ticker fires. Downstream consumers should expect lines in arrival order rather than grouped by tick.
    • If only a single --addr is provided, the instance field is omitted, and the output is identical to standard single-instance JSONL.
    • With --once, exactly one line per instance is produced in alphabetical order before exiting.
    ember --json \
      --addr web1=https://web1.fr \
      --addr web2=https://web2.fr
  4. How Multi-instance Monitoring works

    main

    Ember can monitor multiple Caddy instances simultaneously and aggregate their metrics behind a single Prometheus endpoint. This is achieved using --daemon or --json modes.

    Implementation Methods

    1. Repeated Flags: Use --addr multiple times. ember --daemon --expose :9191 --addr web1=https://a --addr web2=https://b
    2. Environment Variable: Use a semicolon-separated list in EMBER_ADDR. EMBER_ADDR='web1=https://a;web2=https://b' ember --daemon --expose :9191
    3. Config File: Define multiple [[endpoints]] in a TOML file.

    Key Behaviors

    • Labels: When monitoring multiple instances, every emitted Prometheus metric (except ember_build_info) includes an ember_instance="<name>" label. If no alias is provided, the host is slugified (e.g., web1.fr becomes web1_fr).
    • TUI Limitation: The interactive TUI is designed for single-instance monitoring only. Passing multiple addresses to the TUI will result in an error. Use --daemon or --json for multi-instance needs.
    • Per-instance Overrides: You can specify per-instance TLS settings or polling intervals using comma-separated suffixes on the address URL.
      • ,ca=PATH: CA certificate
      • ,cert=PATH: Client certificate
      • ,key=PATH: Client private key
      • ,insecure: Skip TLS verification
      • ,interval=DURATION: Custom polling interval (minimum 100ms)
    # Example: Monitoring multiple instances with explicit aliases and per-instance TLS
    ember --daemon --expose :9191 \
      --addr web1=https://a,ca=/etc/ca1.pem \
      --addr web2=https://b,ca=/etc/ca2.pem
  5. Enable multi-instance support in plugins

    main

    By default, plugins are disabled in multi-instance mode (when Ember is launched with multiple --addr flags). To opt-in, implement the MultiInstancePlugin marker interface.

    Key changes for multi-instance plugins:

    • Provision: PluginConfig.Instances is populated with all monitored instances.
    • Fetch: Called once per instance, per tick. Use plugin.InstanceFromContext(ctx) to identify which instance is being polled.
    • WriteMetrics: Called once per instance. You should write metrics 'naked'; Ember automatically injects ember_instance="<name>" into every line.
    • OnMetrics: Called once per instance per tick in parallel. Note that the snapshot does not carry instance identity; if you need per-instance core metrics, do it in Fetch instead.
    type myPlugin struct{}
    
    func (p *myPlugin) EmberMultiInstance() {} // marker only, never called
    
    // In Fetch:
    func (p *myPlugin) Fetch(ctx context.Context) (any, error) {
        inst, _ := plugin.InstanceFromContext(ctx)
        return queryMyBackend(inst.Addr)
    }
    
    // In WriteMetrics:
    func (p *myPlugin) WriteMetrics(w io.Writer, data any, prefix string) {
        fmt.Fprintln(w, "# HELP my_plugin_requests_total ...")
        fmt.Fprintln(w, "# TYPE my_plugin_requests_total counter")
        fmt.Fprintf(w, "my_plugin_requests_total %d\n", count)
    }
  6. Scrape multiple Caddy instances with one Ember container

    main

    Ember can act as an aggregator for multiple Caddy instances. You can define multiple targets using the EMBER_ADDR environment variable with a semicolon-separated list of name=address pairs.

    When using this mode, every emitted Prometheus metric (except ember_build_info) will include an ember_instance label corresponding to the name provided (e.g., ember_instance="blue").

    For large fleets, instead of using EMBER_ADDR, use a configuration file and point the EMBER_CONFIG environment variable to its path.

    services:
      caddy-blue:
        image: caddy:latest
        volumes:
          - ./Caddyfile:/etc/caddy/Caddyfile
    
      caddy-green:
        image: caddy:latest
        volumes:
          - ./Caddyfile:/etc/caddy/Caddyfile
    
      ember:
        image: alexandredaubois/ember
        environment:
          EMBER_ADDR: blue=http://caddy-blue:2019;green=http://caddy-green:2019
        ports:
          - "9191:9191"
        depends_on:
          - caddy-blue
          - caddy-green
  7. Separate the Renderer for complex state

    main

    For plugins with complex state, you can follow the Elm architecture by returning a separate Renderer from the Update method.

    When using this pattern:

    1. Ember calls View, HandleKey, etc., on the plugin struct itself until the first Update returns a new Renderer.
    2. Once a Renderer is returned, all subsequent calls to Update, View, HandleKey, etc., are directed to that returned Renderer.

    This allows the main plugin struct to handle the initial "no data yet" state in its own View method while the Renderer handles the actual data visualization.

    func (p *statsPlugin) Update(data any, _, _ int) plugin.Renderer {
    	return &statsRenderer{count: data.(int64), ts: time.Now()}
    }
    
    // Initial View, before the first Fetch completes
    func (p *statsPlugin) View(_, _ int) string { return " Waiting for data..." }
    
    type statsRenderer struct {
    	count int64
    	ts    time.Time
    }
    
    func (r *statsRenderer) Update(data any, _, _ int) plugin.Renderer {
    	return &statsRenderer{count: data.(int64), ts: time.Now()}
    }
    
    func (r *statsRenderer) View(_, _ int) string {
    	return fmt.Sprintf("\n  Tick counter:  %d\n  Last update:   %s\n",
    		r.count, r.ts.Format("15:04:05"))
    }
    
    func (r *statsRenderer) HandleKey(msg tea.KeyMsg) bool {
    	return msg.String() == "x"
    }
    
    func (r *statsRenderer) StatusCount() string {
    	return fmt.Sprintf("%d ticks", r.count)
    }
    
    func (r *statsRenderer) HelpBindings() []plugin.HelpBinding {
    	return []plugin.HelpBinding{{Key: "x", Desc: "Example action"}}
    }
  8. Use Multi-instance monitoring to aggregate Caddy metrics

    main

    Ember can scrape multiple Caddy instances simultaneously and aggregate their metrics into a single /metrics endpoint. This is useful for monitoring a fleet of small Caddy services through a single Prometheus target.

    Every emitted metric (except ember_build_info) includes an ember_instance="<name>" label, allowing you to split or aggregate data per instance using PromQL.

    Note: If you are running Ember as a sidecar next to a single Caddy instance, you do not need multi-instance mode; use a single --addr instead.

    ember --daemon --expose :9191 --addr blue=http://10.0.0.10:2019 --addr green=http://10.0.0.11:2019
  9. Plugin Name Rules and Validation

    main

    When developing a plugin for Ember, the name you provide during registration must follow specific rules. If these rules are violated, the Register() function will panic at startup.

    Naming Requirements:

    • Must not be empty.
    • Must not contain whitespace (spaces, tabs, or newlines).
    • Must not contain underscores (use hyphens instead).
    • Must be unique across all registered plugins.
    • Collision Rule: Names must be distinguishable even after hyphens are removed. For example, my-plugin and myplugin both map to the same environment variable prefix (EMBER_PLUGIN_MYPLUGIN_) and therefore cannot coexist.
  10. Manage log scroll modes (Following vs Frozen)

    main

    The log table operates in two modes:

    1. Following (Default): Pins the newest entry at the top and redraws as new lines arrive.
    2. Frozen: Stops the stream so you can inspect specific lines. You can scroll through the full buffer available at the time of freezing. A ● PAUSED indicator appears in the header, showing how many new lines have been captured in the background.

    Switching Modes:

    • Implicitly: Scrolling (, , PgUp, PgDn, End) automatically enters Frozen mode.
    • Explicitly: Press p to toggle between Frozen and Following.
    • Resume: Press f or Home to resume live following.
  11. Use the By Route view for request aggregation

    main

    Selecting By Route in the sidepanel provides an aggregated view of every request seen during the session. Unlike the standard log table, these counts are independent of the 10,000-entry access ring buffer and continue to climb.

    Each row represents a (host, method, pattern) bucket. For example, GET /users/:id on api.localhost is a distinct row from GET /users/:id on app.localhost.

    Key Features:

    • Root View: The host is shown as a soft prefix in the Pattern column (e.g., api.localhost /users/:id).
    • Drill-down: Selecting a specific host filters the table to that host and removes the host prefix from the Pattern column.
    • Sorting: Press s or S to cycle through sort fields: Count -> Pattern -> Avg -> Max -> Avg Mem -> Max Mem.