DOCSEARCHbyAlgolia
Submit Docs

observer_cli

repository·main·Indexed 23 days ago

https://github.com/zhongwencool/observer_cli

A production-ready diagnostic tool for Erlang and Elixir systems providing a CLI for automation and AI agents, and a TUI for interactive live exploration of BEAM nodes. It supports Erlang/OTP 26–29 and allows users to connect to remote nodes to perform diagnostics, snapshots, and tracing while managing the observer effect through bounded work and resource limits.

Tokens
15.9K
Snippets
25
Records
68
Agent score
81%

What's inside observer_cli

  1. Choose the appropriate command invasiveness level

    main

    Commands vary in their impact on the target system. Choose the least invasive option that satisfies your investigation:

    • Point-in-time facts: snapshot, memory, and distribution read bounded runtime facts. These are relatively low impact but still create a distribution peer and target worker.
    • Inventories and sampling: processes, applications, ets, mnesia, ports, and sockets enumerate resources. Using duration or observation modes repeats work and increases impact. If scan admission refuses a command, narrow the scope instead of retrying.
    • State and supervision (High Risk): otp-state and supervision-tree are high-risk. otp-state copies process state before reducing it to bounded shapes and has a 5-second sys:get_state/2 timeout. supervision-tree can be linear and blocking.
    • Retained logs: The logs command reads a configured-path file. It is a bounded file read and does not flush the Logger. It is treated as sensitive, untrusted content, and identifier redaction is unavailable.
    • Tracing: trace call modifies node-global static tracing. It requires an exact MFA, a target-local PID, a bounded duration/event rate, and the --replace-existing-trace flag.

  2. Understand bounded work and the observer effect

    main

    Observer CLI commands (probes) execute within a monitored target worker. To prevent runaway processes, the dispatcher applies several constraints:

    • Target-side deadlines (shorter than the controller deadline)
    • Worker heap limits
    • Response size and depth limits
    • Scan admission budgets
    • Response normalization

    The Observer Effect: The act of collecting data (processes, memory, scheduler measurements) adds its own overhead (processes, memory, reductions, ports, and activity) to the target VM. Reports may include contaminated-count fields to acknowledge this effect. Evidence is captured from a bounded window rather than an atomic snapshot of an untouched VM.

  3. Understand limits for Messages, Dictionary, and State views

    main

    When viewing Messages, Dictionary, or State, the TUI uses a monitored worker on the observed node with specific safety boundaries to prevent overwhelming the system:

    • Message Queue Limit: The Messages view only fetches if the queue length is $\le$ 10,000. Queues larger than this return too_large.
    • Worker Constraints: The worker has a 5-second deadline and a 512 * 1024-word heap cap.
    • Term Refusal: Terms are refused if they exceed 64 KiB of external representation or have a structural depth $> 32$.
    • Output Refusal: Formatter output is refused if the final rendered detail (including view prefixes) exceeds 65,536 characters or 64 KiB of UTF-8. In these cases, the UI displays too_large.

  4. Understand TUI data values and terminology

    main

    When reading values in the TUI, note the following conventions:

    • Units: Byte-valued exceptions use B, KiB, MiB, or GiB via the byte formatter.
    • Total vs. Delta:
      • A total is cumulative for the VM or resource.
      • An increment or delta is the current sample minus the preceding sample (it is not a per-second rate).
    • Recon Metrics:
      • A count is a point-in-time ranking.
      • A window is the difference between two samples separated by the configured interval.
    • Special States:
      • dead: The resource disappeared.
      • unavailable: A required capability is missing.
      • undefined: An unsupported or absent property.
      • -: Represents an empty state or an unconnected endpoint.

  5. Code delivery: CLI vs TUI

    main

    Observer CLI handles code execution differently depending on the interface used:

    CLI (Command Line Interface)

    • Mechanism: Uses the target's already installed diagnostics bundle.
    • Requirement: The target must have a compatible bundle. The CLI calls observer_cli_snapshot:capabilities/0 to perform a handshake.
    • Handshake Tokens:
      • bundle_version (e.g., 2.0.0)
      • protocol_version (e.g., 1)
    • Constraint: The CLI will not inject or replace missing modules on the target. If the bundle is missing or incompatible, diagnostic commands will reject the target with a capability error.

    TUI (Terminal User Interface)

    • Mechanism: Can load code remotely using recon.
    • Auto-load: If core modules are missing or incompatible, the TUI sends the controller's compiled BEAM files to the target.
    • Constraint: To use auto-load, the controller must be built on the same OTP major version as the target. Cross-major bytecode loading is unsupported.
    • Plugins: Automatic loading only applies to core modules; plugin applications must already be packaged on the target.

  6. Choose between CLI and TUI interfaces

    main

    Observer CLI provides two distinct interfaces depending on your use case:

    1. CLI (Command Line Interface): Recommended for production runbooks, automation, incident capture, and AI-agent workflows. It provides stable text, Erlang-term, or JSON output.
    2. TUI (Terminal User Interface): Designed for live interactive exploration, ranking changes, and detailed drill-down.

    Note: The old positional command form observer_cli NODE [COOKIE REFRESH_MS] is no longer supported. Use the explicit tui command for interactive sessions.

    | Interface | Choose it for | Start it with |
| --- | --- | --- |
| **CLI — recommended** | Production runbooks, automation, incident capture, and AI-agent workflows | `observer_cli diagnose` |
| TUI | Live exploration, ranking changes, and detail drill-down | `observer_cli tui app@host` |

  7. Understand the structured response schema

    main

    All structured responses (term and JSON) follow a strict schema with exactly six top-level keys. This allows for predictable parsing by automation tools.

    Top-level keys:

    • schema: The protocol identity ("observer_cli.cli/v1").
    • command: The identity of the command executed.
    • outcome: The result status: "complete", "partial", or "error".
    • data: The command-specific payload (shape varies by command).
    • meta: Contextual metadata containing target and capture info.
    • issues: A list of non-probe problems found during execution.

    Metadata and Probes:

    • meta.capture includes timestamps, duration, and probes.
    • Each probe records its id, required status, status, reason_code, duration, samples, and coverage.
    • Note: A probe failure does not necessarily make a command partial if the probe was optional.

    Issues: Every issue contains severity, class, reason_code, and message.

    The full JSON Schema 2020-12 definition is available in the repository at priv/schema/observer_cli.cli.v1.schema.json.

    #{
  <<"schema">> => <<"observer_cli.cli/v1">>,
  <<"command">> => CommandOrNull,
  <<"outcome">> => <<"complete">> | <<"partial">> | <<"error">>,
  <<"data">> => CommandDataOrNull,
  <<"meta">> => #{
    <<"target">> => TargetOrNull,
    <<"capture">> => CaptureOrNull
  },
  <<"issues">> => Issues
}

  8. Understand data collection boundaries

    main

    By default, snapshot and diagnose collect metadata and bounded metrics. They do not read:

    • Mailbox contents
    • Process dictionaries
    • ETS or Mnesia contents
    • Application environment values
    • Cookies
    • Trace arguments or returns
    • Arbitrary operator expressions

    Exceptions to the default boundary:

    • process command: Includes a bounded, normalized stacktrace.
    • otp-state: Deliberately acquires full state but returns only bounded, behavior-aware shapes.
    • logs: An explicit exception for sensitive content. It can return up to 64 KiB from a single admitted configured file path and is never included in default snapshot, diagnose, or TUI collections.
    • TUI Subviews: The TUI provides explicit subviews for messages, dictionary, stack, and state. These are only read and rendered when the specific subview is opened.

  9. How the Controller and Target interact

    main

    Observer CLI operates using two primary entities:

    • Controller: The machine running the observer_cli escript. It acts as an ephemeral command executor.
    • Target: The remote distributed BEAM node being inspected.

    Lifecycle of a Remote Command

    Commands are not persistent daemons. Each command follows a strict lifecycle:

    1. The controller resolves the target, name mode, and cookie source.
    2. A hidden, outbound-only distribution controller starts (it does not listen on a port).
    3. The controller connects to the target and reads its diagnostic capabilities.
    4. A single bounded request is dispatched to the target.
    5. The target returns a normalized response.
    6. The controller validates the response and performs cleanup.
    7. The controller encodes the validated response and writes it to the output.

    Important: You must invoke the controller from a non-distributed Erlang node (nonode@nohost). Invoking it from an already distributed node is refused to ensure lifecycle guarantees.

    sequenceDiagram
    participant CLI as observer_cli command
    participant C as temporary controller
    participant T as target node
    CLI->>C: start hidden, outbound-only distribution
    C->>T: connect and read capabilities
    C->>T: dispatch bounded request
    T-->>C: normalized response and cleanup status
    C->>C: validate response
    CLI->>C: stop and confirm cleanup
    CLI->>CLI: encode validated envelope and write output

  10. Analyze Scheduler utilization

    main

    The TUI displays scheduler rows showing the scheduler ID and the active-time percentage between two statistics(scheduler_wall_time) samples.

    • Coverage: Covers normal and dirty CPU scheduler IDs (excludes dirty IO schedulers).
    • Threshold: Rows with utilization at or above 80% are highlighted in red.
    • Implementation: Uses recon_lib:scheduler_usage_diff/2 to calculate usage.

  11. Install observer_cli on the target (Erlang or Elixir)

    main

    To use observer_cli for diagnostics, the target release must include both observer_cli and recon. A 2.0.0 controller requires target bundle 2.0.0 and protocol 1. The target must be running as a distributed Erlang node.

    ### Erlang
Add to `rebar.config`:
```erlang
{deps, [
    {observer_cli, "2.0.0"}
]}.

    Run:

    rebar3 compile

    Elixir

    Add to mix.exs:

    defp deps do
  [
    {:observer_cli, "2.0.0"}
  ]
end

    Run:

    mix deps.get
mix compile

  12. Navigate the Process State Pager

    main

    When viewing detailed process data (like Messages or Dictionary) that requires paging, use the following inputs:

    InputAction
    F or jNext page
    B or kPrevious page
    q or QQuit the TUI from the process-state pager

    Note: If B is reserved for returning to a plugin, use k for the previous pager page.