DOCSEARCHbyAlgolia
Submit Docs

Nerdlog

repository·master·Indexed 23 days ago

https://github.com/dimonomid/nerdlog

A fast, remote-first TUI log viewer that queries logs from multiple remote hosts simultaneously via SSH without a central server. It features an interactive timeline histogram for visual log density analysis, supports logstreams from consecutive files or journalctl, and uses Awk patterns for filtering. It includes Vim-like keybindings, a command-line interface for session management, and field expressions for custom log output formatting.

Tokens
14.2K
Snippets
21
Records
101
Agent score
82%

What's inside nerdlog

  1. Overview of Nerdlog

    master

    Nerdlog is a fast, remote-first, multi-host TUI (Terminal User Interface) log viewer. It is designed to query logs from multiple remote machines simultaneously without requiring a central server.

    Key features include:

    • Multi-host support: Query logs from several remote machines at once.
    • Timeline Histogram: Provides visual insight into log density over time.
    • Efficient Filtering: Filter logs by time range and patterns.
    • No Central Server: Operates in a decentralized, remote-first manner.
    • Low Overhead: Inspired by Graylog/Kibana but optimized for speed and minimal setup.

  2. Understand the Nerdlog UI and Status Line

    master

    The Nerdlog TUI provides real-time feedback via the status line and a timeline histogram.

    Status Line Indicators

    Left side (Connection Status):

    • Green: Number of logstreams fully connected and idle.
    • Orange: Number of logstreams fully connected and executing a query.
    • Red: Number of logstreams attempting to connect.

    Right side (Log Statistics): Format: [Cursor Position] / [Loaded Lines] / [Total Matched Messages]

    • Total Matched Messages: The total number of logs matching the current query and time range (including those in the histogram).
    • Loaded Lines: The number of log lines currently loaded in the app.
    • Cursor Position: The current position within the loaded logs.

    Navigation and Interaction

    • Timeline Histogram: Use arrow keys, PgUp/PgDown, Home/End, or Enter to visually select and apply a time range.
    • Command Line: Hit : to enter Vim-like command mode.
    • Logs Table: The latest messages appear at the bottom of the table.

  3. Use `custom` transport for advanced authentication

    master

    The custom:<arbitrary command> transport allows you to run arbitrary commands to connect to remote hosts, which is useful for tools like Teleport.

    Commands support shell-like variable expansion (e.g., ${MYVAR}, ${MYVAR:-default}, ${MYVAR:+alternative}), but do not support logic constructs (if, while) or command substitution ($(...)). To use a full shell, invoke it explicitly: custom:/bin/sh -c "<script>".

    Available Variables

    Nerdlog provides the following variables for expansion:

    • NLHOST: The hostname (from logstreams input or config).
    • NLPORT: The port (if specified in input or config).
    • NLUSER: The username (if specified in input or config).

    All standard environment variables are also available.

  4. Understand the design choice of log files over journalctl

    master

    Nerdlog defaults to reading plain log files rather than using journalctl. This design choice is based on several factors:

    • Performance: journalctl is significantly slower than reading plain log files.
    • Reliability: journalctl can occasionally miss logs.
    • Simplicity: Using journalctl adds a layer of complexity and potential bugs (e.g., systemd-specific issues) that plain log files avoid.

  5. How Nerdlog manages SSH connections and agents

    master

    Nerdlog uses a distributed agent model to query logs across multiple hosts. When you submit a query for one or more logstreams:

    1. SSH Connections: Nerdlog initiates a separate SSH connection for every logstream (except for localhost). Note that even if multiple logstreams reside on the same host, separate SSH connections are currently established for each.
    2. Agent Deployment: For every logstream, Nerdlog uploads a bash agent script to /tmp on the target host.
    3. Host Initialization: The agent is immediately invoked to detect host details, including the timezone, awk version, and sample log lines to determine the timestamp format.
    4. Query Execution: Once initialized, the agent executes the requested query and streams results to stdout and stderr, which Nerdlog reads and displays in the UI.

  6. Understand nerdlog test types

    master

    Nerdlog uses four distinct types of tests to ensure quality across different layers of the application:

    1. Regular Go unit tests: Standard Go tests covering small, specific functional units.
    2. Agent script tests: Tests the nerdlog_agent.sh script (the backend responsible for log filtering and processing). These rely on environment tools like bash, gawk, tail, and head. They are designed to work on Linux, FreeBSD, and MacOS. Test cases are defined in YAML files under ../core/core_testdata/test_cases_agent.
    3. Core tests: Covers the ../core package, including LStreamClient, LStreamsManager, and helpers. These tests use YAML scenarios in ../core/core_testdata/test_cases_core. They can test different transport layers (e.g., ShellTransportSSH) by setting the NERDLOG_CORE_TEST_HOSTNAME environment variable.
    4. End-to-end (E2E) tests: Smoke tests that run the final nerdlog binary and capture TUI screen snapshots using tmux. Test cases are located in ../cmd/nerdlog/e2e_testdata.

  7. How Nerdlog merges and parses query results

    master

    After the remote agents return data, Nerdlog performs the following on the client side:

    1. Aggregation: It waits for all agents across all logstreams to return their timeline histogram data and raw log lines, then merges them.
    2. Parsing: Nerdlog parses the raw messages into structured columns. For example, syslog messages are parsed into hostname, program, and pid fields.
    3. Metadata: Every log message is enriched with an lstream field, which identifies the source logstream.
    4. Rendering: The final structured data is rendered in the UI table.

  8. How the Nerdlog index file optimizes queries

    master

    To avoid scanning entire large log files for every query, the agent maintains an index file located at /tmp/nerdlog_agent_index_......

    Key Characteristics:

    • Mapping: It maps timestamps (with a resolution of 1 minute) to the specific line number and byte offset in the log file.
    • Granularity: Because the index resolution is 1 minute, you cannot query time ranges more granular than one minute.
    • Indexing Logic:
      • If the requested timestamp exists in the index, the agent immediately knows which part of the file to cut.
      • If the timestamp is newer than the last indexed entry, the agent performs an "index up" operation to add new lines.
      • If log files are rotated, the index files are invalidated and regenerated.
    • Performance: Indexing a 2GB log file takes approximately 10 seconds. Once indexed, time-range-based queries are highly efficient.

  9. Use awk syntax for log filtering patterns

    master

    Nerdlog uses awk syntax for its filtering patterns. This allows you to write complex queries using boolean operators (e.g., &&, ||, !) to combine multiple patterns. This approach is used because it is highly efficient, allowing filtering, timeline histogram generation, and line tracking to be performed in a single pass via an internal awk script.

    Example of a complex query pattern: /foo/ && !/bar/ && (/this/ || /that/)

    /foo/ && !/bar/ && (/this/ || /that/)

  10. Understand the performance impact of Nerdlog on hosts

    master

    Unlike centralized logging systems (e.g., Graylog or Kibana), Nerdlog fetches logs directly from the generating hosts. This means Nerdlog consumes CPU and IO on the target hosts to perform filtering and analysis.

    Risks:

    • Resource Contention: If a host is already overloaded during an emergency, running Nerdlog may exacerbate the resource pressure.
    • Unresponsiveness: If a host becomes unresponsive, Nerdlog will be unable to retrieve logs from it.

    Mitigation: Sync logs to a separate logging server (using tools like rsyslog) to avoid direct resource consumption on production hosts.

  11. Configure Logstreams in Nerdlog

    master

    A logstream is a contiguous stream of log messages from a specific server (via SSH) or the local machine. Nerdlog supports two types:

    • Consecutive log files (e.g., /var/log/syslog and /var/log/syslog.1).
    • Logs returned from journalctl.

    Logstream Syntax

    Default Behavior

    By default, Nerdlog searches for logs in this order:

    1. /var/log/messages (and .1)
    2. /var/log/syslog (and .1)
    3. journalctl (as a last resort)

    Explicit Connection Strings

    To specify a host, port, and log file, use the format user@host:port:logfile:

    • Standard SSH: myuser@myserver.com (uses SSH config for port/user).
    • Custom Port and File: myuser@myserver.com:1234:/some/other/logfile.
    • Explicit journalctl: myuser@myserver.com:1234:journalctl.

    Multiple Logstreams

    Separate multiple logstreams with commas: myuser@myserver.com, myuser@myserver.com:1234:/some/other/logfile

    Nerdlog also supports globs if defined in your ~/.ssh/config (e.g., myhost-*).

    Persistent Configuration

    For non-default log files, use the configuration file ~/.config/nerdlog/logstreams.yaml:

    log_streams:
  myhost-01:
    hostname: actualhost1.com
    port: 1234
    user: myuser
    log_files:
      - /some/custom/logfile
  myhost-02:
    hostname: actualhost2.com
    port: 7890
    user: myuser
    log_files:
      - /some/custom/logfile
    log_streams:
  myhost-01:
    hostname: actualhost1.com
    port: 1234
    user: myuser
    log_files:
      - /some/custom/logfile

  12. Compare Nerdlog with lnav

    master

    Nerdlog and lnav serve different primary use cases:

    FeaturelnavNerdlog
    Primary FocusLocal log filesRemote logs
    StrengthsAutomatically discovering and managing local log directoriesEfficiency with large files and many remote nodes (e.g., 20+ nodes with 500MB files each)
    Remote LogsSupported as an extra feature, but not the primary focusBuilt from the ground up for remote log efficiency