mtail Documentation

mtail is a tool designed to extract internal monitoring metrics from application logs. These metrics can then be exported to a timeseries database or a timeseries calculator for use in alerting and dashboarding. It acts as a bridge for applications that do not natively export internal state, allowing operators to instrument them via logs without patching the application code.

Extraction is controlled by mtail programs that define patterns and actions. Metrics can be exported via:

• HTTP in JSON or Prometheus format (for scraping by collectors like Prometheus).
• Periodically sent to collectd, StatsD, or Graphite collector sockets.

mtail supports two primary classes of metrics, which determine how they are aggregated and interpreted:

• Counters: Monotonically increasing values used to accumulate events. They are resistant to sampling frequency errors and allow for calculating rates of change. They are ideal for tracking event counts and can indicate system restarts if they reset.
• Gauges: Variable values that record instantaneous measurements. They are used for recording queue lengths, resource usage, or quotas. Note that gauges are dependent on sampling rates and may miss transient spikes.

When writing tests for mtail components:

• Use the testutil module where possible.
• Avoid using time.Sleep; instead, poll for events using the PollWatched() method provided by TestServer.
• Use if testing.Short() for tests involving disk access to ensure commands like make smoke remain fast.
• Do not comment out tests; use t.Skip() with a reason to keep tests visible and compilable.

To prevent unbounded memory growth in dimensioned metrics, use del or limit.

• del [metric]: Removes a specific datum from a dimensioned metric. Can be used with after [duration] (e.g., del session_start[$session] after 24h) to expire data after a period of inactivity. Durations follow Go's time.ParseDuration.
• limit [size]: Sets a maximum size for a dimensioned metric. When the limit is exceeded, the oldest values (by timestamp) are removed.

mtail metrics can hold either integer or floating-point values. By default, metrics are treated as integers unless the compiler infers a floating-point type through type checking. Inference is based on the expressions used and heuristics applied to regular expression capturing groups.

To ensure a metric is treated as a floating-point type, use a regular expression that captures decimal points.

# This will be inferred as an integer
counter a
/(\S+)/ {
  a = $1
}

# This will be inferred as a floating point due to the decimal point in the regex
counter a
/(\d+\.\d+)/ {
  a = $1
}

To fetch the entire repository and install using make, you must enable Go Modules. This is useful if you want to ensure all dependencies are handled correctly in the source tree:

GO111MODULE=on go get -u github.com/google/mtail
cd $GOPATH/src/github.com/google/mtail
make install

To prevent high cardinality in metrics caused by unique identifiers in URLs or paths, use subst() with a regex pattern to rewrite capture groups into a generic format. This allows you to count occurrences of static routes rather than every unique URL.

hidden text route
counter http_requests_total by method, route

/(?P<method\S+) (?P<url>\S+)/ {
  # Replace digits following a slash with a literal string to group routes
  route = subst(/\d+/, "/:num", $url)
  http_requests_total[method][route]++
}

To optimize performance and avoid unnecessary work, use the getfilename() function to check the input filename. You can use the stop keyword to terminate the program for files that do not match your required pattern.

getfilename() !~ /apache.access.?log/ {
  stop
}

Decorators allow you to define repetitive functions (like timestamp extraction) that wrap other blocks.

1. Define a decorator using the def keyword and a block ending in next.
2. Apply the decorator to a block using the @decorator_name syntax.

def syslog {
  /(?P<date>\w+\s+\d+\s+\d+:\d+:\d+)/ {
    strptime($date, "Jan 02 15:04:05")
    next
  }
}

@syslog {
  /some event/ {
    variable++
  }
}