pino-loki

Because Pino transports run in worker threads, non-serializable options like a logFormat function cannot be passed directly via pino.transport. To use a function for logFormat, you must create a custom local module that wraps pino-loki and then point your transport to that local file.

// main.ts
import pino from 'pino'

const logger = pino({
  transport: {
    target: './my-custom-pino-loki.js',
    options: { labels: { application: 'MY-APP' } }
  },
})

// my-custom-pino-loki.js
import { pinoLoki } from 'pino-loki'

export default function customPinoLoki(options) {
  return pinoLoki({
    ...options,
    logFormat: (log) => {
      return `hello ${log.msg} ${log.lokilevel} ${log.req.id} ${log.level}`
    },
  })
}

The recommended way to use pino-loki is within a Pino worker thread. This is compatible with Pino v7.0.0 and above. You configure the transport using pino.transport and provide a LokiOptions object containing the Loki host and optional basicAuth credentials.

import pino from 'pino'
import type { LokiOptions } from 'pino-loki'

const transport = pino.transport<LokiOptions>({
  target: "pino-loki",
  options: {
    host: 'https://my-loki-instance:3100',
    basicAuth: {
      username: "username",
      password: "password",
    },
  },
});

const logger = pino(transport);
logger.error({ foo: 'bar' })

Batching is enabled by default. It accumulates logs in a buffer and sends them at regular intervals to reduce HTTP requests.

Options:

• batching: Set to false to disable batching and send logs immediately.
• batching.interval: Interval in seconds to send logs. Defaults to 5.
• batching.maxBufferSize: Maximum number of logs to keep in the buffer. When full, the oldest logs are dropped (FIFO). Defaults to 10000. Set to 0 for an unlimited buffer (not recommended due to OOM risk).

// Batching enabled with default options (interval: 5s, maxBufferSize: 10000)
pinoLoki({ host: '...' })

// Batching with custom options
pinoLoki({
  host: '...',
  batching: {
    interval: 2,        // Send logs every 2 seconds
    maxBufferSize: 5000 // Keep max 5000 logs in buffer
  }
})

// Batching disabled - logs sent immediately
pinoLoki({ host: '...', batching: false })

You can use the grafana/otel-lgtm:latest image via Docker Compose to set up a local environment containing Grafana and Loki. This setup is useful for testing pino-loki locally.

By default, the service exposes:

• Port 3000: Grafana UI
• Port 3100: Loki HTTP API (the endpoint pino-loki will send logs to)

The configuration below enables anonymous Admin access to Grafana for easier local development.

services:
  lgtm:
    image: grafana/otel-lgtm:latest
    ports:
      - '3000:3000' # Grafana
      - '3100:3100' # Loki HTTP API
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin

Out-of-order errors

Due to the asynchronous nature of Pino, you may encounter out-of-order errors in Loki. To resolve this, configure your Loki instance to allow out-of-order logs.

Dropped logs

Logs can be lost in two scenarios:

1. Network issues: If Loki is unreachable, the current batch of logs will be lost.
2. Buffer overflow: If batching is enabled and the buffer reaches maxBufferSize (default: 10000), the oldest logs are dropped (FIFO) to prevent memory exhaustion (OOM).

Batching improves performance by sending multiple logs in a single request. You can enable or disable it using the batching key in LokiOptions.

• To disable batching: Set batching: false.
• To enable/configure batching: Provide a BatchingOptions object.

BatchingOptions properties:

• interval (number): The interval at which batched logs are sent in seconds. Default: 5.
• maxBufferSize (number): Maximum number of logs to buffer before dropping the oldest ones. Set to 0 for an unlimited buffer (not recommended). Default: 10_000.

// Example: Custom batching configuration
batching: {
  interval: 10,
  maxBufferSize: 5000
}

Loki supports structured metadata. You can configure pino-loki to move extra log properties into a structured metadata block instead of just the log line.

• structuredMetaKey: Specify the key name to be used for the structured metadata. Default: 'meta'. Set to false to disable structured metadata.
• convertArrays: If set to true, arrays in log messages are converted to objects where the index is the key, which can help with metadata compatibility. Default: false.

Batching improves performance by grouping logs together before sending them to Loki. You can configure batching via the batching property in LokiOptions.

If you set batching: false, batching is disabled and logs are pushed immediately.

If you provide an object, the following keys apply:

• interval: The time interval (in seconds) between batches. Default is 5.
• maxBufferSize: The maximum number of logs to hold in the buffer before triggering a push or dropping logs. Default is 10000. If set to 0, the buffer size limit is ignored (though the transport still uses the interval).

const transport = pinoLoki({
  batching: {
    interval: 10,
    maxBufferSize: 500
  }
});

When setting up the pino-loki transport, you provide a LokiOptions object. This object controls how logs are sent to Loki, including connection details, batching behavior, and how Pino log properties are mapped to Loki labels or structured metadata.

const options: LokiOptions = {
  host: 'http://loki:3100',
  endpoint: 'loki/api/v1/push',
  labels: {
    env: 'production',
    service: 'my-app'
  },
  batching: {
    interval: 10,
    maxBufferSize: 5000
  },
  propsToLabels: ['service_name', 'version'],
  structuredMetaKey: 'meta'
};