Timberjack

repository·main·Indexed 41 days ago

https://github.com/deruina/timberjack

A Go package for writing logs to rolling files with enhanced rotation capabilities. It provides size-based, time-based, and scheduled rotation (via RotateAt and RotateAtMinutes), and supports compression using gzip or zstd. Timberjack implements io.Writer, making it compatible with the standard library log package and Uber Zap's zapcore.WriteSyncer.

Tokens
5.9K
Snippets
13
Records
23
Agent score
51%

What's inside Timberjack

  1. How log rotation works in Timberjack

    main

    Timberjack supports several rotation triggers. Depending on your configuration, rotation can be triggered by file size, time intervals, specific clock times, or manual calls.

    Rotation Triggers

    1. Size-Based: Triggered when a write operation would cause the file to exceed MaxSize. The backup filename includes -size.
    2. Time-Based (Interval): Triggered on the next write after RotationInterval has elapsed since the last rotation. The backup filename includes -time.
    3. Scheduled (Clock-Aligned): Triggered by a background goroutine at specific minutes (RotateAtMinutes) or specific times (RotateAt). These use -time as the reason.
    4. Manual:
      • Logger.Rotate(): Forces immediate rotation.
      • Logger.RotateWithReason(reason): Forces rotation and tags the file with a sanitized custom reason.

    Filename Patterns

    By default, rotated files follow the pattern: <name>-<timestamp>-<reason>.log.

    If you set AppendTimeAfterExt: true, the pattern changes to: <name>.log-<timestamp>-<reason> (useful for shell TAB completion).

    Example of custom reason: If you call logger.RotateWithReason("reload-now v2"), the file might be named foo-2025-05-01T10-30-00.000-reload-now-v2.log.

    // Manual rotation with a custom reason
    _ = logger.RotateWithReason("reload-now v2")
  2. Configure log compression

    main

    You can specify the compression algorithm for rotated logs using the Compression key. Compression occurs in a background goroutine after rotation.

    Supported values:

    • "none"
    • "gzip"
    • "zstd"

    Important Notes:

    • The Compression key takes precedence. If it is set, the legacy Compress: true setting is ignored.
    • Output files will use .gz or .zst extensions based on the chosen algorithm.
    • The legacy Compress field is deprecated and will be removed in v2.
  3. How Timberjack sanitizes rotation reasons

    main

    When a log rotates, a 'reason' is appended to the filename (e.g., size or time). Timberjack sanitizes this string to ensure it is safe for filesystems and doesn't break the timestamp parsing logic.

    Sanitization Rules:

    • Converts to lowercase.
    • Only allows alphanumeric characters [a-z0-9] and underscores _.
    • Replaces all other characters (including hyphens - and whitespace) with a single underscore _.
    • Collapses consecutive underscores.
    • Trims leading/trailing underscores.
    • Maximum length is 32 characters.

    Critical Note: Hyphens - are intentionally replaced by underscores because Timberjack uses the last hyphen in the filename to separate the timestamp from the reason. If a hyphen were allowed in the reason, it could corrupt the timestamp parsing and prevent the logger from correctly identifying and cleaning up old files.

  4. How Timberjack rotation works

    main

    Timberjack rotates the active log file based on several triggers. When a rotation occurs, the current file is closed and renamed using the pattern: name-timestamp-<reason>.<ext>.

    Rotation Triggers:

    1. Size-based: The file grows beyond MaxSize.
    2. Interval-based: The RotationInterval elapses since the last rotation.
    3. Clock-based: A scheduled time is reached via RotateAt or RotateAtMinutes.
    4. Manual: Triggered explicitly by calling Rotate() or RotateWithReason(reason).

    Backup Filename Formats:

    By default, backups follow: <name>-<timestamp>-<reason>.<ext>.

    • timestamp format: 2006-01-02T15-04-05.000 (unless BackupTimeFormat is customized).
    • reason: "size", "time", or a custom tag provided via RotateWithReason.

    If AppendTimeAfterExt is set to true, the format changes to: <name>.<ext>-<timestamp>-<reason>.

  5. Configure the timberjack.Logger

    main

    The Logger struct is used to define how logs are written and rotated.

    Warning: Timberjack assumes one process writes to a given file. Reusing the same configuration from multiple processes on the same machine may lead to unexpected behavior.

    type Logger struct {
        Filename          string        // File to write logs to
        MaxSize           int           // Max size (MB) before rotation (default: 100)
        MaxAge            int           // Max age (days) to retain old logs
        MaxBackups        int           // Max number of backups to keep
        LocalTime         bool          // Use local time in rotated filenames
    
        // Compression controls post-rotation compression:
        //   "none" | "gzip" | "zstd"
        // Unknown/empty default to "none", unless the legacy Compress is true (see below).
        Compression       string
    
        // Deprecated: prefer Compression.
        // If Compression is empty and Compress is true = gzip compression.
        // Back-compat shim for old configs; will be removed in v2.
        Compress          bool
    
    
        RotationInterval  time.Duration // Rotate after this duration (if > 0)
        RotateAtMinutes   []int         // Specific minutes within an hour (0–59) to trigger rotation
        RotateAt          []string      // Specific daily times (HH:MM, 24-hour) to trigger rotation
        BackupTimeFormat  string        // Optional. If unset or invalid, defaults to 2006-01-02T15-04-05.000 (with fallback warning)
        AppendTimeAfterExt    bool      // if true, name backups like foo.log-<timestamp>-<reason> defaults to foo-<timestamp>-<reason>.log
        FileMode          os.FileMode   // Use custom permissions for newly created files. If zero (unset or 0), defaults to 640.
    }
  6. Manage old log files with MaxBackups and MaxAge

    main

    Timberjack automatically cleans up old log files whenever a new log file is created. Cleanup is governed by two parameters:

    1. MaxBackups: Retains the most recent files (based on timestamp) up to the specified count. If set to 0, all files are kept.
    2. MaxAge: Deletes any files with a timestamp older than the specified number of days, regardless of how many backups exist.

    Note: The timestamp used for these calculations is the rotation time encoded in the filename, not the file's last modification time.

  7. Understand how Timberjack manages log rotation and cleanup

    main

    Timberjack uses a background goroutine (the mill process) to handle post-rotation tasks. When a log file rotates, the mill process is signaled to perform compression and remove stale log files based on your retention settings. This ensures that the heavy lifting of compression and file deletion doesn't block your application's main logging path.

    Key behaviors:

    • Compression: Files are compressed using gzip or zstd (depending on your configuration) and written to a .tmp file first to ensure atomic publishing.
    • Cleanup: The logger identifies old log files by parsing the timestamp embedded in their filenames. It supports both uncompressed and compressed (.gz, .zst) backup files.
    • Filename Pattern: Backup files follow a specific pattern: prefix-timestamp-reason.ext or prefix.ext-timestamp-reason.ext. The reason is a sanitized string used to describe why the rotation occurred (e.g., size or time).
  8. Configure log cleanup (MaxBackups and MaxAge)

    main

    Timberjack automatically manages old log files during the creation of every new log file. You can control this via:

    • MaxBackups: The maximum number of backup files to keep. Timberjack deletes the oldest backups when this limit is exceeded.
    • MaxAge: The maximum age of a backup file in days. Files older than this are deleted.
    • Automatic Compression: If compression is enabled, Timberjack will also compress any uncompressed backups during the cleanup process.
  9. Troubleshooting rotation and logger lifecycle

    main

    Common Issues and Warnings

    • Logger Must Be Closed: Always call logger.Close() when finished. This shuts down background goroutines for scheduled rotation and cleanup. Failing to do so causes memory leaks and orphaned processes.
    • Invalid BackupTimeFormat: This must follow standard Go time layout rules (e.g., 2006-01-02). If invalid, Timberjack logs a warning to os.Stderr and falls back to 2006-01-02T15-04-05.000.
    • Invalid Schedule Values: If RotateAtMinutes contains values outside 0-59 or RotateAt contains invalid HH:MM strings, they are ignored with a warning to os.Stderr.
    • Size-Based Rotation is Always Active: Even if you only configure time-based rotation, MaxSize is always enforced. A single write larger than MaxSize will return an error.
    • Interaction between Interval and Scheduled Rotation: If both RotationInterval and RotateAt/RotateAtMinutes are set, the scheduled rotation will reset the lastRotationTime, which may delay or prevent the RotationInterval from triggering as expected.
  10. Prevent compressed backup clobbering during rotation failures

    main

    Known Issue: Re-compression clobbering

    In certain failure scenarios (e.g., when resolvedRemove fails with EACCES after a successful rename), the library may attempt to re-compress a file that has already been successfully compressed and moved to its destination.

    Impact: This can result in an existing, valid compressed backup being overwritten by a fresh compression attempt of the original source file.

    Status: This is a known issue (OPEN-2) being addressed in future releases.