rotating-file-stream

repository·master·Indexed 43 days ago

https://github.com/iccicci/rotating-file-stream

A Node.js utility that provides a stream.Writable to a file that automatically rotates based on size, time intervals, or customizable rules. It serves as a logrotate alternative, supporting internal or external compression, immutable file names for log shippers, and rotation limits via maxFiles and maxSize. Version 3.2.10 requires Node.js v14 or higher.

Tokens
5K
Snippets
12
Records
24
Agent score
56%

What's inside rotating-file-stream

  1. How to use a file name generator function

    master

    If a simple string is insufficient for your naming requirements, you can provide a function as the filename argument. This function is called to determine the name of the rotated files.

    Function Signature: filename(time[, index])

    • time (Date):
      • By default: the time when the rotation job started.
      • If both options.interval and options.intervalBoundary are enabled: the start time of the rotation period.
      • If you return null, the generator indicates that the not-rotated file name should be used.
    • index (number): The progressive index of rotation by size within the same rotation period.

    Important: If both options.interval and options.size are used, the returned file name must be a function of both time and index.

    If options.rotate (classical UNIX logrotate behavior) is enabled, the file name is only a function of index.

    const pad = num => (num > 9 ? "" : "0") + num;
    const generator = (time, index) => {
      if (!time) return "file.log";
    
      var month = time.getFullYear() + "" + pad(time.getMonth() + 1);
      var day = pad(time.getDate());
      var hour = pad(time.getHours());
      var minute = pad(time.getMinutes());
    
      return `${month}/${month}${day}-${hour}${minute}-${index}-file.log`;
    };
    
    const rfs = require("rotating-file-stream");
    const stream = rfs.createStream(generator, {
      size: "10M",
      interval: "30m"
    });
  2. How rotating-file-stream handles file rotation and limits

    master

    The package is designed to avoid overwriting existing log files.

    Rotation Logic

    • Initial Rotation: If the active log file already exists and exceeds the configured rotation size at stream creation, an initial rotation is triggered.
    • Collision Avoidance: During rotation, the package checks if the destination rotated file name already exists. If it does, it generates a new name and repeats the check until a unique name is found.
    • Limit Enforcement: If options.maxFiles or options.maxSize are configured, limits are checked at the end of a rotation job. This means limits might be temporarily exceeded while a rotation job is actively running.
    • Cleanup Order: The package first checks options.maxFiles and removes files if necessary, then checks options.maxSize and removes additional files to satisfy the size constraint.
    • History File: When maxFiles or maxSize are enabled for the first time, an history file may be created containing one rotated filename per line.

    Error Handling

    Once an error event is emitted, the stream is automatically closed and no further operations will occur.

  3. Upgrade from v2 to v3

    master

    Version 3 was refactored to use async/await and added TypeScript types.

    Breaking Changes:

    • By default, the .gz extension is added to rotated compressed files.
    • The execution of the external compression command has changed slightly.

    Migration Guide:

    • If you use a file name generator or do not use options.compress: No changes required.
    • If you use a file name AND options.compress: Use options.omitExtension or verify how your rotated files are being treated.
  4. Upgrade from v1 to v2

    master

    Version 2 introduced significant interface changes.

    Breaking Changes:

    • The package no longer has a default export. You must use the createStream exported function instead of calling the RotatingFileStream constructor directly.
    • rotationTime was removed in favor of intervalBoundary.

    Migration Guide:

    • If using options.rotation: No changes required.
    • If NOT using options.rotation:
      • If NOT using options.interval: No changes required.
      • If using options.interval:
        • If you were using options.rotationTime: Remove it.
        • If you were NOT using options.rotationTime: Use options.intervalBoundary instead.
  5. Configure file compression

    master

    The compress option specifies the compression method for rotated files.

    • Internal Gzip: Set compress: 'gzip' to use the Node.js internal compression library.
    • External Compression: Set compress: true to use default external compression, or provide a function. The function must accept (source, dest) as arguments and return a shell command string that executes the compression.

    Important Notes:

    • This option is ignored if options.immutable is used.
    • The shell command should not remove the source file; the package handles source removal upon successful rotation.
    • In v3, .gz is added to compressed files by default. Use omitExtension: true to prevent this.
    var rfs = require("rotating-file-stream");
    var stream = rfs.createStream("file.log", {
      size: "10M",
      compress: true
    });
    
    // Or with a custom command:
    var stream = rfs.createStream("file.log", {
      size: "10M",
      compress: (source, dest) => `cat ${source} | gzip -c9 > ${dest}`
    });
  6. Configure rotation limits (maxFiles and maxSize)

    master

    To prevent an unbounded number of rotated files, use maxFiles or maxSize:

    • maxFiles: The maximum number of rotated files to keep.
    • maxSize: The maximum size of rotated files to keep (uses the same syntax as size).

    Note: If you use options.rotate (UNIX logrotate behavior), these options are ignored.

  7. Configure rotation by time interval

    master

    Use the interval option to rotate files at specific time intervals. The value must be a positive integer followed by one of these units:

    • s: seconds (accepts integer dividers of 60)
    • m: minutes (accepts integer dividers of 60)
    • h: hours (accepts integer dividers of 24)
    • d: days (accepts integer)
    • M: months (experimental; rotates at midnight between two distinct months)

    The package uses a setTimeout to schedule rotation jobs.

      interval: '5s', // rotates at seconds 0, 5, 10, 15 and so on
      interval: '5m', // rotates at minutes 0, 5, 10, 15 and so on
      interval: '2h', // rotates at midnight, 02:00, 04:00 and so on
      interval: '1d', // rotates at every midnight
      interval: '1M', // rotates at every midnight between two distinct months
  8. Configure immutable file names for log shippers

    master

    Setting immutable: true ensures that generated file names never change. New files are immediately generated with their rotated name. This is useful for tools like Filebeat that monitor specific file paths.

    Constraints:

    • This option is ignored if options.interval is not used.
    • If immutable is used, options.compress is ignored.
    • The filename argument in the 'open' event will evaluate to the newly created file name.
  9. Configure history file for rotation tracking

    master
    When using maxFiles or maxSize, the package needs to track the number/size of rotated files. The history option specifies the name of the file used to store this information. If history is null (the default), the package uses the not-rotated filename with a .txt suffix.