minipass

repository·main·Indexed 42 days ago

https://github.com/isaacs/minipass

A highly optimized, minimal implementation of a Node.js PassThrough stream designed for high-speed handling of buffers, strings, and objects. Version 7.1.3 features synchronous data emission by default for maximum performance, support for async mode, and TypeScript generics for type-safe reading and writing.

Tokens
7.1K
Snippets
22
Records
30
Agent score
4%

What's inside minipass

  1. What is minipass

    main

    minipass is a minimal, high-performance implementation of a Node.js PassThrough stream. It is optimized for speed when handling objects, strings, and buffers.

    Key features include:

    • High Performance: Extremely fast for objects, strings, and buffers.
    • Reliable Piping: Supports pipe() including multi-pipe() and backpressure transmission.
    • Data Buffering: Buffers data until a data event handler or a pipe() is added, preventing the loss of the first chunk of data.
    • Minimal Copying: Performs minimal Buffer copying to ensure proper stream semantics when read(n) is called, unless objectMode: true is used.
  2. How Minipass timing differs from Node.js core streams

    main

    By default, Minipass streams are designed for synchronous use-cases. Data is emitted as soon as it is available and is buffered only until it is read. This means stream.write() may trigger 'data' events immediately within the same execution tick, rather than deferring them to the next tick like Node.js core streams.

    Key behaviors:

    • stream.write() is synchronous with the logic writing into it.
    • resume() may result in the entire buffer being consumed immediately.
    • This approach maximizes performance by minimizing waiting and unnecessary buffering.
    import { Minipass } from 'minipass'
    
    const stream = new Minipass()
    stream.on('data', () => console.log('data event'))
    console.log('before write')
    stream.write('hello')
    console.log('after write')
    // output:
    // before write
    // data event
    // after write
  3. Understand Minipass backpressure and water marks

    main

    Minipass does not use highWaterMark or lowWaterMark for optimistic buffering. Instead, it uses a simpler mechanism:

    • stream.write() returns true if the data has somewhere to go (i.e., it was successfully flushed through the pipeline).
    • stream.write() returns false if the data has nowhere to go, in which case it sits in a buffer to be drained as soon as a consumer is ready.

    Because Minipass communicates backpressure immediately and minimizes unnecessary buffering, it is generally faster than Node.js core streams in piped pipelines.

  4. Iterate over streams synchronously or asynchronously

    main

    Minipass streams support both synchronous and asynchronous iteration.

    Synchronous Iteration

    Synchronous iteration (for...of) ends when the currently available data is consumed, even if the end event has not been reached. In string or buffer mode, data is concatenated, so sync loops often only run once per tick. To consume chunks exactly as written without flattening, use { objectMode: true }.

    Asynchronous Iteration

    Asynchronous iteration (for await...of) continues until the end event is reached, consuming all data emitted by the stream.

    // Synchronous example (objectMode to prevent flattening)
    const mp = new Minipass({ objectMode: true })
    mp.write('a')
    mp.write('b')
    for (let letter of mp) {
      console.log(letter) // a, b
    }
    
    // Asynchronous example
    const mp = new Minipass({ encoding: 'utf8' })
    // ... data source writes to mp ...
    async function consume() {
      for await (let chunk of mp) {
        console.log(chunk)
      }
      return 'ok'
    }
    // Synchronous iteration
    const mp = new Minipass({ objectMode: true })
    mp.write('a')
    mp.write('b')
    for (let letter of mp) {
      console.log(letter) // a, b
    }
    
    // Asynchronous iteration
    const mp = new Minipass({ encoding: 'utf8' })
    async function consume() {
      for await (let chunk of mp) {
        console.log(chunk)
      }
      return 'ok'
    }
  5. Handle immediate 'end' and 'error' events

    main

    Minipass streams are designed to be resilient to the timing of event listener attachment:

    • Immediate end: If end() is called on an empty, non-paused stream, it emits 'end' immediately. If you need to prevent immediate execution of end logic, call stream.pause() on creation and stream.resume() when ready.
    • Re-emitting end: If you attach an 'end' listener after the stream has already emitted 'end', the listener is called immediately (similar to a resolved Promise).
    • Re-emitting error: If an error was previously emitted, adding a new 'error' listener will trigger that handler immediately (or on process.nextTick if in async mode).
  6. How to transform data by extending Minipass

    main

    minipass is a PassThrough stream, meaning it does not transform data by default. To implement a transformation stream, you must extend the Minipass class and override the write() method. After performing your transformation, call super.write() with the transformed output.

    Important Note on objectMode: objectMode can only be set at instantiation. If you attempt to write something other than a String or Buffer without having set objectMode: true in the options, the stream will throw an error.

    class MyTransformStream extends Minipass {
      write(data, encoding, cb) {
        const transformed = data.toString().toUpperCase();
        return super.write(transformed, encoding, cb);
      }
    }
  7. Extend Minipass to create custom transforms

    main

    You can create custom stream logic by subclassing Minipass and overriding write(chunk, encoding, callback) and end(chunk, encoding, callback). This is useful for creating logging wrappers, custom encoders, or decoders.

    Example: NDJSON Encoder

    To create a transform that converts objects into newline-delimited JSON:

    class NDJSONEncode extends Minipass {
      write(obj, cb) {
        try {
          return super.write(JSON.stringify(obj) + '\n', 'utf8', cb)
        } catch (er) {
          this.emit('error', er)
        }
      }
      end(obj, cb) {
        if (typeof obj === 'function') {
          cb = obj
          obj = undefined
        }
        if (obj !== undefined) {
          this.write(obj)
        }
        return super.end(cb)
      }
    }

    Example: NDJSON Decoder

    To create a transform that parses newline-delimited JSON into objects (requires objectMode: true):

    class NDJSONDecode extends Minipass {
      constructor(options) {
        super({ objectMode: true })
        this._jsonBuffer = ''
      }
      write(chunk, encoding, cb) {
        if (typeof chunk === 'string' && typeof encoding === 'string' && encoding !== 'utf8') {
          chunk = Buffer.from(chunk, encoding).toString()
        } else if (Buffer.isBuffer(chunk)) {
          chunk = chunk.toString()
        }
        if (typeof encoding === 'function') {
          cb = encoding
        }
        const jsonData = (this._jsonBuffer + chunk).split('\n')
        this._jsonBuffer = jsonData.pop()
        for (let i = 0; i < jsonData.length; i++) {
          try {
            super.write(JSON.parse(jsonData[i]))
          } catch (er) {
            this.emit('error', er)
            continue
          }
        }
        if (cb) cb()
      }
    }
  8. Avoid data loss when using Tee-streams or multiple listeners

    main

    Because Minipass processes data immediately, piping a single source to multiple destinations (Tee-streaming) or adding multiple 'data' listeners can result in data loss. The first destination or listener will consume the buffered data immediately, leaving nothing for subsequent ones.

    Incorrect approach (Data Loss):

    const src = new Minipass()
    src.write('foo')
    src.pipe(dest1) // 'foo' flows to dest1 immediately
    src.pipe(dest2) // dest2 receives nothing

    Correct approach (Use a Tee-junction): Create a dedicated Minipass stream to act as a junction that pipes to both destinations.

    const src = new Minipass()
    src.write('foo')
    const tee = new Minipass()
    tee.pipe(dest1)
    tee.pipe(dest2)
    src.pipe(tee) // tee receives 'foo' and distributes it to both

    Note: All these hazards can be avoided by enabling { async: true } mode, though this incurs a performance penalty.

    // Safe example: tee to both places
    const src = new Minipass()
    src.write('foo')
    const tee = new Minipass()
    tee.pipe(dest1)
    tee.pipe(dest2)
    src.pipe(tee) // tee gets 'foo', pipes to both locations
  9. Handle stream lifecycle and errors

    main

    Minipass streams emit several key events for lifecycle management:

    • 'data': Emitted when a chunk is available.
    • 'end': Emitted when the stream has no more data to provide (after end() is called).
    • 'finish': Emitted when the stream has finished writing.
    • 'error': Emitted when an error occurs. If an AbortSignal was provided, errors may be handled via the signal.
    • 'abort': Emitted when the stream is aborted (e.g., via an AbortSignal).
    • 'close': Emitted when the stream is closed.
    • 'drain': Emitted when a writable stream is ready to receive more data.

    Special Behavior:

    • Adding a 'data' listener automatically triggers the flow of data.
    • Adding an 'error' listener after an error has already been emitted will cause the error to be re-emitted immediately.
    • Adding an 'end' (or 'finish') listener after the event has passed will cause the event to be emitted immediately.
  10. Configure Minipass options

    main

    When instantiating new Minipass(options), you can provide the following configuration keys:

    • encoding: How data coming out of the stream is encoded. Accepts any values compatible with Buffer.toString().
    • objectMode: If true, data is emitted exactly as it comes in. This is enabled by default if you write() anything other than a string or Buffer. Note that setting objectMode: true prevents setting an encoding.
    • async: Defaults to false. If true, data emission is deferred until the next tick, making timing behavior closer to Node.js core streams at a slight performance cost.
    • signal: An AbortSignal that causes the stream to unhook itself and become inert. When a signal is provided, 'error' events will no longer throw if unhandled, though they are still emitted to attached handlers.
  11. Basic usage of Minipass

    main

    Minipass is a stream implementation that can be used like a standard Node.js stream. You can write data to it, pipe it to other streams, and signal the end of the stream.

    import { Minipass } from 'minipass'
    const mp = new Minipass(options) // options is optional
    mp.write('foo')
    mp.pipe(someOtherStream)
    mp.end('bar')
  12. Enable async mode in Minipass streams

    main

    If your code relies on the deferred behavior of Node.js core streams (where 'data' events occur on the next tick), you can opt into async mode. This can be done via the constructor or by setting the async property on an existing stream.

    Note: Once a stream is set to async: true, attempting to set stream.async = false is a no-op and will be ignored to prevent data corruption.

    // Via constructor
    const asyncStream = new Minipass({ async: true })
    
    // Via property
    const stream = new Minipass()
    stream.async = true
    
    asyncStream.on('data', () => console.log('data event'))
    console.log('before write')
    asyncStream.write('hello')
    console.log('after write')
    // output:
    // before write
    // after write
    // data event <-- deferred until the next tick