readdirp

repository·master·Indexed 19 days ago

https://github.com/paulmillr/readdirp

A high-performance, recursive version of fs.readdir for Node.js designed with a small RAM and CPU footprint. Version 5.0.0 provides both a Stream API (via readdirp()) for memory-efficient traversal of large directory trees and a Promise API (via readdirpPromise()) for retrieving all entries as an array. It supports customizable filtering via fileFilter and directoryFilter, recursion depth control, and configurable entry types.

Tokens
2.2K
Snippets
10
Records
14
Agent score
15%

What's inside readdirp

  1. How the Stream API works

    master

    The readdirp(root, options) function returns a stream that recursively reads the root directory.

    Event Lifecycle:

    • data: Emitted for every file or directory found.
    • warn: Emitted for non-fatal errors (e.g., a directory that is inaccessible). The stream continues.
    • error: Emitted for fatal errors (e.g., invalid options). The stream ends.
    • end: Emitted when all entries have been found and no more will be emitted.
    • close: Emitted when the stream is destroyed via stream.destroy(). This is useful for manually aborting the process.
  2. Install readdirp

    master

    You can install readdirp using npm or add it via JSR depending on your environment.

    npm install readdirp
    # or
    jsr add jsr:@paulmillr/readdirp
  3. Use the Promise API

    master

    If you prefer working with Promises and do not require the low memory footprint of streams, use readdirpPromise. Note that this consumes more RAM and CPU than the stream-based approach as it returns a full list of entries.

    import { readdirpPromise } from 'readdirp';
    
    const files = await readdirpPromise('.');
    console.log(files.map(file => file.path));
  4. Use the Stream API with for-await

    master

    The Stream API is the most efficient way to use readdirp, providing a small RAM and CPU footprint. In Node.js 10+, you can use the asyncIterator pattern with for await...of to iterate over entries.

    import readdirp from 'readdirp';
    
    for await (const entry of readdirp('.')) {
      const {path} = entry;
      console.log(`${JSON.stringify({path})}`);
    }
  5. Use the Stream API with event listeners

    master

    You can also consume the stream using standard Node.js stream events. This is useful for handling non-fatal warnings, fatal errors, and stream completion.

    import readdirp from 'readdirp';
    
    readdirp('.', {alwaysStat: true, fileFilter: (f) => f.basename.endsWith('.js')})
      .on('data', (entry) => {
        const {path, stats: {size}} = entry;
        console.log(`${JSON.stringify({path, size})}`);
      })
      .on('warn', error => console.error('non-fatal error', error))
      .on('error', error => console.error('fatal error', error))
      .on('end', () => console.log('done'));
  6. Configure readdirp options

    master

    You can pass a Partial<ReaddirpOptions> object to readdirp() or readdirpPromise() to customize the traversal.

    Available Options

    • root: (Required) The starting directory path.
    • fileFilter: A Predicate to filter files. Can be a Tester function, a string (exact basename match), or a string[] (array of exact basename matches).
    • directoryFilter: A Predicate to filter directories.
    • type: Determines which entries are emitted. Use EntryTypes for valid values.
    • lstat: If true, uses lstat instead of stat (useful for symlink-friendly traversal).
    • depth: Maximum recursion depth.
    • alwaysStat: If true, always performs a stat call. If false (default), it uses dirent for better performance.
    • highWaterMark: Sets the maximum amount of resources per entry in the stream.
  7. Reference: readdirp options

    master

    The following options are available for the readdirp function:

    • fileFilter: A function (entry) => boolean used to include or exclude files.
    • directoryFilter: A function used to include or exclude directories. Directories that fail this filter will not be recursed into.
    • depth: A number specifying the maximum depth to recurse.
    • type: Determines which entries are emitted. Options: 'files' (default), 'directories', 'files_directories', 'all' (includes character devices, unix sockets, etc.).
    • alwaysStat: Boolean. If true, returns the stats property for every file. Note: This can double execution time. Default is false (returns dirent instead).
    • lstat: Boolean. If true, includes symlink entries using fs.lstat instead of fs.stat. Default is false.
  8. Configure readdirp options

    master

    When calling readdirp(root, options), you can pass an options object to control filtering, recursion depth, and entry types.

    import readdirp from 'readdirp';
    
    readdirp('test', {
      fileFilter: (f) => f.basename.endsWith('.js'),
      directoryFilter: (d) => d.basename !== '.git',
      type: 'files_directories',
      depth: 1
    });
  9. Reference: EntryInfo properties

    master

    Each entry emitted by the stream or returned by the promise is an EntryInfo object containing:

    • path: Path to the file/directory (relative to the given root).
    • fullPath: The absolute path to the file/directory found.
    • basename: The name of the file/directory.
    • dirent: The built-in fs.Dirent object (only available if alwaysStat is false).
    • stats: The built-in fs.Stats object (only available if alwaysStat is true).
  10. Use readdirpPromise() to get all entries as an array

    master

    If you need all directory entries at once and are not concerned about high memory usage (e.g., for small directory trees), use readdirpPromise(). This returns a Promise that resolves to an array of EntryInfo objects.

    Warning: For very large file systems (e.g., millions of files), this will consume a significant amount of RAM.

    import { readdirpPromise } from 'readdirp';
    
    const entries = await readdirpPromise('.', { type: 'files' });
    console.log(entries);
  11. Use readdirp() for streaming directory traversal

    master

    The readdirp() function returns a ReaddirpStream, which is a Node.js Readable stream in object mode. This is the recommended way to traverse directories because it uses a small, constant amount of RAM regardless of the number of files in the tree. You can consume the stream using an for await...of loop.

    Each emitted entry is an EntryInfo object containing the relative path, absolute fullPath, basename, and either stats or dirent depending on your configuration.

    import readdirp from 'readdirp';
    
    for await (const entry of readdirp('.')) {
      const {path} = entry;
      console.log(`${JSON.stringify({path})}`);
    }
  12. Reference EntryInfo interface

    master

    Each entry emitted by the stream or returned by the promise follows the EntryInfo interface.

    export interface EntryInfo {
      path: string;      // Relative path from root
      fullPath: string;  // Absolute path
      stats?: Stats;    // File stats (if alwaysStat is true or if symlink needs resolving)
      dirent?: Dirent;   // Directory entry (if alwaysStat is false)
      basename: string;  // The filename or directory name
    }