fastwalk: Parallel Directory Traversal for Go

The fastwalk package provides a high-performance, parallel version of filepath.WalkDir. It uses multiple goroutines to concurrently stat the filesystem and execute the callback function, making it significantly faster than the standard library on macOS, Linux, and Windows.

Important Requirement: Because fastwalk executes the callback function concurrently, your walkFn (the function that processes each file/directory) must be safe for concurrent use (thread-safe).

// Example of basic usage
conf := &fastwalk.Config{
    Follow: true, // Set to true to follow symbolic links
}

walkFn := func(path string, d fs.DirEntry, err error) error {
    // Your logic here. 
    // MUST BE THREAD-SAFE
    fmt.Println(path)
    return nil
}

err := fastwalk.Walk(conf, "/path/to/root", walkFn)

To generate performance comparisons against filepath.WalkDir and godirwalk, use the bench_comp make target.

$ make bench_comp

You can control the behavior of the traversal using the fastwalk.Config struct. The primary configuration option is:

• Follow (bool): If set to true, the walker will follow symbolic links during traversal.

conf := fastwalk.Config{
    Follow: true,
}

The package provides wrapper functions to help avoid processing the same files or directories multiple times during traversal:

• IgnoreDuplicateFiles()
• IgnoreDuplicateDirs()

The Config struct allows you to tune the performance and behavior of the Walk function. If nil is passed to Walk, DefaultConfig is used.

conf := &fastwalk.Config{
	Follow:     true,
	NumWorkers: 16,
	Sort:       fastwalk.SortFilesFirst,
	MaxDepth:   5,
}

The DefaultNumWorkers() function calculates a sensible number of goroutines for parallel traversal based on the system's CPU count and operating system.

• General: Clamps GOMAXPROCS to a range of 4 to 32.
• Darwin (macOS): Uses a specialized heuristic because high concurrency can degrade IO performance on APFS. It attempts to use the number of performance cores on modern ARM Macs, or falls back to specific counts (4, 6, or 10) based on core count for Intel Macs.
• Windows/Linux: Uses the clamped GOMAXPROCS value.

You can override this by setting the NumWorkers field in your Config object.

On Windows, EntryFilter provides a mechanism to track and filter filesystem entries using unique file identifiers (Volume Serial Number and File Index). This is useful for preventing infinite loops or redundant processing caused by symbolic links or hard links during filesystem traversal.

To use it, instantiate a new filter with NewEntryFilter() and call its Entry(path string, _ fs.DirEntry) bool method. The Entry method returns true if the file has been seen before, and false if it is a new entry.

import "github.com/charlievieth/fastwalk"

filter := fastwalk.NewEntryFilter()

// Inside a walk function:
// If Entry returns true, the file has already been encountered.
if filter.Entry(currentPath, entry) {
    // Skip this entry to avoid duplicates
    return
}

The StatDirEntry function provides a way to obtain an fs.FileInfo for a given directory entry.

If the provided de is a fastwalk.DirEntry, the function uses its Stat method, which may return a cached FileInfo to improve performance. If you require a fresh os.Stat call that bypasses any potential caching, you should call os.Stat directly instead.

If de is not a fastwalk.DirEntry, the function falls back to calling os.Stat(path).

import (
	"io/fs"
	"os"
	"github.com/charlievieth/fastwalk"
)

// Example usage within a walk function
// path is the current file path, de is the fs.DirEntry provided by the walker
info, err := fastwalk.StatDirEntry(path, de)
if err != nil {
	// handle error
}
// use info (fs.FileInfo)

Use IgnorePermissionErrors to wrap an fs.WalkDirFunc. This wrapper intercepts fs.ErrPermission (or any error where os.IsPermission(err) is true) and returns nil instead of passing the error to the caller. This allows the directory traversal to continue even when certain files or directories are inaccessible.

Key Behaviors:

• Unlike the duplicate-checking wrappers, the returned fs.WalkDirFunc may be reused safely.

// Example usage wrapping a standard walk function
walkFn := func(path string, d fs.DirEntry, err error) error {
    fmt.Println(path)
    return nil
}

wrappedWalk := fastwalk.IgnorePermissionErrors(walkFn)
// Use wrappedWalk in your Walk implementation

The EntryFilter type is used to track visited directory entries. It helps detect and avoid symlink loops or prevent processing the same file multiple times during a filesystem walk. It works by tracking unique file identifiers consisting of a device number (Dev) and an inode number (Ino).

To use it, initialize a filter with NewEntryFilter() and check entries using the Entry method during your walk process.

import "github.com/charlievieth/fastwalk"

filter := fastwalk.NewEntryFilter()
// Inside your walk function:
if filter.Entry(path, de) {
    // This entry has been seen before (e.g., a symlink loop or duplicate)
    return
}