otiai10/copy Go Library

repository·main·Indexed 20 days ago

https://github.com/otiai10/copy

A Go library for efficient and highly configurable recursive directory copying. It provides the `Copy` function to handle files, directories, symlinks, and named pipes, with an `Options` struct for fine-grained control over concurrency (NumOfWorkers), file filtering (Skip), symlink behavior (OnSymlink), directory collisions (OnDirExists), and permission management (PermissionControl).

Tokens
4K
Snippets
16
Records
18
Agent score
71%

What's inside otiai10/copy

  1. Configure advanced copy behavior with Options

    main

    For fine-grained control over the copy process, use cp.Copy(src, dest, opt) where opt is an instance of Options.

    Common use cases for Options include:

    • Skipping files: Use the Skip function to filter files based on os.FileInfo or path strings.
    • Handling existing directories: Use OnDirExists to define an action when the destination directory already exists.
    • Symlink behavior: Use OnSymlink to decide how to handle symbolic links.
    • Error handling: Use OnError to intercept and decide how to handle specific errors during the copy process.
    • Permissions: Use PermissionControl to either preserve, add, or ignore permissions (e.g., AddPermission(0222) or DoNothing).
    • Performance and Concurrency: Adjust NumOfWorkers to control concurrent copying or CopyBufferSize to change the buffer size (defaults to 32KB).
    opt := Options{
    	Skip: func(info os.FileInfo, src, dest string) (bool, error) {
    		return strings.HasSuffix(src, ".git"), nil
    	},
    }
    err := Copy("your/directory", "your/directory.copy", opt)
  2. Basic usage of copy.Copy

    main

    The copy package provides a simple way to copy directories recursively. Use cp.Copy(src, dest) to perform a standard recursive copy from a source path to a destination path.

    package main
    
    import (
    	"fmt"
    	cp "github.com/otiai10/copy"
    )
    
    func main() {
    	err := cp.Copy("your/src", "your/dest")
    	fmt.Println(err) // nil
    }
  3. Configure Copy behavior with Options

    main

    The Options struct (passed as variadic arguments to Copy) allows fine-grained control over the copy process. While the full definition of Options is not in this file, the following functional capabilities are exposed:

    • NumOfWorkers: Controls the number of concurrent workers used during directory copies.
    • FS: An fs.FS implementation to use for reading the source.
    • Skip: A function func(info os.FileInfo, src, dest string) (bool, error) to decide whether to skip a specific file or directory.
    • OnDirExists: A function to handle cases where the destination directory already exists. Supported return values include Replace (removes existing dir) and Untouchable (skips copying).
    • OnSymlink: A function to define how symlinks are handled. Supported return values include Shallow (copy the link itself), Deep (copy the target of the link), and Skip.
    • RenameDestination: A function func(src, dest string) (string, error) to transform the destination path.
    • PermissionControl: A function to manage file permissions during the copy.
    • WrapReader: A function to wrap the source file reader (e.g., for compression or logging).
    • CopyBufferSize: Sets the buffer size for io.CopyBuffer.
    • Sync: If true, calls f.Sync() to ensure data is flushed to disk.
    • PreserveOwner: If true, attempts to preserve file ownership.
    • PreserveTimes: If true, preserves file timestamps (mtime, atime, ctime).
    • OnError: A custom error handler func(src, dest string, err error) error.
  4. Reference: Options struct fields

    main

    The Options struct allows customizing the recursive copy operation.

    FieldTypeDescription
    OnSymlinkfunc(src string) SymlinkActionSpecifies action on symlinks.
    OnDirExistsfunc(src, dest string) DirExistsActionSpecifies action when destination directory exists.
    OnErrorfunc(src, dest, string, err error) errorCustom error handling logic.
    Skipfunc(srcinfo os.FileInfo, src, dest string) (bool, error)Returns true to skip a file/directory.
    RenameDestinationfunc(src, dest string) (string, error)Allows renaming the destination path.
    PermissionControlPermissionControlFuncControls permissions (e.g., PreservePermission, AddPermission(mode), DoNothing).
    SyncboolIf true, syncs file after copy (higher reliability, lower performance).
    PreserveTimesboolPreserves atime and mtime.
    PreserveOwnerboolPreserves uid and gid.
    CopyBufferSizeuintBuffer size in bytes (0 uses 32KB default).
    WrapReaderfunc(src io.Reader) io.ReaderWraps the source reader (useful for rate limiting).
    FSfs.FSUse a custom filesystem (e.g., embed.FS) instead of the OS filesystem.
    NumOfWorkersint64Number of concurrent workers. 0 or 1 disables goroutines.
    PreferConcurrentfunc(srcdir, destdir string) (bool, error)Determines if goroutines should be used for specific directories.
  5. Use AddPermission to modify file permissions

    main

    The AddPermission function is a helper that generates a PermissionControlFunc. It allows you to bitwise OR additional permissions onto the original source permissions.

    When using AddPermission(perm):

    • If the source is a directory, it first creates the destination directory using 0755 permissions (tmpPermissionForDirectory) to ensure the copy process can continue even if the source directory is read-only.
    • The returned chmodfunc will attempt to apply the original mode combined with the provided perm bits to the destination.
    AddPermission = func(perm os.FileMode) PermissionControlFunc {
  6. Handle errors with OnError

    main

    The OnError field in Options allows you to decide whether the copy operation should continue or stop when an error occurs. The callback receives the source path, destination path, and the error encountered. To continue, return nil. To stop the operation, return the error.

    opt := copy.Options{
        OnError: func(src, dest string, err error) error {
            if isRecoverable(err) {
                return nil // Continue
            }
            return err // Stop
        },
    }
  7. Rename destination files with RenameDestination

    main

    The RenameDestination field in Options allows you to modify the destination path for a specific file or directory before the copy occurs. The callback receives the source and destination paths and must return the new destination path string.

    opt := copy.Options{
        RenameDestination: func(src, dest string) (string, error) {
            return dest + ".bak", nil
        },
    }
  8. Copy files or directories with Copy()

    main

    The Copy function is the primary entry point for the library. It copies a source (src) to a destination (dest), automatically handling whether the source is a single file or a directory. It supports various configurations via the Options variadic parameter.

    Key behaviors:

    • Automatic Type Detection: It uses a switchboard to decide whether to perform a file copy (fcopy), directory copy (dcopy), symlink copy (onsymlink), or named pipe copy (pcopy).
    • Concurrency: If NumOfWorkers is set to more than 1 in Options, the library uses a semaphore to perform concurrent copies.
    • Filesystem Abstraction: If FS is provided in Options, the library uses that filesystem instead of the default os package.
    • Error Handling: If an OnError handler is provided in Options, it will be called instead of returning the error immediately.
    import "github.com/otiai10/copy"
    
    err := copy.Copy("src/path", "dest/path")
    if err != nil {
        // handle error
    }
  9. Configure concurrency with NumOfWorkers and PreferConcurrent

    main

    You can control how the library performs concurrent copying of directory contents:

    • NumOfWorkers: The number of workers used for concurrent copying. If set to 0 or 1, concurrency is disabled for directory contents.
    • PreferConcurrent: A callback function func(srcdir, destdir string) (bool, error) that allows you to decide on a per-directory basis whether to use goroutines for copying. If nil (default), it uses concurrent copying for all directories (provided NumOfWorkers > 1).
    opt := copy.Options{
        NumOfWorkers: 4,
        PreferConcurrent: func(srcdir, destdir string) (bool, error) {
            // Only use concurrency for specific paths
            if strings.Contains(srcdir, "large_data") {
                return true, nil
            }
            return false, nil
        },
    }
  10. Filter files with Skip

    main

    The Skip field in Options allows you to exclude specific files from the copy operation based on their metadata or paths. The callback receives the os.FileInfo, the source path, and the destination path. Return true to skip the file, or false to include it.

    opt := copy.Options{
        Skip: func(srcinfo os.FileInfo, src, dest string) (bool, error) {
            if srcinfo.IsDir() && src == "/some/ignored/dir" {
                return true, nil
            }
            return false, nil
        },
    }
  11. Configure directory collision behavior with OnDirExists

    main

    The OnDirExists field in Options allows you to define a callback function that determines what happens when a directory already exists at the destination path. The callback receives the source and destination paths and must return a DirExistsAction.

    Available DirExistsAction values:

    • Merge: Preserves or overwrites existing files under the directory (default behavior).
    • Replace: Deletes all contents under the destination directory and copies the source files.
    • Untouchable: Does nothing; the existing directory is left as is.
    opt := copy.Options{
        OnDirExists: func(src, dest string) copy.DirExistsAction {
            return copy.Replace
        },
    }
  12. Implement custom permission handling with PermissionControlFunc

    main

    The PermissionControlFunc type allows you to define custom logic for how file permissions (modes) are applied to destination files and directories during a copy operation.

    It is a function that takes the source file information (fs.FileInfo) and the destination path (string) and returns:

    1. A chmodfunc (a function that accepts a pointer to an error *error) used to apply the permission changes.
    2. An error if the initial setup (like directory creation) fails.

    This is useful if you need to intercept the permission application process or handle directory creation specifically.

    type PermissionControlFunc func(srcinfo fs.FileInfo, dest string) (chmodfunc func(*error), err error)