s3fs Go Library

s3fs provides an implementation of the Go 1.16 io/fs (filesystem) interface for Amazon S3.

Because S3 is a flat key-value store rather than a hierarchical filesystem, s3fs simulates a directory structure using key prefixes and the / delimiter. Note that ModTime on simulated directories will always return the zero value.

To install the version of s3fs that uses AWS SDK v1, use the following command:

go get github.com/jszwec/s3fs

To use the version that uses AWS SDK v2, import github.com/jszwec/s3fs/v2.

S3 is a flat key-value store, not a hierarchical filesystem. S3FS simulates directories using prefixes and the / delimiter.

Because directories are simulated rather than being real entities, ModTime for directory entries is always a default zero time.Time value (where IsZero() returns true).

To use s3fs with the AWS SDK v1, create an S3 service client using a session, then initialize s3fs with s3fs.New(s3Client, bucketName). The resulting object implements the fs.FS interface, allowing you to use standard library functions like fs.WalkDir to traverse your S3 bucket.

const bucket = "my-bucket"

s, err := session.NewSession()
if err != nil {
    log.Fatal(err)
}

s3fs := s3fs.New(s3.New(s), bucket)

// print out all files in s3 bucket.
_ = fs.WalkDir(s3fs, ".", func(path string, d fs.DirEntry, err error) error {
    if err != nil {
        return err
    }

    if d.IsDir() {
        fmt.Println("dir:", path)
        return nil
    }
    fmt.Println("file:", path)
    return nil
})

Use openFile to obtain an object that implements the fs.File and io.Seeker interfaces. This allows you to treat an S3 object like a local file, supporting standard operations like Read, Stat, and Seek.

Note that Seek operations use the S3 Range header and IfMatch (using the object's ETag) to ensure data consistency. If the remote file changes during a seek, it will return an error wrapping fs.ErrNotExist.

// Assuming 'cl' is an initialized s3fs.Client
file, err := s3fs.openFile(cl, "my-bucket", "path/to/object")
if err != nil {
    log.Fatal(err)
}
defer file.Close()

// Use as a standard fs.File
info, err := file.Stat()
fmt.Printf("Size: %d\n", info.Size())

Once initialized, S3FS provides standard Go filesystem operations:

• Open(name string) (fs.File, error): Opens a file or directory. If the path is a directory, it returns an fs.ReadDirFile.
• Stat(name string) (fs.FileInfo, error): Returns file information for the given path.
• ReadDir(name string) ([]fs.DirEntry, error): Reads the directory entries for the specified directory.

By default, files opened via S3FS.Open() do not support seeking. To enable Seek functionality, pass WithReadSeeker as an option to New().

Warning: Seeking on S3 requires reopening the file at the specified position. If the file changes between the initial Open() and the Seek() call, fs.ErrNotExist may be returned. Callers should be prepared to handle this error.

// Enable seeking for files opened with this fs
fsys := s3fs.New(cl, "my-bucket-name", s3fs.WithReadSeeker)

Use New() to create a new S3FS instance that implements the Go fs.FS, fs.StatFS, and fs.ReadDirFS interfaces. You must provide an implementation of the Client interface and the name of the S3 bucket you wish to work with. You can optionally pass Option functions to configure the filesystem behavior.

import "github.com/jszwec/s3fs"

// Assuming cl is an implementation of s3fs.Client
fsys := s3fs.New(cl, "my-bucket-name")

The s3fs package provides a dir type that implements the fs.ReadDirFS and fs.ReadDirFile interfaces. This allows you to treat S3 buckets (or specific prefixes within them) as standard Go file systems.

Key behaviors:

• ReadDir(n int): Returns up to n directory entries. If n <= 0, it returns all entries. It handles pagination automatically using S3 markers.
• Stat(): Returns the fs.FileInfo for the directory.
• Read(): Returns an error because directories cannot be read as files.
• Close(): A no-op implementation to satisfy the io.Closer interface.

When calling ReadDir, the implementation merges S3 CommonPrefixes (representing sub-directories) and Contents (representing files) into a single sorted list of fs.DirEntry objects.