c2fo/vfs

repository·main·Indexed 18 days ago

A unified, backend-agnostic interface for interacting with various storage systems such as Local, S3, Azure, GCS, SFTP, and Dropbox using standard Go interfaces like io.Reader and io.Writer. It includes a backend registry for URI-based access (e.g., dbx:// for Dropbox) and a conformance test suite for implementing new storage backends.

Tokens: 50.4K
Snippets: 149
Records: 205
Agent score: 62%

What's inside c2fo/vfs

Manage SFTP AutoDisconnect and idle timeouts
main
Because vfs.FileSystem lacks an explicit Close() method, the SFTP backend implements an automatic disconnection mechanism to release resources.
How it works
- After a connection is established, a timer starts.
- The connection is closed if it remains idle for the duration specified in Options.AutoDisconnect (default is 10 seconds).
- Any server-side request (e.g., List, Read, Touch) resets this timer.
- If the timer expires, the connection closes. The next request will trigger a fresh reconnection.
Configuration
Set Options.AutoDisconnect (integer seconds) to adjust the idle timeout.
```
// Example of idle behavior
fs := sftp.NewFileSystem()
loc, _ := fs.NewLocation("user@server:22", "/path/")
file, _ := loc.NewFile("file.txt")

file.Touch()                // Starts 10s timer
_, _ = loc.List()           // Resets timer

time.Sleep(15 * time.Second) // Timer expires, connection closes
_, _ = loc.List()           // Reconnects automatically
```
https://github.com/c2fo/vfs/blob/main/docs/sftp.md
Understand S3 authentication credential chains
main
When Client() is called, the S3 backend automatically attempts to authenticate using the following providers in order of preference:
1. StaticProvider: Credentials set programmatically via s3.Options. These do not expire.
2. EnvProvider: Credentials from environment variables:
  AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY
  AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY
3. SharedCredentialsProvider: Uses the AWS_SHARED_CREDENTIALS_FILE environment variable. Defaults to the current user's home directory if the variable is empty:
  Linux/OSX: $HOME/.aws/credentials
  Windows: %USERPROFILE%\.aws\credentials
4. RemoteCredProvider: Default remote endpoints such as EC2 or ECS IAM Roles.
5. EC2RoleProvider: Credentials from the EC2 service (tracks expiration).
https://github.com/c2fo/vfs/blob/main/docs/s3.md
Authenticate with SFTP
main
Authentication occurs lazily when the client is first initialized.
Username
The username must be provided in the URI authority section: sftp://username@server.com:22/path/to/file.txt
Password and SSH Keys
You can authenticate using passwords or SSH keys via sftp.Options or environment variables:
Method sftp.Options Key Environment Variable
Password Password VFS_SFTP_PASSWORD
SSH Key Path KeyFilePath VFS_SFTP_KEYFILE
Key Passphrase KeyPassphrase VFS_SFTP_KEYFILE_PASSPHRASE
https://github.com/c2fo/vfs/blob/main/docs/sftp.md
Performance and resource considerations for Dropbox backend
main
Disk Space
The backend uses temporary disk space for:
- Read operations: Downloads the entire file when Read() or Seek() is called.
- Write operations: Buffers all writes until Close() is called.
Network Usage
- Full Downloads/Uploads: Every read requires a full download, and every write requires a full upload on Close(). Even a Touch operation on an existing file results in a download followed by a full re-upload.
Memory Usage
Memory usage remains low because the backend streams data through temporary files rather than holding entire files in memory.
https://github.com/c2fo/vfs/blob/main/contrib/backend/dropbox/README.md
How the in-memory FileSystem works
main
The mem backend implements the vfs.FileSystem interface using an in-memory map.
Key Behaviors:
- Existence: Calling NewFile creates a file object, but the file is not considered 'existent' in the filesystem until Touch() is called or Write() is performed. Touch() links the file with the filesystem's internal map.
- Concurrency: The FileSystem uses a sync.Mutex. For individual File objects, multiple threads can read simultaneously, but writing and closing are protected by locks.
- Locations: Locations in this backend always exist. Creating a file on a location that hasn't been explicitly added to the filesystem's map will automatically create that location.
- Paths: When using NewFile on the FileSystem directly, paths must be absolute. When using NewFile on a Location, paths are relative to that location.
https://github.com/c2fo/vfs/blob/main/docs/mem.md

Method	`sftp.Options` Key	Environment Variable
Password	`Password`	`VFS_SFTP_PASSWORD`
SSH Key Path	`KeyFilePath`	`VFS_SFTP_KEYFILE`
Key Passphrase	`KeyPassphrase`	`VFS_SFTP_KEYFILE_PASSPHRASE`

Understand S3 to VFS event mapping

main

The s3events watcher uses operation-based logic to map S3 event types to VFS event types. This provides semantic accuracy by distinguishing between direct uploads, copies, and restores.

S3 Event Type	VFS Event Type	Description
`s3:ObjectCreated:Put`	`EventCreated`	Direct file uploads (typically new files)
`s3:ObjectCreated:Post`	`EventCreated`	Form-based uploads (typically new files)
`s3:ObjectCreated:Copy`	`EventModified`	Copy operations (often overwrites/modifications)
`s3:ObjectCreated:CompleteMultipartUpload`	`EventModified`	Large uploads (often significant changes)
`s3:ObjectCreated:*`	`EventCreated`	Wildcard for broad compatibility
`s3:ObjectRestore:Post`	`EventModified`	Restore initiation from Glacier
`s3:ObjectRestore:Completed`	`EventModified`	Restore completion (object available)
`s3:ObjectRestore:Delete`	`EventDeleted`	Temporary restored copy expires
`s3:ObjectRemoved:Delete`	`EventDeleted`	Object deletion
`s3:ObjectRemoved:DeleteMarkerCreated`	`EventDeleted`	Versioned object deletion
`s3:ObjectRemoved:*`	`EventDeleted`	Wildcard deletion events

Note on Mapping Trade-offs:

Copy operations are mapped to EventModified because they often represent overwrites, though they may sometimes create new keys.
Multipart uploads are mapped to EventModified as they represent significant changes, though initial uploads are technically creations.
Applications requiring precise distinction should inspect the event.Metadata for the original eventName and operation type.

How event debouncing works in FSNotify
main
Event debouncing consolidates multiple related filesystem events into single logical events. This reduces noise and improves performance for build tools, hot reload systems, or network filesystems (like SFTP/NFS) with delayed writes.
Consolidation Rules:
1. Delete events take priority over Create/Modified events.
2. Create events take priority over Modified events.
3. Multiple events for the same file are merged into a single event.
4. Event metadata includes "fsnotify_op": "multiple" for consolidated events.
5. Timestamp reflects the first event time in the sequence.
Performance Impact:
- Event Reduction: Typically 50-80% fewer events processed.
- Memory Overhead: ~200 bytes per pending file during the debounce period.
- Latency: The delay is configurable via the debounce duration.
https://github.com/c2fo/vfs/blob/main/contrib/vfsevents/watchers/fsnotify/README.md

Compare available vfsevents Watchers

main

Choose a watcher implementation based on your storage backend and latency requirements:

Watcher	Best For	Latency	Key Features
FSNotify	Local filesystems, development	< 1ms	Kernel-level events, recursive watching, very low resource usage
VFS Poller	Cloud storage (S3/GCS), Network FS	30s - 5m	Universal (any VFS backend), retry logic, modification detection
S3Events	Amazon S3 production workflows	~1-5s	Uses SQS, native S3 event notifications, automatic retry
GCSEvents	Google Cloud Storage workflows	~1-5s	Uses Pub/Sub, native GCS event notifications, subscription management

Authenticate with FTP
main
Authentication in the FTP backend is handled via the URI authority section and/or configuration options.
Username
The username must be part of the URI authority section: ftp://username@server.com:21/path/
If no username is provided in the URI (e.g., ftp://service.com/), the backend defaults to anonymous.
Password
Passwords follow this precedence (highest to lowest):
1. ftp.Options.Password
2. Environment variable VFS_FTP_PASSWORD
3. Default value: anonymous
If no password is provided via any method, it defaults to anonymous.
https://github.com/c2fo/vfs/blob/main/docs/ftp.md

How VFS backends work and how to use them

main

VFS uses a self-registration pattern for its backends. Each backend package implements an init() function that calls backend.Register("scheme", vfs.FileSystem). This allows consumers to use specific file systems simply by importing the backend package and then retrieving the file system via the backend package using its scheme name.

To use multiple backends, import the desired backend packages (e.g., os, s3) and use backend.Backend(scheme) to obtain the vfs.FileSystem instance.

package main

import (
	"github.com/c2fo/vfs/v7"
	"github.com/c2fo/vfs/v7/backend"
	"github.com/c2fo/vfs/v7/backend/os"
	"github.com/c2fo/vfs/v7/backend/s3"
)

func main() {
    var err error
    var osfile, s3file vfs.File

    // Retrieve the OS file system using its scheme
    osfile, err = backend.Backend(os.Scheme).NewFile("", "/path/to/file.txt")
    if err != nil {
        panic(err)
    }

    // Retrieve the S3 file system using its scheme
    s3file, err = backend.Backend(s3.Scheme).NewFile("mybucket", "/some/file.txt")
    if err != nil {
        panic(err)
    }

    // Perform cross-backend operations like CopyTo
    err = osfile.CopyTo(s3file)
    if err != nil {
        panic(err)
    }
}

Thread safety in the Dropbox backend
main
Individual File and Location objects are not thread-safe. You must use synchronization (like mutexes) if sharing these specific instances across goroutines.
However, it is safe to create multiple FileSystem, Location, and File instances across different goroutines simultaneously.
https://github.com/c2fo/vfs/blob/main/contrib/backend/dropbox/README.md

Understand GCS Event Mapping and Overwrite Suppression

main

The watcher translates GCS Pub/Sub events into VFS events using semantic logic. A key feature is Overwrite Suppression: when a file is overwritten, GCS typically emits both a finalize event and a delete event. The watcher suppresses the redundant delete event to ensure an overwrite is treated as a single, atomic EventModified operation rather than a delete followed by a create.

| GCS Event Type | VFS Event Type | Condition | Description |
|---|---|---|---|
| `OBJECT_FINALIZE` | `EventCreated` | No `overwroteGeneration` | New file creation |
| `OBJECT_FINALIZE` | `EventModified` | Has `overwroteGeneration` | File overwrite, copy, or restore |
| `OBJECT_METADATA_UPDATE` | `EventModified` | Always | Metadata changes |
| `OBJECT_DELETE` | `EventDeleted` | No `overwrittenByGeneration` | True file deletion |
| `OBJECT_DELETE` | *Suppressed* | Has `overwrittenByGeneration` | Part of overwrite (not emitted) |
| `OBJECT_ARCHIVE` | `EventDeleted` | No `overwrittenByGeneration` | File archival |
| `OBJECT_ARCHIVE` | *Suppressed* | Has `overwrittenByGeneration` | Part of overwrite (not emitted) |

c2fo/vfs

What's inside c2fo/vfs

How it works

Configuration

Username

Password and SSH Keys

Disk Space

Network Usage

Memory Usage

Key Behaviors:

Username

Password