Files SDK

repository·main·Indexed 23 days ago

https://github.com/haydenbleasel/files-sdk

A unified storage SDK providing a web-standard API for interacting with object and blob storage backends including S3, GCS, Azure, and Vercel Blob. It features web-standard I/O, key-scoped file handles, bulk operations with bounded fan-out, and ready-made tool integrations for AI frameworks like Vercel AI SDK, OpenAI, and Anthropic.

Tokens
192.7K
Snippets
405
Records
846
Agent score
78%

What's inside files-sdk

  1. Overview of Files SDK Plugins

    main

    A plugin is an opt-in, ordered pipeline passed to the createFiles constructor. Unlike hooks which only observe operations, plugins can transform inputs, veto calls (by throwing), or observe results. They can also contribute entirely new namespaced methods to the Files instance.

    Use plugins for logic that requires changing behavior, such as:

    • Envelope-encrypting bodies at rest.
    • Gating uploads through a virus scanner.
    • Metering bandwidth.
    • Mirroring writes to a backup region.
    import { createFiles, handlers } from "files-sdk";
    import { s3 } from "files-sdk/s3";
    
    const files = createFiles({
      adapter: s3({ bucket: "uploads" }),
      plugins: [
        {
          name: "uppercase",
          wrap: handlers({
            upload: (op, next) =>
              next({ ...op, body: (op.body as string).toUpperCase() }),
          }),
        },
      ],
    });
    
    await files.upload("a.txt", "hello"); // stored as "HELLO"
  2. Overview of Files SDK

    main

    Files SDK provides a unified TypeScript API for object storage across 40+ providers, including AWS S3, Cloudflare R2, Vercel Blob, Google Cloud Storage, Azure, and various S3-compatible services (e.g., MinIO, DigitalOcean Spaces, Backblaze B2). It also supports consumer-style providers (Dropbox, Google Drive), upload services (Cloudinary), BaaS stacks (Firebase, Appwrite), and a local fs adapter for testing.

    Key features include:

    • Unified API: The same method signatures work across all adapters.
    • Normalized Errors: Uses a single FilesError type with standardized codes (NotFound, Unauthorized, Conflict, ReadOnly, Provider), preserving the original error in the cause property.
    • Lazy Loading: Adapters are subpath exports (e.g., files-sdk/s3), ensuring you only bundle the provider SDKs you actually use.
    • Typed Escape Hatch: Use files.raw to access the underlying native client (like S3Client or VercelBlobClient) when provider-specific features are required.
    • Agent-friendly CLI: A files binary with JSON output and an MCP server for AI agents.
  3. Project structure of Files SDK videos

    main

    The video project is organized into shared components and version-specific directories:

    • src/shared/: Contains components used across multiple videos, such as Background, IntroScene, Outro, CodeWindow, and typewriter. It also includes the highlight tokenizer and animation utilities (fadeUp/fadeInOut).
    • src/launch/, src/v1-3/, etc.: Contains the bespoke scenes, panels, composition.tsx, and timings.ts for each specific video release.
    • src/root.tsx: The entry point that registers all compositions in the project.
  4. List files with folder-like structure

    main

    You can use list({ delimiter: "/" }) to simulate a folder structure (similar to S3 common prefixes). This is useful for building file-browser UIs.

    • With a prefix (e.g., "photos/"), the items array contains direct files, and ListResult.prefixes contains subfolder paths (e.g., ["photos/2023/", "photos/2024/"]).
    • Compatibility: Object stores and folder-based providers support this. However, flat stores (like UploadThing, Appwrite, PocketBase, Convex, or bun-s3) will throw an error. Check adapter.supportsDelimiter before using this feature.
    • Cursors: A cursor is only valid for the exact prefix and delimiter combination used when it was generated.
  5. How files-sdk works (Mental Model)

    main

    The files-sdk provides a unified API for interacting with various object and blob storage providers (S3, R2, GCS, Azure, Vercel Blob, local filesystem, etc.).

    Core Concepts

    • The Files Instance: You configure a single Files class instance with an adapter at construction. This adapter remains fixed for the life of the instance.
    • Subpath Imports: To keep bundles small, adapters are imported via subpaths (e.g., files-sdk/s3, files-sdk/r2, files-sdk/fs).
    • Unified vs. Native API: The unified API represents the common subset of features supported by all adapters. For provider-specific features (like S3 versioning), use files.raw to access the underlying native client.
    • Web-Standard I/O: All file bodies use web standards: Blob, File, ReadableStream<Uint8Array>, Uint8Array, ArrayBuffer, ArrayBufferView, or string.
    • Error Handling: If an adapter cannot perform a requested unified operation (like a range download), it throws a FilesError rather than failing silently. You can check capability flags like supportsRange or supportsDelimiter at runtime.
  6. Understand Filesystem adapter storage layout

    main

    The adapter uses a sidecar pattern to manage metadata:

    • File Body: Stored at ${root}/${key}
    • Metadata: Stored at ${root}/${key}.meta.json

    Key behaviors:

    • Metadata Persistence: Sidecar files survive operations like cp -r, git mv, or partial-tree deletions.
    • Listing: The list() method automatically hides the .meta.json files from results.
    • ETag: A SHA-1-derived stable hash is computed at upload time.
    • Manual Files: Files written into the root directory manually without a .meta.json sidecar are still readable. In this case, contentType defaults to application/octet-stream and the etag will be absent.
  7. Important considerations for using the zip plugin

    main

    Keep these technical constraints and behaviors in mind when working with archives:

    • Streaming vs Buffering: zip() is a stream; it has flat memory usage because it only keeps one entry in flight at a time. unzip() is a buffer; it must download the entire archive into memory first to read the central directory at the end of the file.
    • ZIP Limits: The plugin supports classic ZIP format only (no ZIP64). There is a limit of 65,535 entries and 4 GiB per entry/archive. Exceeding these limits will cause a failure.
    • Security & Validation: Entry names are validated to prevent attacks like zip-slip. The plugin rejects duplicate names, .. segments, backslashes, and absolute paths. Extracted data is verified against recorded CRC-32 and size.
    • Compression Method: Use method: "store" for files that are already compressed (e.g., JPEGs, videos) to save CPU cycles.
    • Error Handling: For zip(), selection errors (like a missing key or unsafe name) surface on the stream. The stream will reject during the first read.
  8. Handle `signedUploadUrl` return shapes (PUT vs POST)

    main

    The signedUploadUrl method returns a discriminated union. Your client-side code must check the method property to determine how to perform the upload.

    Return Types

    • PUT: Used when maxSize is not provided. Requires a url and optional headers.
    • POST: Used when maxSize is provided (S3/R2 family). Requires a url and a fields object containing policy credentials.

    Critical Requirement for POST

    When using the POST method, you must use multipart/form-data. The file must be the last field appended to the FormData object. S3/R2 providers evaluate the policy against fields following the policy fields; if the file is appended first, the policy will not be applied to it.

    type SignedUpload =
      | { method: "PUT"; url: string; headers?: Record<string, string> }
      | { method: "POST"; url: string; fields: Record<string, string> };
  9. Core API features and capabilities

    main

    The Files instance provides a unified API across all supported providers (S3, GCS, Azure, Vercel Blob, local filesystem, etc.).

    Supported Operations

    • upload(key, body, options)
    • download(key)
    • head(key)
    • exists(key): Returns false only when the provider reports NotFound. Auth or transport errors will still throw.
    • delete(key)
    • copy(sourceKey, destinationKey)
    • move(sourceKey, destinationKey)
    • list(options) / listAll(options)
    • url(key, options)
    • signedUploadUrl(key, options)
    • file(key): Returns a key-scoped handle.

    Key Characteristics

    • Web-standard I/O: Bodies are handled as Blob, File, ReadableStream, Uint8Array, ArrayBuffer, or string. No provider-specific types leak into your code.
    • Escape Hatch: Access the native client for any provider via files.raw to use provider-specific features.
    • Tree-shakeable: Each adapter is a separate entry point, ensuring you only bundle what you use.
  10. Handling errors in bulk operations

    main

    Bulk operation methods—such as upload([...]), download([...]), delete([...]), head([...]), and exists([...])—do not throw on partial failures. Instead, they resolve to a structured result object that contains an errors[] array.

    Each entry in the errors array follows the shape { key, error: FilesError }. This ensures that a single failed key does not cause the entire batch operation to fail. The results (both successes and errors) are returned in the same order as the input keys.

  11. Order compression and encryption plugins

    main

    When using multiple plugins, the order in the plugins array matters.

    Compression must run BEFORE encryption.

    Encryption turns data into effectively random bytes, which cannot be compressed. By placing compression() earlier in the array, the plugin sees the plaintext. Because the SDK unwinds the plugin 'onion' in reverse during downloads, the SDK will automatically handle the sequence: decrypt $\rightarrow$ decompress.

    plugins: [compression(), encryption(key)];
  12. Convex adapter storage layout and metadata

    main

    Files are stored in Convex and tracked via the built-in _storage system table.

    Mapping Details:

    • Key: The file's key is its Id<"_storage">.
    • Metadata Mapping:
      • size $\rightarrow$ size
      • contentType $\rightarrow$ type
      • sha256 $\rightarrow$ etag
      • _creationTime $\rightarrow$ lastModified
    • Limitations:
      • Convex does not support user-defined metadata fields, so the metadata property is always undefined.
      • Warning: Do not set a prefix on the Files instance when using this adapter. Doing so will prepend the prefix to the storage ID and corrupt it.