Giftless Git LFS Server

Giftless is a Python-based implementation of a Git LFS (Large File Storage) server. It is designed to be highly flexible through a pluggable architecture that allows users to customize storage backends, transfer methods, and authentication methods.

Supported Storage Backends

Giftless supports the standard Git LFS transfer mode with the following backends:

• Local storage
• Google Cloud Storage
• Azure Blob Storage (supports both direct-to-cloud and streamed transfers)
• Amazon S3 Storage

Transfer Modes

• Basic Git LFS: The standard transfer mode supported by most Git LFS clients.
• multipart-basic: A custom transfer mode designed to leverage multipart upload capabilities from various cloud vendors. Note that this mode requires a specialized client and is not supported by standard Git LFS clients. For a compatible Python client, use giftless-client.

In Giftless, Storage Backend classes manage the actual file storage (e.g., local filesystem or cloud providers). Transfer Adapters define how clients interact with these files.

Storage backends implement specific interfaces that determine which transfer adapters they can work with:

• StreamingStorage: Supports streaming object upload/download via the Giftless HTTP server. Compatible with the basic_streaming transfer adapter.
• ExternalStorage: Allows referring clients to upload/download objects using an external HTTP server. Compatible with the basic_external transfer adapter. Typically used with Cloud Storage.
• MultipartStorage: Supports the multipart-basic transfer mode. Typically used with Cloud Storage.
• VerifiableStorage: Provides APIs to verify that an object was uploaded correctly. Most concrete adapters implement this.

To use a backend, you must configure it within the options of a specific transfer adapter in your giftless.yaml file.

TRANSFER_ADAPTERS:
  basic:  # The name of the transfer mode
    factory: giftless.transfer.basic_external:factory
    options:
      storage_class: giftless.storage.google_cloud:GoogleCloudStorage
      storage_options:
        # Backend-specific settings go here

JWT tokens can include an authorization payload in the scopes claim. Scopes follow a specific pattern to grant access to specific objects, repositories, or organizations.

Scope Pattern: obj:{org}/{repo}/{oid}:{subscope}:{actions}

Components:

• {org}: The organization name.
• {repo}: The repository name. Use * to grant access to all repositories in the organization.
• {oid}: The Object ID. Use * to grant access to all objects in the repository.
• {subscope}: Optional. Use metadata to grant access to metadata only (e.g., verifying existence without downloading files).
• {actions}: A comma-separated list of read, write, or verify. Use * for all actions.

Scope Examples

{
  "scopes": [
    // read a specific object
    "obj:datopian/somerepo/6adada03e86b154be00e25f288fcadc27aef06c47f12f88e3e1985c502803d1b:read",

    // read the same object, but do not limit to a specific prefix
    "obj:6adada03e86b154be00e25f288fcadc27aef06c47f12f88e3e1985c502803d1b:read",

    // full access to all objects in a repo
    "obj:datopian/my-repo/*",

    // Read only access to all repositories for an organization
    "obj:datopian/*:read",

    // Metadata read only access to all objects in a repository
    "obj:datopian/my-repo:meta:verify"
  ]
}

Giftless uses a two-step process for security:

1. Authenticators: Subclasses of giftless.auth.Authenticator that attempt to extract a user identity from an HTTP request. Multiple authenticators can be chained; if one fails, the next is tried. If all fail, a 401 Unauthorized is returned by default.
2. Identity: An object representing the authenticated user. It contains user metadata (name, email, id) and provides the is_authorized method used by Authorizers to check permissions.

When implementing a custom authenticator, you must return an Identity object that implements the is_authorized interface.

from typing import Optional
from giftless.auth.identity import Permission

class Identity:
    name: Optional[str] = None
    id: Optional[str] = None
    email: Optional[str] = None

    def is_authorized(
        self, 
        organization: str, 
        repo: str, 
        permission: Permission, 
        oid: Optional[str] = None
    ) -> bool:
        """Tell if user is authorized to perform an operation on an object / repo"""
        pass

The multipart-basic transfer mode is an extension to the standard Git LFS basic protocol that allows for chunked file uploads.

Key Characteristics

• Compatibility: Clients must include basic in the transfers list (e.g., ["multipart-basic", "basic"]) to ensure fallback if the server doesn't support this mode.
• Download: download operations behave identically to the basic protocol.
• Upload Lifecycle: Unlike the single-step basic upload, multipart-basic breaks uploads into several optional actions:
  • init: Initialize the upload.
  • parts: Zero or more requests to upload specific file chunks.
  • commit: Finalize the upload.
  • abort: Cancel the upload and clean up state.
  • verify: Verify the file exists in storage.
• Skipping Uploads: If a file is already fully committed to storage, the server will return a response with no actions, allowing the client to skip the upload entirely.

{
  "transfers": ["multipart-basic", "basic"],
  "operation": "upload",
  "objects": [
    {
      "oid": "20492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e",
      "size": 10000000
    }
  ]
}