TigerFS Documentation

repository·main·Indexed 20 days ago

https://github.com/timescale/tigerfs

TigerFS is a versioned filesystem backed by PostgreSQL that provides a filesystem interface to database tables. It allows humans and AI agents to interact with structured data using standard Unix tools via two modes: File-first for content management with automatic versioning and undo operations, and Data-first for exploring existing databases. It features pipeline queries via path segments, schema management through filesystem operations, and support for mounting cloud backends like Tiger Cloud and Ghost.

Tokens
105K
Snippets
294
Records
413
Agent score
70%

What's inside TigerFS

  1. Overview of TigerFS

    main
    TigerFS is a FUSE-based (Filesystem in Userspace) filesystem that exposes PostgreSQL database contents as mountable directories. It allows users to interact with tables, rows, and columns using standard Unix tools like ls, cat, grep, and rm instead of writing SQL queries. This makes database data accessible to AI coding assistants (like Claude Code) and standard command-line tools that expect a filesystem interface.
  2. Key features and capabilities of TigerFS

    main

    TigerFS is a transactional, concurrent filesystem designed for human-agent collaboration. It treats files as the primary API for interacting with databases.

    Core Capabilities:

    • Transactional Workspaces: Supports Markdown and plaintext workspaces with YAML frontmatter and directory hierarchies. Per-file version browsing is available via .history/<file>/.
    • Undo and Recovery: Features savepoints, an operation log, and atomic undo. Every log entry includes before/current/after diff symlinks to preview changes or roll back states.
    • ACID Concurrent Writes: Allows multiple agents and humans to operate on the same workspace simultaneously without merge conflicts.
    • Schema Management via Files: Manage database schemas using standard filesystem commands:
      • mkdir to create a table.
      • Edit a .modify/ draft to propose schema alterations.
      • touch .commit to apply the changes.
    • Pipeline Queries: Use ls and cat to navigate complex queries. You can chain filters, sorting, pagination, and column projection into a single path that translates into an optimized SQL query.
    • Format Agnostic I/O: Read and write rows using JSON, YAML, CSV, or single columns. You can look up data by any indexed column, not just the primary key.
    • Migration Framework: Use tigerfs migrate for in-place, idempotent updates to schemas, triggers, and indexes across releases. Supports --dry-run and --describe flags.
  3. Explore TigerFS capabilities via the implementation checklist

    main

    The TigerFS implementation roadmap outlines the project's core capabilities, ranging from basic FUSE filesystem mounting to advanced synthesized applications. Key functional areas include:

    • Core Filesystem Operations: Database connection, schema discovery, and CRUD operations (Create, Read, Update, Delete) mapped to filesystem actions.
    • Data Formats: Support for TSV, CSV, and JSON formats.
    • Advanced Querying: Index-based queries, pagination (.first/N/, .last/N/), random sampling (.sample/N/), and large table escape hatches (.all/).
    • Synthesized Applications: Specialized views for Markdown/Plain Text, Task management, and Directory Hierarchies.
    • DDL via Filesystem: Performing Data Definition Language operations (creating/modifying/deleting tables, views, and indexes) directly through the filesystem.
    • Version Control & Recovery: Support for .history/, .log/, .savepoint/, and .undo/ interfaces to manage data changes and recovery.
  4. Handle composite primary keys in file paths

    main

    For tables with multi-column primary keys, TigerFS uses a comma-delimited path representation.

    Path Format:

    • For a primary key consisting of (customer_id, product_id) with values (5, 42), the directory name is 5,42.
    • Values containing commas are URL-encoded as %2C to prevent path ambiguity.

    Example: If a table order_items has a composite PK (customer_id, product_id):

    • ls /mount/order_items/ might show 1,100.
    • cat /mount/order_items/1,100/quantity retrieves the value for that specific composite row.
    # Manual verification of composite PK
    psql -c "CREATE TABLE order_items (customer_id int, product_id int, quantity int, PRIMARY KEY (customer_id, product_id));"
    psql -c "INSERT INTO order_items VALUES (1, 100, 5), (1, 200, 3);"
    
    ls /mount/order_items/            # Should show: 1,100
    cat /mount/order_items/1,100      # Should show row data
  5. Use advanced filesystem capabilities for querying and navigation

    main

    TigerFS provides several special path patterns and virtual files to interact with database content using standard filesystem tools:

    Querying and Pagination

    • .all/: An escape hatch for interacting with large tables.
    • .first/N/ and .last/N/: Used for pagination to retrieve the first or last N records.
    • .sample/N/: Used for random sampling of N records.
    • .count: A virtual file representing the total row count.
    • .order/<column>/: Capability to navigate or query data ordered by a specific column.
    • .by/: Used for index-based navigation.

    Metadata and Schema

    • .columns, .schema, and .count: Metadata files describing the table structure and size.
    • .indexes: Metadata file describing available indexes.
    • .ddl: Extended schema file.
    • .info/: A subdirectory containing metadata about the table or view.

    Data Management

    • .export/: Bulk read capability.
    • .import/: Bulk write capability.
  6. Identify when data is committed to the database

    main

    In TigerFS, data stored in the memFile cache is committed to the underlying database based on the following triggers:

    TriggerDescription
    Sync()Triggered by editor saves (e.g., fsync). Performs an immediate commit.
    Close() with refCount=0The last open handle to the file is closed.
    Idle timeout (5 min)A background reaper commits entries that have been idle for more than 5 minutes (handles crashed clients).
    Server shutdownA graceful shutdown flushes all dirty entries to the database.
  7. How TigerFS resolves database credentials

    main

    TigerFS uses a layered credential resolution strategy to find PostgreSQL database credentials. It prioritizes standard PostgreSQL conventions to ensure compatibility across desktop, server, container, and CI/CD environments.

    When looking for a password, TigerFS checks sources in the following order of precedence (highest to lowest):

    1. Connection String: The password provided directly in the connection string.
    2. Environment Variables: The PGPASSWORD or TIGERFS_PASSWORD environment variables.
    3. password_command: An external command configured in the settings used to fetch secrets from a manager.
    4. ~/.pgpass file: The standard PostgreSQL password file (handled via the pgx library).

    Note: TigerFS does not use system keyrings (like macOS Keychain or Windows Credential Manager) and does not support plain-text passwords in the configuration file.

  8. Store custom metadata in a headers JSONB column

    main

    To support arbitrary user-defined metadata in markdown files without changing the database schema, TigerFS uses a headers JSONB column.

    How it works:

    • Synthesis: During file generation, any key-value pairs stored in the headers JSONB column are merged into the YAML frontmatter block (sorted alphabetically after known columns).
    • Parsing: When a user adds a key to the markdown frontmatter that does not correspond to a fixed column (like title or author), TigerFS collects these "unknown" keys and stores them in the headers JSONB column.
    • Overwrite Semantics: If a key is removed from the markdown frontmatter, it is removed from the headers JSONB column in the database.

    Requirement: The underlying table must have a headers JSONB DEFAULT '{}'::jsonb column for this to function.

  9. Understand TigerFS file permissions and ownership

    main

    TigerFS maps PostgreSQL table privileges to standard Unix filesystem permissions. Permissions are checked lazily and cached in memory for the lifetime of the mount.

    Permission Mapping

    PostgreSQL PrivilegeFilesystem PermissionDescription
    SELECTr--Read permission
    UPDATE-w-Write permission on existing rows
    INSERT-w-Write permission on new rows
    DELETErm capabilityAbility to remove files/directories

    Example:

    • A user with SELECT + UPDATE on a users table will see files as -rw-r--r--.
    • A user with only SELECT will see files as -r--r--r--.

    Ownership and Size

    • Ownership: All files are owned by the user running the FUSE daemon.
    • File Size:
      • Column files: Byte length of text representation.
      • Row files: Byte length of the serialized format (TSV/CSV/JSON/YAML).
    • Directories: Size is 0 or 4096 (standard).
  10. Use Cloud Backends with prefix schemes

    main

    TigerFS supports cloud-native backends like Tiger Cloud and Ghost using a URI prefix scheme. Instead of passing traditional database flags, you can use prefixes to identify the backend service.

    Supported Prefixes:

    • tiger:<id>
    • ghost:<id>

    Cloud Commands:

    • create [BACKEND:]NAME [MOUNTPOINT]: Creates a new resource on the specified backend.
    • fork SOURCE [DEST]: Forks an existing resource.
    • info [MOUNTPOINT]: Displays detailed information about a cloud-backed mount.

    Mounting with a prefix: Instead of using --host or --user, pass the prefixed connection string directly to the mount command.

    Example:

    go run ./cmd/tigerfs mount tiger:my-service-id /tmp/cloudmount
  11. Access tables via schema prefixes or the .schemas/ directory

    main

    TigerFS allows you to access tables from different PostgreSQL schemas using two methods:

    1. Schema Prefixes: Use the schema name as a directory prefix in your path.

      • /users/ maps to public.users (default schema).
      • /analytics/reports/ maps to analytics.reports.
    2. Explicit Access via .schemas/: Use the .schemas/ directory to browse all available schemas explicitly.

      • /.schemas/public/users/ maps to public.users.

    The default schema is configurable via the TigerFS configuration.

    # Example: Accessing a table in the 'analytics' schema
    ls /tmp/testmount/analytics/
    
    # Example: Explicit access via .schemas directory
    ls /tmp/testmount/.schemas/public/
  12. Navigate indexed columns via the `.by/` directory

    main

    To prevent column names from colliding with reserved capability names (like .first or .last), all index-based navigation is moved under a .by/ subdirectory. This directory contains subdirectories for every indexed column (single or composite).

    Example Structure:

    • .by/email/: Access via the email index.
    • .by/last_name.first_name/: Access via the composite last_name and first_name index.
    # Example directory structure
    /mnt/db/users/
    ├── .by/
    │   ├── email/           # single-column index
    │   │   └── foo@example.com/
    │   └── last_name.first_name/  # composite index
    │       └── Smith/
    │           └── John/
    ├── .info/
    └── 1/