DOCSEARCHbyAlgolia
Submit Docs

VectorVFS

repository·main·Indexed 19 days ago

https://github.com/perone/vectorvfs

A lightweight Python package (v0.3.0) that transforms a Linux filesystem into a vector database by storing embeddings directly in file extended attributes (xattrs). It enables semantic file retrieval without external databases or indexing services, utilizing native VFS functionality. It supports Meta's Perception Encoders (PE) for vision-language understanding and provides a CLI tool (`vfs search`) for performing similarity searches across directories.

Tokens
2.9K
Snippets
9
Records
15
Agent score
63%

What's inside vectorvfs

  1. Overview of VectorVFS

    main
    VectorVFS is a lightweight Python package that transforms a Linux filesystem into a vector database. It achieves this by leveraging native VFS (Virtual File System) extended attributes (xattrs) to store vector embeddings directly alongside each file. This approach eliminates the need for external databases, separate index files, or background daemons, allowing your existing directory structure to function as a semantically searchable embedding store.

  2. Key features of VectorVFS

    main

    VectorVFS provides the following capabilities:

    • Zero-overhead indexing: Embeddings are stored as extended attributes (xattrs) on each file, removing the need for external index services.
    • Seamless retrieval: Search across your filesystem to retrieve files based on embedding similarity.
    • Flexible embedding support: You can plug in any embedding model, including pre-trained transformers or custom feature extractors.
    • Lightweight and portable: Built on native Linux VFS functionality, it requires no additional daemons or background processes.

  3. How VectorVFS stores embeddings using extended attributes

    main

    VectorVFS integrates vector search capabilities directly into the filesystem by leveraging extended attributes (xattr). Instead of using an external database or modifying file contents, VectorVFS stores data embeddings within the file's metadata (the inode).

    Key Mechanisms:

    • Inode Storage: Embeddings are stored in the inode's reserved space (extra space) rather than in data blocks. This ensures embeddings are tightly coupled with file metadata and minimizes lookup overhead.
    • Handling Size Constraints: Since filesystems like Ext4 impose limits on attribute size (e.g., a 4KB budget), VectorVFS employs several strategies to manage large embeddings:
      • Quantization/Compression: Reducing the precision of embeddings to shrink their size.
      • Splitting: Dividing a single embedding into multiple separate xattr entries.
      • Filesystem Spilling: Allowing the filesystem to automatically move attributes to external "xattr blocks" if they exceed the inode's extra space.

    In its current implementation, VectorVFS stores 1024D embeddings using half-precision to ensure they fit within the standard 4KB budget.

  4. Understanding Inodes and Extended Attributes in VectorVFS

    main

    To use VectorVFS effectively, it is important to understand that it operates at the filesystem metadata layer.

    An inode (index node) is a filesystem object that stores metadata such as file size, timestamps, permissions, and pointers to data blocks. VectorVFS specifically utilizes the extended attributes (xattr) field within the inode.

    Storage Behavior in Ext4:

    • Small Attributes: Stored directly within the inode's "extra space" (the area following standard fields in larger inode sizes, e.g., 512 or 1024 bytes).
    • Large Attributes: If the embedding exceeds the available inode extra space, Ext4 stores them in separate xattr blocks on disk and maintains a pointer in the inode. VectorVFS is designed to work with both scenarios to maintain seamless integration.

  5. Supported embedding models in VectorVFS

    main
    VectorVFS currently uses Meta's Perception Encoders (PE), which includes image and video encoders for vision-language understanding. This model is noted for outperforming InternVL3, Qwen2.5VL, and SigLIP2 in zero-shot image tasks. The project is designed to support additional models in the future.

  6. Search files using the `vfs search` command

    main

    The vfs search command allows you to perform vector searches directly against a directory. It automatically iterates over files in the specified folder, identifies supported file types, and either generates new embeddings or loads existing ones from the filesystem.

    Usage: vfs search <query> <path>

    Example: To search for images containing cats in /my_folder:

    $ vfs search cat /my_folder

  8. Implement or use the DualEncoder interface

    main

    The DualEncoder is an abstract base class (ABC) that defines the interface for models capable of encoding both vision (images) and text into a shared embedding space. This is useful for multimodal retrieval tasks where images and text need to be compared using similarity metrics.

    To implement a custom dual encoder, you must provide implementations for:

    • encode_vision(file: Path) -> torch.Tensor
    • encode_text(text: str) -> torch.Tensor
    • logit_scale() -> torch.Tensor
    from abc import ABC, abstractmethod
from pathlib import Path
import torch

class DualEncoder(ABC):
    @abstractmethod
    def encode_vision(self, file: Path) -> torch.Tensor:
        pass

    @abstractmethod
    def encode_text(self, text: str) -> torch.Tensor:
        pass

    @abstractmethod
    def logit_scale(self) -> torch.Tensor:
        pass

  9. Configure `vfs search` options

    main

    When using the vfs search command, you can use the following flags to control the search behavior:

    • -f (or --force): Forces VectorVFS to re-index the files. Use this if you want to ensure changes in files are captured, bypassing the automatic change detection.
    • -n <number>: Limits the number of results returned. For example, to show only the top 3 most similar files, use -n 3.
    # Force re-indexing
$ vfs search -f cat /my_folder

# Limit to top 3 results
$ vfs search -n 3 cat /my_folder

  11. Use PerceptionEncoder for CLIP-based embeddings

    main

    The PerceptionEncoder is a concrete implementation of DualEncoder that uses a CLIP-based model to generate embeddings. It automatically handles device placement (CUDA if available, otherwise CPU) and includes necessary preprocessing and tokenization.

    Initialization

    Initialize with a model_name. The default is "PE-Core-L14-336".

    Methods

    • encode_vision(file: Path): Opens an image file via PIL, applies necessary transforms, and returns a torch.Tensor of image features.
    • encode_text(text: str): Tokenizes the input string and returns a torch.Tensor of text features.
    • logit_scale(): Returns the exponential of the model's logit scale parameter, used for scaling similarity scores during computation.
    from vectorvfs.encoders import PerceptionEncoder
from pathlib import Path

# Initialize the encoder
encoder = PerceptionEncoder(model_name="PE-Core-L14-336")

# Encode an image
image_features = encoder.encode_vision(Path("path/to/image.jpg"))

# Encode text
text_features = encoder.encode_text("a photo of a cat")

# Get logit scale for similarity
scale = encoder.logit_scale()

  12. Reference: `vfs search` command options and arguments

    main

    The vfs search command uses the following arguments and options:

    TypeNameDescription
    ArgumentqueryThe text string to search for.
    ArgumentpathThe directory path to search (must exist and be a directory).
    Option-n, --num <int>Number of results to return (default: 5).
    Option-f, --force-reindexIf set, forces re-indexing of files even if they already have vector data.
    Option-r, --recursiveIf set, performs a recursive search through subdirectories.