DOCSEARCHbyAlgolia
Submit Docs

PyFilesystem2 Documentation

repository·master·Indexed 24 days ago

https://github.com/pyfilesystem/pyfilesystem2

A filesystem abstraction layer for Python that provides a consistent API to interact with various storage backends, including local disks, ZIP files, and FTP servers. It features platform-independent paths, sandboxing, a unified exception hierarchy, and support for custom filesystem extensions via Openers. Key functionality includes recursive directory traversal with fs.walk, globbing for batch operations, and SubFS for managing sub-directories.

Tokens
8.7K
Snippets
22
Records
63
Agent score
81%

What's inside PyFilesystem2

  1. What is PyFilesystem2?

    master
    PyFilesystem2 is a Python filesystem abstraction layer. It provides FS objects that abstract an entire filesystem, similar to how Python's built-in file objects abstract a single file. This allows you to write code that is independent of where and how files are physically stored (e.g., local disk, ZIP files, or FTP servers) by using a consistent API.

  3. Handle filesystem errors with a common hierarchy

    master
    PyFilesystem converts various underlying filesystem errors into a unified exception hierarchy defined in the fs.errors module. This allows you to write error-handling code that works consistently across different types of filesystems (e.g., local OS, S3, MemoryFS) without needing to catch platform-specific exceptions.

  5. Use a Sub Filesystem to access a specific directory

    master

    A Sub Filesystem allows you to treat a specific directory within an existing filesystem as if it were the root directory itself. This is useful for scoping operations to a particular subdirectory and simplifying path management when working with nested structures.

    You can create a sub filesystem using fs.subfs.subfs(filesystem, path). The resulting object implements the same Filesystem interface as the original, but all operations are relative to the specified path.

  6. Understand PyFilesystem namespaces

    master

    Resource information in PyFilesystem is organized into logical key/value groups called namespaces. When calling getinfo(), the filesystem always returns the basic namespace, but you must explicitly request others via the namespaces argument.

    Basic Namespace (Always included)

    Contains core identity information:

    • name (str): Name of the resource.
    • is_dir (bool): Whether the resource is a directory.

    Details Namespace

    Contains file metadata:

    • accessed (datetime): Last access time.
    • created (datetime): Creation time.
    • metadata_changed (datetime): Last metadata change time.
    • modified (datetime): Last data change time.
    • size (int): Size in bytes.
    • type (ResourceType): The resource type.

    Access Namespace (Optional)

    Reports ownership and permissions (e.g., supported by OSFS):

    • gid (int): Group ID.
    • group (str): Group name.
    • permissions (Permissions): Permissions object.
    • uid (int): User ID.
    • user (str): Owner username.

    Other Namespaces

    • stat: Information from os.stat (OS-mapped filesystems).
    • lstat: Information from os.lstat (OS-mapped filesystems).
    • link: Symlink information (contains target path).

    Note: Requesting a namespace that a filesystem does not support is not an error; the unknown namespace will simply be ignored.

  7. How MultiFS works as an overlay filesystem

    master

    A MultiFS is a filesystem composed of a sequence of other filesystems. The directory structure of each filesystem in the sequence overlays the previous ones. This allows you to selectively override files or customize behavior by providing a 'higher priority' filesystem that masks files in 'lower priority' filesystems.

    When multiple filesystems in the sequence contain the same path, the version from the filesystem added most recently (or the one appearing later in the sequence, depending on implementation details of the overlay) is what the user sees. In the provided example, the theme directory is used to override or extend the templates directory.

    from fs.osfs import OSFS
from fs.multifs import MultiFS

theme_fs = MultiFS()
theme_fs.add_fs('templates', OSFS('templates'))
theme_fs.add_fs('theme', OSFS('theme'))

  8. How sandboxing works in PyFilesystem

    master

    PyFilesystem enforces sandboxing by preventing any operation from accessing files outside of the filesystem's root. If you attempt to use a backreference (like ../) to escape the filesystem instance, a fs.errors.IllegalBackReference exception is thrown.

    Important: Sandboxing only applies if you use the PyFilesystem interface. It does not prevent standard OS-level file manipulation if you bypass the FS object.

    To work within a specific sub-directory without exposing the entire filesystem, use the opendir() method. This creates a new FS object where the specified sub-directory becomes the new root.

  9. Choose between breadth and depth search algorithms

    master

    When walking a filesystem, you can specify the search algorithm using the search parameter in most Walker methods:

    • "breadth" (default): Yields resources at the top of the directory tree first before moving to sub-directories. This is generally more efficient for searching.
    • "depth": Yields the most deeply nested resources first, working backwards to the top. This is recommended if you intend to delete resources as you walk through them.

  11. Open a filesystem with open_fs()

    master

    Use fs.open_fs() to create an FS object for various storage backends. The backend is determined by the URL scheme provided in the connection string. This allows the same code to operate on local directories, compressed archives, or remote servers without modification.

    Common connection string patterns:

    • Local directory: '~/projects'
    • ZIP file: 'zip://projects.zip'
    • FTP server: 'ftp://ftp.example.org/projects'

  12. Format of FS URLs

    master

    FS URLs follow this structure:

    <protocol>://<username>:<password>@<resource>

    Components:

    • <protocol>: Identifies the filesystem type (e.g., osfs, ftp, mem).
    • <username>: (Optional) Username.
    • <password>: (Optional) Password.
    • <resource>: A domain, path, or both.

    Note: Usernames and passwords must be percent-encoded if they contain colons (:) or @ symbols.

    osfs://~/projects
osfs://c://system32
ftp://ftp.example.org/pub
mem://
ftp://will:daffodil@ftp.example.org/private