DOCSEARCHbyAlgolia
Submit Docs

universal_pathlib

repository·main·Indexed 19 days ago

https://github.com/fsspec/universal_pathlib

A pathlib-compatible interface for interacting with various filesystems (S3, GCS, Azure, local, etc.) using fsspec backends. It provides the UPath class, which implements most standard pathlib.Path methods and backports newer Python version features, allowing developers to use standard path manipulation syntax for remote cloud storage.

Tokens
22.6K
Snippets
60
Records
79
Agent score
63%

What's inside universal_pathlib

  1. What is fsspec and its core functionality

    main

    fsspec is a Python library that provides a unified, pythonic interface for working with different storage backends (local, cloud, remote, etc.). It abstracts the differences between storage systems so you can use a consistent API regardless of the backend.

    All filesystem implementations inherit from fsspec.spec.AbstractFileSystem, which defines a standard interface for common operations like listing files, checking existence, reading, writing, copying, and deleting.

  2. How pathlib-abc abstract base classes work

    main

    The pathlib-abc library provides abstract base classes (ABCs) that define formal interfaces for path-like objects. This allows developers to create custom path implementations (like those for cloud storage) that follow standard pathlib conventions.

    There are three primary levels of abstraction:

    1. JoinablePath: The base level. Handles path construction, string manipulation, component access (name, stem, suffix), and joining paths using the / operator. Equivalent to PurePath.
    2. ReadablePath: Extends JoinablePath. Adds read-only filesystem operations such as .exists(), .is_file(), .iterdir(), .glob(), and reading file contents via .read_text() or .read_bytes().
    3. WritablePath: Extends JoinablePath (but not ReadablePath). Adds write operations such as .write_text(), .mkdir(), and .symlink_to().

    Note: A path that is WritablePath is not automatically a ReadablePath. For full filesystem access (like UPath), a class must inherit from both.

  3. How Universal Pathlib combines fsspec and pathlib

    main

    Universal Pathlib provides a unified, Pythonic interface for file operations across diverse storage systems by integrating two core components:

    1. fsspec: Acts as the foundation, providing a specification and various implementations for accessing local storage, cloud services (like S3, GCS), and remote systems.
    2. pathlib: Provides the familiar object-oriented API standard in Python for filesystem path manipulation.
    3. Universal Pathlib (upath): The bridge that implements the pathlib-abc interface on top of fsspec filesystems. This allows you to use standard pathlib-style methods (like .exists(), .mkdir(), .glob()) on files located anywhere, not just on your local machine.

  4. Understand the difference between Pure and Concrete paths

    main

    In the pathlib ecosystem, paths are categorized into two types based on whether they interact with the physical filesystem:

    Pure Paths (PurePath, PurePosixPath, PureWindowsPath)

    • Purpose: Purely for string-based path manipulation.
    • Behavior: They do not access the filesystem. Operations like .exists() will raise an AttributeError.
    • Use Case: Use these when you need to manipulate path strings (e.g., getting a parent directory or changing a suffix) without performing I/O, or when working with paths for a different OS (e.g., using PureWindowsPath on Linux).

    Concrete Paths (Path, PosixPath, WindowsPath)

    • Purpose: For actual filesystem operations.
    • Behavior: They inherit from pure paths but add the ability to interact with the OS.
    • Use Case: Use these when you need to check if a file exists, read/write content, or get file metadata.
    from pathlib import PurePath, Path

# Pure path: string manipulation only
pure = PurePath("/home/user/file.txt")
print(pure.parent)  # Works
# pure.exists()    # Raises AttributeError

# Concrete path: filesystem operations
concrete = Path("/home/user/file.txt")
if concrete.exists():
    print(concrete.read_text())

  5. Unified filesystem access with UPath

    main

    Universal Pathlib provides a single, unified interface (UPath) that brings the pathlib.Path API to any filesystem supported by fsspec. This allows you to interact with local files, S3, Azure Blob Storage, Google Cloud Storage, and more using the same method calls, regardless of the underlying storage backend.

    Instead of using service-specific libraries (like boto3 for S3 or azure-storage-blob for Azure), you can use UPath with URL-style paths to perform file operations.

    from upath import UPath

# Local files
local_file = UPath("data/results.csv")

# S3 files
s3_file = UPath("s3://my-bucket/data/results.csv")

# Azure Blob Storage
azure_file = UPath("az://my-container/data/results.csv")

# The same API works for all
for path in [local_file, s3_file, azure_file]:
    with path.open('r') as f:
        data = f.read()

  6. How chained filesystems work

    main

    fsspec supports composing filesystems using the :: separator. This allows you to treat one filesystem as the target for another (e.g., accessing a file inside an archive that is itself stored on a remote cloud provider).

    Syntax: [inner_protocol]://[inner_path]::[outer_protocol]://[outer_path]

    import fsspec

# Access a file inside a ZIP archive on S3
with fsspec.open('zip://data.csv::s3://bucket/archive.zip', 'r', anon=True) as f:
    content = f.read()

# Read a compressed file
with fsspec.open('tar://file.txt::s3://bucket/archive.tar', 'r', anon=True) as f:
    content = f.read()

  7. How UPath instantiation and protocol detection works

    main

    When you call UPath(...), the UPath.__new__() method determines the path protocol and returns a registered implementation from upath.registry.

    1. Registered Implementations: If a protocol is registered (e.g., s3 maps to S3Path), UPath returns that specific subclass.
    2. Fsspec Fallback: If the protocol is not registered but is mapped to an fsspec filesystem, UPath returns a standard UPath instance that provides access via the fsspec filesystem.
    3. Default Behavior: If no registration or fsspec mapping exists, a default UPath instance is returned (and a warning may be emitted).

    The protocol is determined by the URI scheme of the first argument or by explicitly passing the protocol keyword argument.

    from upath import UPath
from upath.implementations.cloud import S3Path
from upath.implementations.memory import MemoryPath

# Protocol detected from URI scheme
p0 = UPath("s3://bucket/file.txt")
assert type(p0) is S3Path

# Protocol explicitly provided via keyword
p1 = UPath("/some/path/file.txt", protocol="memory")
assert type(p1) is MemoryPath

# Fallback to default UPath for unknown protocols
p2 = UPath("ftp://ftp.ncbi.nih.gov/snp/archive")
assert type(p2) is UPath

  8. Understand path interfaces via pathlib-abc base classes

    main

    Universal Pathlib uses base classes and protocols re-exported from pathlib-abc to define core path interfaces. These interfaces ensure that both the standard library pathlib and UPath implementations conform to a consistent set of behaviors for joining, reading, writing, and parsing paths.

    Key interfaces include:

    • JoinablePath: For paths that support joining segments.
    • ReadablePath: For paths that support read operations.
    • WritablePath: For paths that support write operations.
    • PathInfo: For objects providing metadata about a path.
    • PathParser: For objects capable of parsing path strings.

  9. Local Path Compatibility with UPath

    main

    When working with local filesystems, UPath behaves in two distinct ways depending on how you initialize it:

    1. Without a protocol: If you provide a standard path string (e.g., /home/user/file.txt), UPath returns a platform-specific implementation (PosixUPath or WindowsUPath). These are 100% compatible with the standard library pathlib.Path and implement os.PathLike.
    2. With the file:// protocol: If you use the file:// prefix (e.g., file:///home/user/file.txt), UPath returns a FilePath. FilePath is a UPath subclass that uses fsspec's LocalFileSystem instead of the standard library's local path logic. This is useful for maintaining consistent fsspec-based access patterns across all your code.

    Summary Table

    InputResulting ClassBackendCompatible with pathlib.Path?
    /path/to/filePosixUPath / WindowsUPathOS NativeYes
    file:///path/to/fileFilePathfsspecNo (but implements os.PathLike)
    from pathlib import Path, PosixPath, WindowsPath
from upath import UPath

# Without protocol -> returns platform-specific UPath
local = UPath("/home/user/file.txt")
assert isinstance(local, UPath)           # True
assert isinstance(local, PosixPath)       # True (on Unix systems)
assert isinstance(local, Path)            # True

# With file:// protocol -> returns FilePath (fsspec-based)
file_path = UPath("file:///home/user/file.txt")
assert isinstance(file_path, UPath)       # True
assert not isinstance(file_path, Path)    # False (uses fsspec instead)

  10. How UPath and Path relate via pathlib-abc

    main

    The universal-pathlib library (imported as upath) bridges Python's pathlib API with fsspec filesystem implementations. It uses pathlib-abc (abstract base classes) to provide a unified interface for both local and remote filesystems.

    Core Abstractions

    • JoinablePath: Basic path manipulation (e.g., joining paths) without filesystem access.
    • ReadablePath: Adds read-only filesystem operations (e.g., .read_text(), .exists()).
    • WritablePath: Adds write filesystem operations (e.g., .write_text()).
    • UPath: The primary universal path class that implements these interfaces for any backend (S3, GCS, HTTP, etc.).

    Key Differences from stdlib pathlib.Path

    Unlike the standard library pathlib.Path, UPath explicitly implements the pathlib-abc interfaces. This allows for formal type hinting and consistent behavior across different storage backends.

    from pathlib import Path
from upath import UPath
from upath.types import JoinablePath, ReadablePath, WritablePath

# UPath explicitly implements pathlib-abc
path = UPath("s3://bucket/file.txt")
assert isinstance(path, JoinablePath)  # True
assert isinstance(path, ReadablePath)   # True
assert isinstance(path, WritablePath)   # True

# pathlib.Path does NOT (yet) inherit from pathlib-abc
local = Path("/home/user/file.txt")
assert isinstance(local, JoinablePath)  # False

  11. Understand the UPath compatibility with stdlib pathlib

    main

    Universal Pathlib's UPath class aims to provide a consistent pathlib.Path interface across different Python versions. While it implements most standard pathlib methods, some methods are only available in newer Python versions (like parser or full_match) and are backported to UPath.

    Important Note on Compatibility: Some methods that are standard in pathlib might raise an unsupported error in certain UPath implementations (marked with ⚠️ in the compatibility matrix), such as readlink(), symlink_to(), or replace(). Always check the compatibility matrix if your code relies on specific filesystem operations like permission changes (chmod) or hardlinking.

  12. How UPath instantiation and types work

    main

    When you instantiate UPath, the returned instance type is determined by the protocol provided in the path string.

    • Local paths (no protocol): Returns PosixUPath or WindowsUPath. These are 100% compatible with Python's pathlib.Path and are subclasses of pathlib.Path.
    • Local URL paths (file:// or local://): Returns a FilePath instance, which uses fsspec's LocalFileSystem.
    • Remote protocols: Returns specialized implementations like S3Path or HttpPath. If no specialized implementation exists but the protocol is supported by fsspec, it returns a generic UPath instance.

    Note that while UPath implements pathlib_abc.JoinablePath, standard pathlib.Path objects do not.

    from pathlib import Path
from upath import UPath
from upath.types import JoinablePath

# Standard pathlib does NOT implement JoinablePath
assert isinstance(Path(), JoinablePath) is False

# UPath DOES implement JoinablePath
assert isinstance(UPath(), JoinablePath) is True