DOCSEARCHbyAlgolia
Submit Docs

FawltyDeps

repository·main·Indexed 43 days ago

https://github.com/tweag/fawltydeps

A dependency checker for Python projects (v3.9 through v3.13) designed to identify undeclared and unused 3rd-party dependencies. It analyzes the Abstract Syntax Tree (AST) of Python code and Jupyter Notebooks to find packages that are imported but not listed in requirements, or declared in requirements but never imported.

Tokens
18.6K
Snippets
56
Records
98
Agent score
53%

What's inside FawltyDeps

  1. What is FawltyDeps?

    main

    FawltyDeps is a dependency checker for Python designed to identify discrepancies between your code's imports and your project's declared dependencies. It specifically detects:

    • Undeclared dependencies: Modules that are imported in your Python code (including regular .py files and Jupyter notebooks) but are missing from your dependency declaration files.
    • Unused dependencies: Dependencies that are declared in your configuration files but are never actually imported anywhere in your project.

    By using FawltyDeps, you can keep your environments lean, reduce installation times, and avoid the 'Works on my machine' problem caused by missing declarations.

  2. What is FawltyDeps and what problem does it solve?

    main

    FawltyDeps is a dependency checker for Python designed to ensure that all packages imported in your code are explicitly declared in your project's dependency files (such as requirements.txt, pyproject.toml, or setup.py).

    In Python, there is a gap between imports (what the code uses) and dependencies (what the project declares). A common issue is that code may run successfully in a local environment because a package was installed manually, but the project will fail in a new environment (like CI/CD or a user's machine) because that package was never formally declared.

    FawltyDeps addresses this by:

    • Identifying unmatched imports: Packages used in code but not declared as dependencies (results in an error).
    • Identifying unused dependencies: Packages declared as dependencies but not used in the code (results in a warning).

    Note: FawltyDeps focuses on name matching and package existence/matching. It does not automatically generate requirements files, resolve version compatibility, or handle transitive dependencies.

  3. Supported Python environments and platforms

    main

    FawltyDeps is designed to analyze Python projects without requiring them to run in the same environment as the tool itself.

    • Python Versions: Currently supports Python v3.9 through v3.13.
    • Interpreter: Supports CPython only.
    • Operating Systems: Primarily optimized for Linux. It is expected to run on other Unix-like systems (notably macOS), but Windows support is not currently guaranteed.

  4. How FawltyDeps detects imports

    main

    FawltyDeps identifies imports in Python source code (.py) and Jupyter Notebooks (.ipynb) by analyzing the Abstract Syntax Tree (AST). It looks for specific AST nodes that represent different import styles:

    • Absolute imports: ast.Import (e.g., import packageX)
    • Standard imports: ast.ImportFrom (e.g., from packageX import Y)
    • Aliases: ast.alias (e.g., import packageX as PX)
    • Function calls: ast.Expr containing calls to __import__ or importlib.import_module()

    Note that relative imports (starting with a dot, e.g., from .packageX import Y) are also handled during the parsing process.

  5. Understand the FawltyDeps configuration cascade

    main

    FawltyDeps resolves configuration settings using a specific hierarchy. If a setting is defined in multiple places, the higher-precedence source wins:

    1. Command-line options: Highest precedence; overrides everything else.
    2. Environment variables: Overrides pyproject.toml settings.
    3. pyproject.toml ([tool.fawltydeps]): Overrides hardcoded defaults.
    4. Hardcoded defaults: The fallback values used if no other configuration is provided.

  6. Understand the FawltyDeps class hierarchy and data model

    main

    FawltyDeps uses a layered architecture of dataclasses to represent dependency analysis results. The hierarchy is structured from low-level location data to high-level analysis results.

    Immutability Pattern: The project follows a bottom-up approach to immutability. Low-level objects like Location are immutable (frozen=True), while high-level objects like Analysis are mutable because they are constructed incrementally during the core logic execution.

    The Layers:

    • Level 4: Analysis (Top level, mutable, contains the final results)
    • Level 3: UndeclaredDependency and UnusedDependency (Contains lists of Level 2 objects)
    • Level 2.5: Package
    • Level 2: ParsedImport and DeclaredDependency (Immutable, constructed in single operations)
    • Level 1: Location (Bottom level, immutable value object)

    When implementing or extending these classes, if a dataclass is made immutable (frozen=True) and acts as a value object, it is recommended to also use eq=True and order=True where natural ordering exists.

  7. Match import statements with declared dependencies

    main

    FawltyDeps matches import statements to declared packages using several mechanisms:

    • --pyenv: Specifies where to search for Python environments (e.g., virtualenvs) containing installed dependencies. This is the most accurate way to match import names to package names.
    • --custom-mapping-file: A TOML file containing user-defined mappings of dependencies to import names. This takes precedence over installed package lookups.
    • --install-deps: Automatically installs declared dependencies into a temporary virtualenv to discover the imports they expose. Use this if no local environment with dependencies is available.

  8. What are undeclared and unused dependencies?

    main

    FawltyDeps identifies two specific types of dependency issues:

    • undeclared dependency: A package that is used (e.g., via import) in your code but is missing from your project's dependency declarations (like requirements.txt). This can lead to runtime errors.
    • unused dependency: A package that is declared in your project's requirements but is never actually imported or used in your code. These increase installation size and the surface area for potential breakage without providing any benefit.

  9. Mapping strategies for dependency checking

    main

    FawltyDeps considers three primary ways to map imports to dependencies, which impact accuracy and connectivity requirements:

    1. Identity Mapping

    Assumes the import name is identical to the library name (e.g., import numpy requires numpy in requirements.txt).

    • Pros: Extremely fast, works entirely offline, and is a lightweight check for the majority of libraries.
    • Cons: Fails when names differ (e.g., import googleapiclient vs google-api-python-client).
    • Extension: Supports "relaxed equality" (handling punctuation and snake_case vs kebab-case).

    2. Import to Dependency Mapping

    Attempts to find which library provides a specific import name.

    • Pros: Useful for generating requirements.txt files.
    • Cons: Difficult to implement reliably. Requires searching PyPI (which lacks structured import metadata) or downloading/unpacking libraries to inspect them. Often requires an internet connection or a large, potentially stale offline database.

    3. Dependency to Import Mapping

    Maps declared dependencies back to the import names they expose.

    • Pros: Conceptually aligned with checking existing projects. It can be done on-demand by downloading and unpacking only the dependencies declared in the project. It is not limited to PyPI libraries.
    • Cons: Requires access to dependency repositories (PyPI, GitHub, etc.) unless searching within a local environment.

  10. Local environment mapping vs. Online mapping

    main

    When performing dependency mapping, FawltyDeps can operate in two modes:

    • Local Environment Mapping: The tool searches through libraries already installed in the current Python environment to create the necessary mappings. This is ideal for restricted network environments (CI/CD) and allows for entirely offline operation.
    • Online Mapping: The tool reaches out to public repositories (like PyPI or GitHub) to download and unpack dependencies to inspect their exposed packages. This provides broader coverage but requires internet access and may be subject to network restrictions.

  11. How FawltyDeps maps imports to dependencies

    main

    FawltyDeps does not rely on a simple 1:1 name match. To handle cases where the package name (e.g., PyYAML) differs from the import name (e.g., yaml), or where one package provides multiple imports (e.g., setuptools providing pkg_resources), FawltyDeps inspects installed packages in a Python environment.

    It searches for dependencies in this order:

    1. The environment specified via the --pyenv option.
    2. If --pyenv is not provided, any Python environment found within your project (the basepath or the current directory).
    3. The current Python environment in which FawltyDeps itself is running.

    If no environment can be used to verify a mapping, FawltyDeps falls back to an identity mapping (assuming the import name is identical to the package name).

  12. How FawltyDeps resolves dependencies

    main

    To match import statements with declared dependencies, FawltyDeps uses a prioritized sequence of mapping strategies (resolvers). If a higher-priority strategy fails to resolve a dependency, it falls back to the next one in the sequence.

    Resolver Priority Order:

    1. User-defined mapping: Custom mappings provided via CLI or pyproject.toml.
    2. Local Python environment mapping: Packages found in local virtualenvs (auto-discovered or specified via --pyenv).
    3. Current Python environment: Packages installed in the environment where FawltyDeps is running.
    4. Fallback Strategy (Mutually exclusive):
      • 4a. Mapping via temporary installation: Automatically installs missing dependencies into a temporary virtualenv (activated via --install-deps).
      • 4b. Identity mapping: The default fallback; assumes the dependency name matches the import name (e.g., some_package $\rightarrow$ import some_package).