pyleak

repository·main·Indexed 18 days ago

https://github.com/deepankarm/pyleak

A tool for detecting leaked asyncio tasks, threads, and event loop blocking in Python applications, inspired by Go's goleak. It provides context managers and decorators like no_task_leaks, no_thread_leaks, and no_event_loop_blocking to identify resource leaks and synchronous code blocking the asyncio event loop, offering detailed stack traces and configurable actions (warn, log, cancel, raise) for debugging and automated testing with pytest.

Tokens
9.7K
Snippets
32
Records
40
Agent score
59%

What's inside pyleak

  1. Get stack traces for leaked tasks and event loop blocks

    main

    Pyleak provides detailed stack traces to help locate the source of leaks or blocking:

    • Asyncio Tasks: no_task_leaks provides the current stack trace of the leaked task. You can also include the creation stack trace by setting enable_creation_tracking=True.

      • Warning: enable_creation_tracking monkey patches asyncio.create_task and is not recommended for production.
      • The TaskLeakError exception has a leaked_tasks attribute containing LeakedTask objects.
    • Event Loop Blocks: no_event_loop_blocking provides the stack trace of the code that blocked the loop, along with the duration and timestamp.

    # Example: Getting creation stack trace for tasks
    async with no_task_leaks(action="raise", enable_creation_tracking=True):
        await leaky_function()
    
    # Example: Catching EventLoopBlockError
    try:
        async with no_event_loop_blocking(action="raise"):
            await some_blocking_code()
    except EventLoopBlockError as e:
        print(e)
  2. Quick Start for the PDF Ingest Demo

    main

    To run the event loop detection demo, including the MinIO dependency and load tests, follow these steps:

    1. Synchronize dependencies: Use uv sync to prepare the environment.
    2. Start infrastructure: Use docker-compose up -d to start MinIO.
    3. Run tests: Execute uv run pytest tests/ -v -s to run the detection tests.
    4. Run load test:
      • Start the application: uv run uvicorn pdf_ingest:app --reload
      • In a separate terminal, run the load script: uv run python scripts/load_test.py scripts/sample.pdf 100 (where 100 is the number of concurrent requests).
    uv sync
    docker-compose up -d
    
    # Run tests
    uv run pytest tests/ -v -s
    
    # Run load test
    uv run uvicorn pdf_ingest:app --reload
    uv run python scripts/load_test.py scripts/sample.pdf 100
  3. Install the pyleak pytest plugin

    main

    Install pyleak via pip to use its pytest plugin, which automatically wraps tests with detectors for asyncio task leaks, thread leaks, and event loop blocking based on pytest markers.

    pip install pyleak
  4. Configure the pyleak pytest marker

    main

    To use the no_leaks marker, you must register it in your pytest configuration file. You can do this in pyproject.toml, pytest.ini, or via conftest.py.

    # pyproject.toml
    [tool.pytest.ini_options]
    markers = [
        "no_leaks: detect asyncio task leaks, thread leaks, and event loop blocking"
    ]
    # pytest.ini
    [tool:pytest]
    markers = no_leaks: detect asyncio task leaks, thread leaks, and event loop blocking
    # conftest.py
    import pytest
    
    def pytest_configure(config):
        config.addinivalue_line(
            "markers", 
            "no_leaks: detect asyncio task leaks, thread leaks, and event loop blocking"
        )
  5. Filter leaked resources by name

    main

    You can restrict leak detection to specific resources by providing a name_filter. This filter is applied to the resource's name during the detection phase.

    Supported filter types:

    • String: Performs an exact equality match (resource_name == name_filter).
    • Regex Pattern: Uses re.Pattern to search for matches within the resource name.
    • String-like pattern: If a string is provided that is not an exact match, pyleak attempts to compile it as a regex. If compilation fails, it falls back to an exact string match.
  6. Configuration options for detectors

    main

    no_task_leaks options

    • action: String ("warn", "log", "cancel", "raise").
    • name_filter: String or regex pattern to filter tasks.
    • logger: Custom logger instance.

    no_thread_leaks options

    • action: String ("warn", "log", "raise").
    • name_filter: String or regex pattern to filter threads.
    • logger: Custom logger instance.
    • exclude_daemon: Boolean (default True). If True, daemon threads are ignored.

    no_event_loop_blocking options

    • action: String ("warn", "log", "raise").
    • logger: Custom logger instance.
    • threshold: Float (seconds). Minimum blocking time to report.
    • check_interval: Float (seconds). Frequency of checks.
  7. Monitor tests for leaks using the @pytest.mark.no_leaks marker

    main

    The pyleak pytest plugin allows you to automatically detect leaked asyncio tasks, threads, and event loop blocking by applying the @pytest.mark.no_leaks marker to your test functions.

    By default, if you use @pytest.mark.no_leaks without arguments, it monitors for all types of leaks (tasks, threads, and blocking). You can also specify exactly what to monitor using positional arguments or keyword arguments.

    Supported monitoring modes:

    • tasks: Detects leaked asyncio tasks.
    • threads: Detects leaked threads.
    • blocking: Detects event loop blocking.
    • all: Enables all three detectors (default behavior when no arguments are provided).
    import pytest
    
    # Monitor everything (tasks, threads, and blocking)
    @pytest.mark.no_leaks
    def test_everything():
        pass
    
    # Monitor only asyncio tasks
    @pytest.mark.no_leaks("tasks")
    def test_only_tasks():
        pass
    
    # Monitor tasks and threads using keyword arguments
    @pytest.mark.no_leaks(tasks=True, threads=True)
    def test_specific_leaks():
        pass
  8. Detect asyncio task leaks with `no_task_leaks`

    main

    Use the no_task_leaks context manager or decorator to ensure no asyncio tasks are left running after a specific block of code or function completes. This is useful for identifying tasks that were started but never awaited or cancelled.

    Usage Modes:

    • Async Context Manager: Wrap an async block with async with no_task_leaks():.
    • Decorator: Apply @no_task_leaks to an async def function.

    Configuration Options:

    • action: What to do when a leak is found. Options include LeakAction.WARN, LeakAction.LOG, LeakAction.CANCEL, or LeakAction.RAISE (or their string equivalents).
    • name_filter: A string or regex pattern to filter tasks by name.
    • logger: A logging.Logger instance to use for reporting.
    • enable_creation_tracking: If True, enables automatic tracking of task creation stacks by enabling asyncio debug mode.
    # As a context manager
    async with no_task_leaks():
        await some_async_function()
    
    # As a decorator
    @no_task_leaks
    async def my_function():
        await some_async_function()
  9. Use pyleak in pytest

    main

    Pyleak is ideal for catching leaks during automated testing. Use the detectors as context managers within your test functions, typically setting action="raise" to ensure tests fail when a leak is detected.

    import pytest
    from pyleak import no_task_leaks, no_thread_leaks, no_event_loop_blocking
    
    @pytest.mark.asyncio
    async def test_no_leaked_tasks():
        async with no_task_leaks(action="raise"):
            await my_async_function()
    
    def test_no_leaked_threads():
        with no_thread_leaks(action="raise"):
            my_threaded_function()
    
    @pytest.mark.asyncio        
    async def test_no_event_loop_blocking():
        async with no_event_loop_blocking(action="raise", threshold=0.1):
            await my_potentially_blocking_function()
  10. Debug complex task leaks with TaskLeakError

    main

    When no_task_leaks(action="raise") detects leaked tasks, it raises a TaskLeakError. You can inspect this error to find exactly which tasks leaked by accessing:

    • e.task_count: The number of leaked tasks.
    • e.leaked_tasks: A list of task information objects.

    For each leaked task, you can use:

    • task_info.name: The name of the task.
    • task_info.format_current_stack(): A string representation of the stack where the task is currently executing.
    • task_info.format_creation_stack(): A string representation of the stack where the task was originally created.
    • task_info.task_ref: A reference to the actual asyncio.Task object, allowing you to call .cancel() on it.

    You can also use enable_creation_tracking=True and name_filter (a compiled regex) in no_task_leaks to narrow down the search to specific types of tasks.

    import asyncio
    import random
    import re
    from pyleak import TaskLeakError, no_task_leaks
    
    async def debug_task_leaks():
        """Example showing how to debug complex task leaks."""
    
        async def worker(worker_id: int, sleep_time: int):
            print(f"Worker {worker_id} starting")
            await asyncio.sleep(sleep_time)  # Simulate work
            print(f"Worker {worker_id} done")
    
        async def spawn_workers():
            for i in range(3):
                asyncio.create_task(worker(i, random.randint(1, 10)), name=f"worker-{i}")
    
        try:
            async with no_task_leaks(
                action="raise",
                enable_creation_tracking=True,
                name_filter=re.compile(r"worker-\d+"),  # Only catch worker tasks
            ):
                await spawn_workers()
                await asyncio.sleep(0.1)  # Let workers start
    
        except TaskLeakError as e:
            print(f"\nFound {e.task_count} leaked worker tasks:")
            for task_info in e.leaked_tasks:
                print(f"\n--- {task_info.name} ---")
                print("Currently executing:")
                print(task_info.format_current_stack())
                print("Created at:")
                print(task_info.format_creation_stack())
    
                # Cancel the leaked task
                if task_info.task_ref:
                    task_info.task_ref.cancel()
    
    
    if __name__ == "__main__":
        asyncio.run(debug_task_leaks())
  11. Ensure proper AsyncIO task cleanup

    main

    Use no_task_leaks to ensure that all spawned asyncio tasks are properly cleaned up (e.g., cancelled or awaited) before the context manager exits. Setting action="raise" will cause the context manager to raise a TaskLeakError if any tasks are still running, making it ideal for testing background task management.

    async def test_background_task_cleanup():
        async with no_task_leaks(action="raise"):
            # This would fail the test
            asyncio.create_task(long_running_task())
            
            # This would pass
            task = asyncio.create_task(long_running_task())
            task.cancel()
            try:
                await task
            except asyncio.CancelledError:
                pass