aiohttp-retry

repository·master·Indexed 42 days ago

https://github.com/inyutin/aiohttp_retry

A typed retry client for aiohttp that provides sophisticated retry strategies including exponential backoff, random jitter, Fibonacci sequence, and custom response evaluation. It wraps aiohttp.ClientSession to support automatic retries based on HTTP status codes, exceptions, or callbacks, and features URL rotation and the ability to define complex request sequences using RequestParams.

Tokens: 3.2K
Snippets: 10
Records: 12
Agent score: 5%

What's inside aiohttp-retry

Configure RetryOptions
master
Retry behavior is controlled by RetryOptionsBase subclasses. You can define these at the RetryClient level or per individual request method (which will override the client-level settings).
Key parameters include:
- attempts: Number of retries.
- statuses: An iterable of HTTP status codes that should trigger a retry.
- exceptions: An iterable of exception types that should trigger a retry (defaults to all).
- retry_all_server_errors: If True (default), all 5xx errors are retried. If False, only the statuses explicitly listed in statuses are retried.
- evaluate_response_callback: A callback function that receives a ClientResponse and returns a bool to decide whether to retry (useful for malformed data logic).
Available strategies:
- ExponentialRetry: Exponential backoff.
- RandomRetry: Random backoff.
- ListRetry: Backoff based on a predefined list.
- FibonacciRetry: Fibonacci sequence backoff.
- JitterRetry: Exponential retry with added randomness.
https://github.com/inyutin/aiohttp_retry/blob/master/README.md

Access current attempt number via TraceConfig

master

The RetryClient injects the current_attempt number into the request_trace_ctx within the aiohttp tracing mechanism. This allows you to implement custom logic (like logging) based on which attempt is currently running.

import logging
import sys
from types import SimpleNamespace
from aiohttp import ClientSession, TraceConfig, TraceRequestStartParams
from aiohttp_retry import RetryClient, ExponentialRetry

handler = logging.StreamHandler(sys.stdout)
logging.basicConfig(handlers=[handler])
logger = logging.getLogger(__name__)
retry_options = ExponentialRetry(attempts=2)

async def on_request_start(
    session: ClientSession,
    trace_config_ctx: SimpleNamespace,
    params: TraceRequestStartParams,
) -> None:
    # Access the current attempt from the trace context
    current_attempt = trace_config_ctx.trace_request_ctx['current_attempt']
    if retry_options.attempts <= current_attempt:
        logger.warning('Wow! We are in last attempt')

async def main():
    trace_config = TraceConfig()
    trace_config.on_request_start.append(on_request_start)
    retry_client = RetryClient(retry_options=retry_options, trace_configs=[trace_config])

    response = await retry_client.get('https://httpstat.us/503', ssl=False)
    print(response.status)

    await retry_client.close()

Install aiohttp-retry
master
Install the package using pip. Requires Python 3.9 or higher.
```
pip install aiohttp-retry
```
https://github.com/inyutin/aiohttp_retry/blob/master/README.md
Perform requests with URL rotation
master
To handle cases where a specific endpoint or server is failing, you can provide a list or tuple of URLs to the request methods. The RetryClient will attempt to use the next URL in the sequence for each subsequent retry attempt.
Supported URL types for rotation:
- str or yarl.URL: Single URL (no rotation).
- list[str | yarl.URL]: A list of URLs to rotate through.
- tuple[str | yarl.URL, ...]: A tuple of URLs to rotate through.
Note: The number of URLs provided should ideally match the number of attempts configured in your retry_options to ensure every retry has a target.
```
# If attempts=3, the client will try server1, then server2, then server1 again (cycling)
urls = ["https://server1.com", "https://server2.com"]
async with client.get(urls) as response:
    ... 
```
https://github.com/inyutin/aiohttp_retry/blob/master/aiohttp_retry/client.py

Use RetryClient for basic retries

master

The RetryClient is a wrapper around aiohttp.ClientSession that adds retry logic. You can use it as an async context manager or manually call .close(). It supports various retry strategies like ExponentialRetry and RandomRetry.

from aiohttp_retry import RetryClient, ExponentialRetry

async def main():
    retry_options = ExponentialRetry(attempts=1)
    retry_client = RetryClient(raise_for_status=False, retry_options=retry_options)
    async with retry_client.get('https://ya.ru') as response:
        print(response.status)
        
    await retry_client.close()

Change request parameters between retries using .requests()

master

If you need to change the URL, headers, or other parameters for each subsequent attempt, use the RetryClient.requests() method with a list of RequestParams objects.

from aiohttp_retry import RetryClient, RequestParams, ExponentialRetry

async def main():
    retry_client = RetryClient(raise_for_status=False)

    async with retry_client.requests(
        params_list=[
            RequestParams(
                method='GET',
                url='https://ya.ru',
            ),
            RequestParams(
                method='GET',
                url='https://ya.ru',
                headers={'some_header': 'some_value'},
            ),
        ]
    ) as response:
        print(response.status)
        
    await retry_client.close()

Reference: RequestParams
master
The RequestParams dataclass defines the parameters for a specific attempt within a RetryClient.requests() call. It mirrors the arguments used in ClientSession.request.
```
@dataclass
class RequestParams:
    method: str
    url: _RAW_URL_TYPE
    headers: dict[str, Any] | None = None
    trace_request_ctx: dict[str, Any] | None = None
    kwargs: dict[str, Any] | None = None
```
https://github.com/inyutin/aiohttp_retry/blob/master/README.md
Reference: RetryClient Methods
master
The RetryClient provides the same HTTP verb methods as aiohttp.ClientSession, but they accept an additional retry_options argument to override client-level settings.
```
# Available methods:
# request, get, head, post, put, patch, delete
```
https://github.com/inyutin/aiohttp_retry/blob/master/README.md
Use RetryClient for resilient HTTP requests
master
The RetryClient is the primary entry point for making HTTP requests with automatic retry logic. It wraps an aiohttp.ClientSession and uses RetryOptionsBase (like ExponentialRetry) to determine when and how to retry failed requests.
Key features:
- Automatic Retries: Retries based on status codes, exceptions, or custom callbacks.
- URL Rotation: You can pass a single URL or a list/tuple of URLs. If a list is provided, the client will cycle through them for each retry attempt.
- Context Management: Supports async with for both the client itself and individual request contexts.
- Flexible Methods: Provides standard HTTP methods like get, post, put, delete, etc.
```
from aiohttp import ClientSession
from aiohttp_retry.client import RetryClient, ExponentialRetry

async def main():
    # Initialize with custom retry options
    retry_options = ExponentialRetry(attempts=3)
    
    async with RetryClient(retry_options=retry_options) as client:
        # Single URL request
        async with client.get("https://api.example.com/data") as response:
            print(response.status)

        # Rotating through multiple URLs on retry
        urls = ["https://server1.com", "https://server2.com"]
        async with client.get(urls) as response:
            print(response.status)
```
https://github.com/inyutin/aiohttp_retry/blob/master/aiohttp_retry/client.py
Request methods in RetryClient
master
The RetryClient exposes several methods to perform HTTP requests. Each method returns a _RequestContext object, which is an asynchronous context manager that performs the actual request and handles retries.
Available methods:
- request(method: str, url: StrOrURL, ...): Generic request method.
- requests(params_list: list[RequestParams], ...): Performs a sequence of requests defined by RequestParams objects.
- get(url: _URL_TYPE, ...)
- post(url: _URL_TYPE, ...)
- put(url: _URL_TYPE, ...)
- patch(url: _URL_TYPE, ...)
- delete(url: _URL_TYPE, ...)
- head(url: _URL_TYPE, ...)
- options(url: _URL_TYPE, ...)
All methods accept optional retry_options and raise_for_status overrides for that specific call.
https://github.com/inyutin/aiohttp_retry/blob/master/aiohttp_retry/client.py

Configure RetryClient via constructor

master

When initializing RetryClient, you can configure the following:

Parameter	Type	Description
`client_session`	`ClientSession`	An existing `aiohttp.ClientSession`. If not provided, a new one is created and managed by the client.
`logger`	`logging.Logger`	A logger instance for retry events. Defaults to `logging.getLogger("aiohttp_retry")`.
`retry_options`	`RetryOptionsBase`	The strategy for retries (e.g., `ExponentialRetry`). Defaults to `ExponentialRetry()`.
`raise_for_status`	`bool`	If `True`, the client will call `response.raise_for_status()` if a retry is skipped due to a non-success status. Defaults to `False`.

from aiohttp import ClientSession
from aiohttp_retry.client import RetryClient, ExponentialRetry

# Using an existing session
session = ClientSession()
client = RetryClient(client_session=session, retry_options=ExponentialRetry(attempts=5))

Define complex request sequences with RequestParams
master
The requests() method allows you to define a list of RequestParams to be executed. This is useful for complex retry scenarios where you want to change headers, methods, or URLs specifically for each attempt.
RequestParams fields:
- method: HTTP method (e.g., 'GET', 'POST').
- url: The target URL.
- headers: Dictionary of HTTP headers.
- trace_request_ctx: Dictionary to pass extra context into the request trace.
- kwargs: Additional keyword arguments passed to the underlying aiohttp request.
```
from aiohttp_retry.client import RequestParams

params = [
    RequestParams(method='GET', url='https://api.com/v1'),
    RequestParams(method='GET', url='https://api.com/v2', headers={'X-Retry': 'true'}),
]

async with client.requests(params) as response:
    ... 
```
https://github.com/inyutin/aiohttp_retry/blob/master/aiohttp_retry/client.py