DOCSEARCHbyAlgolia
Submit Docs

HTTPX

repository·master·Indexed 12 days ago

https://github.com/encode/httpx

A next-generation, fully featured HTTP client for Python 3.9+ that supports both synchronous and asynchronous APIs, HTTP/1.1 and HTTP/2, and provides a command-line interface. It is broadly compatible with the requests library and includes features such as connection pooling, strict timeouts, type annotations, and support for WSGI/ASGI applications.

Tokens
39.8K
Snippets
151
Records
189
Agent score
96%

What's inside HTTPX

  1. Overview of HTTPX features

    master

    HTTPX is a next-generation HTTP client for Python 3 with the following key capabilities:

    • API Compatibility: Broadly requests-compatible.
    • Protocol Support: HTTP/1.1 and HTTP/2.
    • Concurrency: Both standard synchronous and async APIs.
    • Application Integration: Ability to make requests directly to WSGI or ASGI applications.
    • Safety: Strict timeouts enabled everywhere.
    • Type Safety: Fully type annotated.
    • Standard Features: Includes Keep-Alive, connection pooling, cookie persistence, SSL verification, authentication (Basic/Digest), multipart file uploads, and proxy support.

  2. Use libraries with built-in HTTPX support

    master

    Several high-level libraries use HTTPX as their underlying engine for making requests:

    • Authlib: A library for OAuth and OpenID Connect that includes a dedicated OAuth HTTPX client.
    • Gidgethub: An asynchronous GitHub API library that includes native HTTPX support.
    • rpc.py: An ASGI/WSGI-based RPC framework where HTTPX can be used as the client for the RPC service.

  4. Explore HTTPX plugins and extensions

    master

    The HTTPX ecosystem includes several third-party plugins that extend its functionality. These can be categorized into plugins that add features (like caching, retries, or WebSockets) and tools for testing or debugging.

    Feature Plugins

    • Hishel: An HTTP Cache implementation for HTTPX and HTTP Core.
    • HTTPX-Auth: Provides authentication classes compatible with HTTPX's authentication parameter.
    • httpx-caching: Adds caching functionality to HTTPX.
    • httpx-secure: Provides SSRF protection with DNS caching and custom validation.
    • httpx-socks: Adds HTTP and SOCKS proxy transport support.
    • httpx-sse: Enables consuming Server-Sent Events (SSE).
    • httpx-retries: Adds a retry layer to HTTPX requests.
    • httpx-ws: Adds WebSocket support.

    Testing and Mocking

    • pytest-HTTPX: Provides a pytest fixture to mock HTTPX within test cases.
    • RESPX: A utility specifically for mocking out HTTPX requests.
    • VCR.py: Records and repeats HTTP requests for testing purposes.

  5. What are extensions in HTTPX

    master

    Extensions provide an untyped space in both requests and responses for additional information that does not fit into the standard request/response model. They are primarily used for features that may not be available on all transports or for advanced low-level control.

    • Request extensions are passed via the extensions argument in request methods (e.g., client.get(..., extensions={...})) and can be accessed via response.request.extensions.
    • Response extensions are accessible via response.extensions.

  6. Understand how Client and Request configurations merge

    master

    When configuration is provided at both the client level and the request level, the merging behavior depends on the parameter type:

    • Combined: headers, params (query parameters), and cookies are merged together. The client-level values and request-level values are both included.
    • Overridden: For all other parameters (e.g., auth), the request-level value takes priority and overrides the client-level value.
    >>> headers = {'X-Auth': 'from-client'}
>>> params = {'client_id': 'client1'}
>>> with httpx.Client(headers=headers, params=params) as client:
...     headers = {'X-Custom': 'from-request'}
...     params = {'request_id': 'request1'}
...     r = client.get('https://example.com', headers=headers, params=params)
>>> r.request.url
URL('https://example.com?client_id=client1&request_id=request1')
>>> r.request.headers['X-Auth']
'from-client'
>>> r.request.headers['X-Custom']
'from-request'

  7. Supported async environments (AsyncIO, Trio, AnyIO)

    master

    HTTPX automatically detects whether you are using asyncio or trio as your concurrency backend.

    AsyncIO

    Standard Python built-in library.

    Trio

    Requires the trio package to be installed.

    AnyIO

    Works on top of either asyncio or trio. You can specify the backend using anyio.run(..., backend='trio').

    # AsyncIO example
import asyncio
import httpx

async def main():
    async with httpx.AsyncClient() as client:
        response = await client.get('https://www.example.com/')
        print(response)

asyncio.run(main())

  8. Use httpx.Client as an equivalent to requests.Session

    master

    The httpx.Client is the functional equivalent of requests.Session. You can pass keyword arguments to the constructor to configure the client's behavior.

    # Equivalent to requests.Session(**kwargs)
client = httpx.Client(**kwargs)

  9. Implement custom authentication by subclassing httpx.Auth

    master

    To create complex authentication flows (e.g., handling 401 challenges or refreshing tokens), subclass httpx.Auth and implement the auth_flow(request) generator method.

    Key concepts:

    • auth_flow(request): A generator that yields requests. You can yield request to send it, receive the response, and then yield a new request if needed (e.g., after a token refresh).
    • requires_request_body = True: Set this property if your authentication logic needs to access request.content (e.g., for signing requests).
    • requires_response_body = True: Set this property if your logic needs to access response.content, response.json(), etc.
    • Sync vs Async: The standard auth_flow is designed to work with both sync and async clients without performing I/O. If you need to perform I/O (like disk access or using locks), you must override sync_auth_flow(request) for httpx.Client and async_auth_flow(request) for httpx.AsyncClient.
    class MyCustomAuth(httpx.Auth):
    def auth_flow(self, request):
        response = yield request
        if response.status_code == 401:
            # Handle re-authentication logic here
            request.headers['X-Authentication'] = 'new-token'
            yield request

  10. Handle content encoding and character sets

    master

    HTTPX uses utf-8 by default for encoding str request bodies. This differs from requests, which uses latin1.

    • Request Encoding: If you need a specific encoding, encode the string to bytes explicitly: content=my_str.encode("latin1").
    • Response Decoding: HTTPX uses charset_normalizer to guess response encoding. If no encoding is provided and the content is small (< 32 octets), it falls back to utf-8 with error="replace".

  11. Understand Forwarding vs Tunnelling proxy mechanisms

    master

    HTTPX utilizes two primary mechanisms for proxying:

    • Forwarding: The proxy server makes the actual request to the destination server on your behalf and returns the response.
    • Tunnelling (HTTP Tunnel): The proxy establishes a TCP connection to the destination server. The client then reuses this connection to send requests and receive responses. This is the mechanism used to access HTTPS websites through an HTTP proxy, allowing the client to perform the TLS handshake directly with the destination server over the established TCP tunnel.

  12. Implement a custom Transport

    master

    To create a custom transport, subclass httpx.BaseTransport (for synchronous Client) or httpx.AsyncBaseTransport (for asynchronous AsyncClient). You must implement the handle_request or handle_async_request method, which receives a Request and must return a Response.

    import httpx

class HelloWorldTransport(httpx.BaseTransport):
    """
    A mock transport that always returns a JSON "Hello, world!" response.
    """
    def handle_request(self, request):
        return httpx.Response(200, json={"text": "Hello, world!"})

client = httpx.Client(transport=HelloWorldTransport())