DOCSEARCHbyAlgolia
Submit Docs

hackney

repository·master·Indexed 23 days ago

https://github.com/benoitc/hackney

A simple, reliable, and fast HTTP client for Erlang and Elixir. It supports HTTP/2 and HTTP/3 (QUIC), connection pooling, streaming, and WebSocket/WebTransport support. Features include multipart uploads, automatic decompression, proxy support, and a process-per-connection architecture for isolation and resource management.

Tokens
22.3K
Snippets
64
Records
102
Agent score
80%

What's inside hackney

  1. How HTTP/2 connection multiplexing works

    master

    Unlike HTTP/1.1, where each request typically requires its own connection, HTTP/2 allows multiple concurrent requests to share a single TCP connection by multiplexing them as independent "streams".

    • Automatic Multiplexing: When using the high-level API, hackney_pool automatically reuses existing HTTP/2 connections for the same {Host, Port, Transport} tuple.
    • Concurrent Requests: You can fire multiple requests in parallel on the same connection. Responses may arrive out of order due to multiplexing.
    • Stream Isolation: Each request is assigned a unique StreamId. The hackney_conn process (a gen_statem) manages these streams and routes responses back to the correct caller.
    %% All three requests share ONE TCP connection
{ok, _, _, _} = hackney:get(<<"https://nghttp2.org/">>).
{ok, _, _, _} = hackney:get(<<"https://nghttp2.org/blog/">>).
{ok, _, _, _} = hackney:get(<<"https://nghttp2.org/documentation/">>).

  2. Use Active Mode for event-driven processing

    master

    In Active Mode, every event (data, datagrams, streams, or closure) is forwarded to the owner process, uniformly tagged with its stream ID. This is useful for event-driven architectures.

    Modes:

    • active, false: Passive mode (default). You must poll with wt_recv.
    • active, true: Continuous forwarding of all events.
    • active, once: Delivers a single message and then reverts to passive mode. Use hackney:wt_setopts(Conn, [{active, once}]) to arm it for the next message.

    Event Tags:

    • {hackney_wt, Conn, {binary, Data}}: Data on the default channel.
    • {hackney_wt, Conn, {datagram, Data}}: Inbound datagram.
    • {hackney_wt, Conn, {stream, Id, Data}}: Data on a server-opened stream.
    • {hackney_wt, Conn, {stream_fin, Id, Data}}: FIN on a server-opened stream.
    • {hackney_wt, Conn, closed}: Connection closed.
    • {hackney_wt_error, Conn, Reason}: Error occurred.
    {ok, Conn} = hackney:wt_connect(URL, [{active, true}]),
receive
    {hackney_wt, Conn, {binary, Data}}         -> handle(Data);
    {hackney_wt, Conn, {datagram, Data}}       -> handle_dgram(Data);
    {hackney_wt, Conn, {stream, Id, Data}}     -> handle_stream(Id, Data);
    {hackney_wt, Conn, {stream_fin, Id, Data}} -> handle_fin(Id, Data);
    {hackney_wt, Conn, closed}                 -> done;
    {hackney_wt_error, Conn, Reason}           -> error
end.

  3. How Hackney's process-per-connection architecture works

    master

    Hackney uses a process-per-connection model where every HTTP connection is managed by its own gen_statem process. This provides clean isolation (no shared mutable state), automatic cleanup (crashed processes clean up their own sockets), and simple ownership (the connection process owns the socket).

    If the process that checked out a connection (the owner) crashes, the connection process detects this via OTP monitoring and terminates automatically to prevent socket leaks.

  4. Compare Hackney 2.x Architecture to 1.x

    master

    Hackney 2.x introduces several architectural improvements over 1.x, focusing on security and reliability:

    • State Storage: Moved from ETS tables to Process state.
    • Socket Ownership: Sockets are always owned by the connection process, rather than being transferred.
    • Error Cleanup: Automatic via process exit instead of manual management.
    • SSL Pooling: 2.x does not pool SSL connections (to prevent security risks like session confusion), whereas 1.x did. 2.x only pools TCP connections.
    • Connection Limits: 2.x uses per-host limits instead of a single global pool size.
    • Prewarm: 2.x supports connection prewarming, which was not available in 1.x.

  5. Middleware Scope and Limitations

    master

    Middleware only runs around the high-level hackney:request/1..5 functions. It does not intercept low-level calls to hackney:connect/* or hackney:send_request/2; those paths bypass the middleware layer entirely.

    Async/Streaming: For async or streaming uploads, the Next function returns a bare {ok, Ref} or {ok, ConnPid}. Middleware cannot observe the completion of these requests via the standard return value; you must proxy stream_to if you need to observe completion in async mode.

    Error Handling: If a middleware crashes, the exception propagates directly to the caller. Hackney does not wrap middleware in try/catch blocks.

  6. How HTTP/2 multiplexing and pooling works

    master

    Unlike HTTP/1.1, which uses a pool of multiple connections per host, HTTP/2 in hackney uses a single shared connection per host to handle multiple concurrent requests via stream multiplexing.

    Key Differences in Pooling

    AspectHTTP/1.1HTTP/2
    Connections per hostMultiple (pool)One (shared)
    Checkout behaviorExclusive accessShared access
    Checkin behaviorReturn to poolKeep in pool
    Request handlingSequentialMultiplexed streams

    Connection Flow

    When calling hackney:get/1 (or similar methods):

    1. The library checks for an existing HTTP/2 connection via checkout_h2/3.
    2. If found, the connection is reused.
    3. If not found, it follows the standard TCP/SSL flow, checks the protocol via get_protocol/1, and if http2 is negotiated, registers it via register_h2/1.
    4. Requests are then assigned a StreamId for multiplexing.

  7. How hackney middleware works

    master

    Hackney uses a RoundTripper-style middleware layer that wraps hackney:request/1..5. A middleware is a plain Erlang function that can observe, rewrite, short-circuit, or wrap a request/response pair.

    Chain Order: Middleware is defined as a list [A, B, C]. The request flows from the outermost to the innermost (A → B → C → transport), and the response unwinds in reverse (transport → C → B → A). The first item in the list is the outermost layer.

    Middleware Signature: A middleware is a function with the signature fun((request(), next()) -> response()).

    -type request() :: #{method  := atom() | binary(),
                     url     := #hackney_url{},
                     headers := [{binary(), binary()}],
                     body    := term(),
                     options := [term()]}.

-type response() :: {ok, integer(), list(), binary()} 
                  | {ok, integer(), list()}         %% HEAD
                  | {ok, pid()}                     %% async or streaming upload
                  | {error, term()}.

-type next()       :: fun((request()) -> response()).
-type middleware() :: fun((request(), next()) -> response()).

  8. How connection pooling works in hackney_pool

    master

    Hackney implements TCP-only pooling. For security and simplicity, SSL connections are never pooled; they are closed after use. Instead, the pool stores raw TCP connections which can be upgraded to SSL in-place when an HTTPS request is made. This allows pooled TCP connections to serve both HTTP and HTTPS requests and ensures SSL session state is never shared across requests.

    Pool State and Limits

    • Keepalive Timeout: Idle connections are closed after a keepalive_timeout (default and max: 2000ms) to prevent stale connections and resource accumulation.
    • Prewarm: The pool maintains a prewarm_count (default 4) of connections per host to reduce latency.

  9. Use Async Responses and Streaming

    master

    Async mode allows you to process responses incrementally.

    • Standard Async: Use [async] in options to receive {hackney_response, Ref, Msg} messages.
    • Async Once: Use [{async, once}] to receive exactly one message, then use hackney:stream_next(Ref) to request the next one.
    • Stream to Another Process: Use {stream_to, Receiver} to delegate connection ownership to a different process. This ensures that if the receiver dies, the connection is cleaned up, but if the caller dies, the connection continues.
    %% Async Once
{ok, Ref} = hackney:get(URL, [], <<>>, [{async, once}]),
receive {hackney_response, Ref, Msg} -> ok end,
hackney:stream_next(Ref).

%% Stream to Another Process
Receiver = spawn(fun() -> receive_loop() end),
{ok, Ref} = hackney:get(URL, [], <<>>, [
    async,
    {stream_to, Receiver}
]).

  10. Use 0-RTT and Session Resumption with HTTP/3

    master

    Hackney automatically caches TLS session tickets to enable 0-RTT (Zero Round-Trip Time) for subsequent connections.

    Important Limitations:

    • 0-RTT only works for bodyless requests (e.g., GET). For requests with a body, only the handshake is abbreviated, resulting in 1-RTT.
    • The zero_rtt option controls this behavior.

    Options:

    • [{zero_rtt, true}]: (Default) Enables automatic resumption and 0-RTT for bodyless requests.
    • [{zero_rtt, false}]: Disables 0-RTT.
    • [{connect_options, [{session_ticket, Ticket}]}]: Manually provide a ticket to override the cache.
    %% Default: resumption/0-RTT used automatically
hackney:get(Url, [], <<>>, [{protocols, [http3]}]).

%% Disable 0-RTT
hackney:get(Url, [], <<>>, [{protocols, [http3]}, {zero_rtt, false}]).

%% Supply a ticket explicitly
hackney:get(Url, [], <<>>, [{protocols, [http3]}, {connect_options, [{session_ticket, Ticket}]}]).

  11. Configure and use connection pooling

    master

    hackney uses connection pooling by default. You can create named pools using hackney_pool:start_pool/2 and specify which pool to use in your request options via {pool, Name}. To disable pooling for a specific request, use {pool, false}.

    %% Create a pool
hackney_pool:start_pool(api_pool, [{max_connections, 50}]).

%% Use the pool
hackney:get(URL, [], <<>>, [{pool, api_pool}]).

%% Disable pooling
hackney:get(URL, [], <<>>, [{pool, false}]).

  12. Manage HTTP/3 UDP blocking and fallback

    master

    If UDP traffic is blocked on a network, HTTP/3 connections will fail. Hackney uses negative caching to prevent repeated failed attempts.

    • Automatic Fallback: If you provide a list of protocols like [{protocols, [http3, http2, http1]}], Hackney will attempt HTTP/3 and fall back to HTTP/2 or HTTP/1.1 if it fails.
    • Checking Blocked Status: Use hackney_altsvc:is_h3_blocked(Host, Port) to check if a host is currently marked as blocked.
    • Manual Marking: Use hackney_altsvc:mark_h3_blocked(Host, Port) to manually trigger the blocked state (useful for testing).
    %% Check if host is marked as H3-blocked
hackney_altsvc:is_h3_blocked(<<"example.com">>, 443). %% true | false

%% Manually mark blocked
hackney_altsvc:mark_h3_blocked(<<"example.com">>, 443).