lua-resty-http

To handle large response bodies without high memory usage, use the streamed request pattern. This involves three distinct steps:

1. Connect: Establish a connection using connect.
2. Request: Send the request using request (providing path and Host header instead of a full URI).
3. Stream: Use the body_reader iterator from the response object to read the body in chunks.

Finally, use set_keepalive() to return the connection to the pool or close it. This pattern is essential for predictable memory usage and allows for request pipelining.

local httpc = require("resty.http").new()

-- First establish a connection
local ok, err, ssl_session = httpc:connect({
    scheme = "https",
    host = "127.0.0.1",
    port = 8080,
})
if not ok then
    ngx.log(ngx.ERR, "connection failed: ", err)
    return
end

-- Then send using `request`, supplying a path and `Host` header instead of a
-- full URI.
local res, err = httpc:request({
    path = "/helloworld",
    headers = {
        ["Host"] = "example.com",
    },
})
if not res then
    ngx.log(ngx.ERR, "request failed: ", err)
    return
end

-- We can use the `body_reader` iterator, to stream the body according to our
-- desired buffer size.
local reader = res.body_reader
local buffer_size = 8192

repeat
    local buffer, err = reader(buffer_size)
    if err then
        ngx.log(ngx.ERR, err)
        break
    end

    if buffer then
        -- process
    end
until not buffer

local ok, err = httpc:set_keepalive()
if not ok then
    ngx.say("failed to set keepalive: ", err)
    return
end

For simple use cases where you don't need to manage connections manually, use the request_uri method. This method handles the entire end-to-end request process: connecting, sending the request, and buffering the entire response body into memory. After the call, the connection is automatically returned to the connection pool or closed.

Key features of request_uri:

• Buffers the entire response body into res.body.
• Supports connection keepalive via keepalive, keepalive_timeout, and keepalive_pool parameters.
• Parameters in the params table (like path or query) will override components of the provided uri (except scheme, host, and port).

local httpc = require("resty.http").new()

-- Single-shot requests use the `request_uri` interface.
local res, err = httpc:request_uri("http://example.com/helloworld", {
    method = "POST",
    body = "a=1&b=2",
    headers = {
        ["Content-Type"] = "application/x-www-form-urlencoded",
    },
})
if not res then
    ngx.log(ngx.ERR, "request failed: ", err)
    return
end

-- The `res` table contains the expected `status`, `headers` and `body` fields.
local status = res.status
local length = res.headers["Content-Length"]
local body   = res.body

The get_client_body_reader method returns an iterator function used to read the downstream client's request body in a streaming fashion. This is useful for proxying large bodies without loading them entirely into memory.

Usage Patterns:

1. Manual iteration: Use the returned iterator in a loop with a specified chunk size.
2. Proxying to upstream: Pass the returned iterator directly into the body field of a request parameter in httpc:request to stream the client body to an upstream server.

syntax: reader, err = httpc:get_client_body_reader(chunksize?, sock?)

• chunksize?: Optional default chunk size (defaults to 65536).
• sock?: An optional already established socket.

-- Example: Streaming client body to an upstream request
local client_body_reader, err = httpc:get_client_body_reader()

local res, err = httpc:request({
    path = "/helloworld",
    body = client_body_reader,
})

The request_pipeline method allows you to send a sequence of requests in order. It accepts a table where each element is a table of parameters for a single request.

Important behaviors:

• It returns a table of response handles.
• Due to the nature of pipelining, responses are not actually read from the socket until you access a response field (like status or headers).
• You must read the entire body (and any trailers) of a response before attempting to access the next response in the pipeline.
• You should check a field like status first to ensure a socket read error hasn't occurred before processing the body.

local responses = httpc:request_pipeline({
    { path = "/b" },
    { path = "/c" },
    { path = "/d" },
})

for _, r in ipairs(responses) do
    if not r.status then
        ngx.log(ngx.ERR, "socket read error")
        break
    end

    ngx.say(r.status)
    ngx.say(r:read_body())
end

The connect method establishes a TCP/SSL connection and handles proxy configuration. It is the recommended way to start a manual connection session.

Syntax: ok, err, ssl_session = httpc:connect(options)

Options Table:

• scheme: scheme to use (e.g., https), or nil for unix domain socket.
• host: target host, or path to a unix domain socket.
• port: port on target host (defaults to 80 or 443 based on scheme).
• pool: custom connection pool name.
• pool_debug: set to true to log the connection pool name at DEBUG level.
• pool_size: connection pool size (per OpenResty docs).
• backlog: TCP backlog (per OpenResty docs).
• proxy_opts: sub-table for proxy configuration.
• ssl_reused_session: SSL session reuse option.
• ssl_verify: SSL verification (defaults to true).
• ssl_server_name: SNI configuration.
• ssl_send_status_req: SSL status request.
• ssl_client_cert: client certificate (requires ngx_lua_http_module >= v0.10.23).
• ssl_client_priv_key: client private key (requires ngx_lua_http_module >= v0.10.23).