Typhoeus

Typhoeus is built around three primary classes:

• Typhoeus::Request: Represents an HTTP request object.
• Typhoeus::Response: Represents an HTTP response object.
• Typhoeus::Hydra: Manages making parallel HTTP connections.

You can instantiate a request with a URL and an optional hash of settings.

request = Typhoeus::Request.new(
  "www.example.com",
  method: :post,
  body: "this is a request body",
  params: { field1: "a field" },
  headers: { Accept: "text/html" }
)

You can install Typhoeus using Bundler or by installing the gem directly via RubyGems.

bundle add typhoeus
# or
gem install typhoeus

Use Typhoeus::Hydra to manage multiple requests in parallel. Queue requests using hydra.queue(request) and then call hydra.run (which is a blocking call). You can also queue new requests inside an on_complete callback to create dependency chains.

hydra = Typhoeus::Hydra.hydra

first_request = Typhoeus::Request.new("http://example.com/posts/1")
first_request.on_complete do |response|
  third_url = response.body
  third_request = Typhoeus::Request.new(third_url)
  hydra.queue third_request
end

second_request = Typhoeus::Request.new("http://example.com/posts/2")

hydra.queue first_request
hydra.queue second_request
hydra.run

For large responses, use the on_body callback to process data in chunks. When on_body is used, Typhoeus will not store the complete response in memory. To stop the stream early and prevent memory leaks, return the :abort symbol from the on_body block.

downloaded_file = File.open 'huge.iso', 'wb'
request = Typhoeus::Request.new("www.example.com/huge.iso")

request.on_headers do |response|
  raise "Request failed" if response.code != 200
end

request.on_body do |chunk|
  downloaded_file.write(chunk)
  # To interrupt the stream halfway:
  # :abort if buffer.size > 1024 * 1024
end

request.on_complete do |response|
  downloaded_file.close
  # Note: response.body will be empty when streaming
end

request.run

Use on_complete to define a callback that executes after a request finishes. This is the recommended way to handle success, timeouts, and connection errors. Note that callbacks must be defined before calling .run.

request = Typhoeus::Request.new("www.example.com", followlocation: true)

request.on_complete do |response|
  if response.success?
    # Success logic
  elsif response.timed_out?
    # Timeout logic
  elsif response.code == 0
    # Connection/Network error logic
    log(response.return_message)
  else
    # Non-successful HTTP response (e.g. 404, 500)
    log("HTTP request failed: " + response.code.to_s)
  end
end

request.run

To avoid setting a custom User-Agent header on every request, you can set a global default via Typhoeus::Config.user_agent.

Typhoeus::Config.user_agent = "custom user agent"

When enabled, Hydra will memoize requests within a single run call. If the same request is queued multiple times, only one actual network request is issued, but on_complete handlers for all queued instances will still be executed.

Typhoeus::Config.memoize = true

hydra = Typhoeus::Hydra.new(max_concurrency: 1)
2.times do
  hydra.queue Typhoeus::Request.new("www.example.com")
end
hydra.run