requests-scala

repository·master·Indexed 20 days ago

https://github.com/com-lihaoyi/requests-scala

A Scala port of the Python Requests library providing a flexible and intuitive API for making HTTP requests. It supports custom headers, timeouts, request/response compression, cookie management, and mTLS via client-side certificates. Features include a `requests.Session` for performance and cookie persistence, and a `Request` case class for pre-configuring request templates. Requires JDK 11 or higher for version 0.9.0+.

Tokens
7.5K
Snippets
27
Records
31
Agent score
73%

What's inside requests-scala

  1. Why use Requests-Scala

    master

    Requests-Scala is designed to provide a lightweight, easy-to-use HTTP client for Scala, inspired by the Python requests library. Unlike many other Scala HTTP clients (such as Akka-HTTP, Play-WS, or Http4s) that require complex setups involving implicit ActorSystems, ExecutionContexts, or fluent builder patterns, Requests-Scala focuses on simplicity. It allows you to make HTTP requests using simple function calls that return a response, making it ideal for applications where the overhead of asynchrony and complex DSLs is not required.

    // Requests-Scala
    val r = requests.get(
      "https://api.github.com/search/repositories",
      params = Map("q" -> "http language:scala", "sort" -> "stars")
    )
    
    r.text()
    // {"login":"lihaoyi","id":934140,"node_id":"MDQ6VXNlcjkzNDE0MA==",...}
  2. Use `requests.Session` for performance and cookie persistence

    master

    A requests.Session provides two main benefits:

    1. Performance: Re-uses the underlying HTTP client infrastructure (like java.net.HttpClient) across multiple requests, reducing overhead.
    2. Cookie Persistence: Automatically handles sending and receiving cookies across all requests made within that session.

    Key Session Features

    • Automatic Cookie Management: If a request receives a Set-Cookie header, the session stores it and includes it in subsequent requests. To disable this, use persistCookies = false.
    • Common Configuration: You can define common headers or cookieValues at the session level so they are applied to every request made by that session.
    • Resource Management: Sessions must be explicitly closed using .close() to prevent leaking client threads and other resources.

    Performance Note

    Requests will spawn a new client (losing the performance benefit of the session) if you override certain parameters on a per-request basis, specifically: proxy, cert, sslContext, verifySslCerts, or connectTimeout.

    val s = requests.Session(
      headers = Map("x-special-header" -> "omg"),
      cookieValues = Map("cookie" -> "vanilla")
    )
    
    try {
      val r1 = s.get("https://httpbin.org/cookies")
      val r2 = s.get("https://httpbin.org/headers")
    } finally {
      s.close() // Always close the session!
    }
  3. Stream requests and responses

    master

    To handle large files or data blobs without loading them entirely into memory, use the .stream variants (e.g., requests.get.stream, requests.post.stream). These return a geny.Readable value.

    These can be used with os.write to save files, or passed directly to JSON parsers like ujson.read for streaming parsing. You can also chain requests by passing the stream from one request as the data for another.

    // Download a file via streaming
    os.write(
      os.pwd / "file.json",
      requests.get.stream("https://api.github.com/events")
    )
    
    // Stream JSON directly into a parser
    ujson.read(requests.get.stream("https://api.github.com/events"))
    
    // Chain requests: stream a POST response directly into a file
    os.write(
      os.pwd / "chained.json",
      requests.post.stream(
        "https://httpbin.org/post",
        data = requests.get.stream("https://api.github.com/events")
      )
    )
  4. Handle non-2xx status codes

    master
    By default, the request methods (get, post, put, delete, head, options, patch) throw a requests.RequestFailedException(val response: Response) if the server returns a non-2xx status code. If you want to handle these status codes manually without an exception being thrown, you can disable this behavior by passing check = false to the request method.
  5. Install Requests-Scala

    master

    Depending on your build tool, use the following dependency configurations to add Requests-Scala to your project:

    • Mill: ivy"com.lihaoyi::requests:0.9.2"
    • sbt: "com.lihaoyi" %% "requests" % "0.9.2"
    • Gradle: compile "com.lihaoyi:requests_2.12:0.9.2"
    • Scala-cli: //> using dep "com.lihaoyi::requests:0.9.2"
    // sbt
    "com.lihaoyi" %% "requests" % "0.9.2"
    
    // scala-cli
    //> using dep "com.lihaoyi::requests:0.9.2"
  6. Handle JSON with third-party libraries

    master

    Requests-Scala does not include built-in JSON support but is designed to work seamlessly with libraries like ujson. You can stream JSON uploads or parse JSON responses efficiently.

    Streaming JSON uploads: Use upickle.default.stream(value) or ujson.Obj(...) as the data parameter in requests.post.

    // Upload JSON via streaming
    requests.post(
      "https://api.github.com/some/endpoint",
      data = upickle.default.stream(Map("user-agent" -> "my-app/0.0.1"))
    )
    
    // Upload JSON using ujson objects
    requests.post(
      "https://api.github.com/some/endpoint",
      data = ujson.Obj("user-agent" -> "my-app/0.0.1")
    )
    
    // Parse JSON response
    val r = requests.get("https://api.github.com/events")
    val json = ujson.read(r.text())
  7. How Requester methods work

    master

    The BaseSession trait provides convenient lazy properties for common HTTP verbs. These properties return a Requester instance configured with the session's settings and the specific verb:

    • get (GET)
    • post (POST)
    • put (PUT)
    • delete (DELETE)
    • head (HEAD)
    • options (OPTIONS)
    • patch (PATCH - unofficial)

    You can also use the generic send(method: String) method to specify any custom HTTP method.

    // Using verb-specific helpers
    session.get(url)
    session.post(url, data = ...)
    
    // Using generic send
    session.send("TRACE")(url)
  8. Manage cookies manually

    master

    You can extract cookies from a Response object using the .cookies property (which returns a Map) and pass them into a subsequent request using the cookies argument.

    val r = requests.get("https://httpbin.org/cookies/set?freeform=test")
    val myCookies = r.cookies
    
    val r2 = requests.get("https://httpbin.org/cookies", cookies = myCookies)
  9. Pass parameters in requests

    master

    GET requests

    Use the params argument with a Map[String, String] to pass URL parameters.

    POST and PUT requests

    Use the data argument to send body content. Supported types include:

    • String
    • Array[Byte]
    • java.io.File
    • java.nio.file.Path
    • requests.MultiPart (for multipart uploads)
    • Any type implementing the geny.Writable interface (e.g., ujson.Value, upickle values, or Scalatags tags).

    Note: For POST/PUT key-value pairs, use the data argument instead of params.

    // GET with URL parameters
    val r = requests.get(
        "http://httpbin.org/get",
        params = Map("key1" -> "value1", "key2" -> "value2")
    )
    
    // POST with various data types
    requests.post("https://httpbin.org/post", data = "Hello World")
    requests.post("https://httpbin.org/post", data = Array[Byte](1, 2, 3))
    requests.post("https://httpbin.org/post", data = new java.io.File("thing.json"))
    requests.post("https://httpbin.org/post", data = java.nio.file.Paths.get("thing.json"))
  10. Configure redirect behavior

    master

    Requests automatically follows redirects by default (up to 5). You can control this using the maxRedirects parameter:

    • maxRedirects = 0: Disables redirect handling.
    • maxRedirects = n: Follows up to n redirects.

    All intermediate responses in a redirect chain are stored in the .history field of the final Response. The .history field acts as a linked list of Response objects, where each response points to the one immediately preceding it in the chain.

    // Disable redirects
    val r0 = requests.get("http://www.github.com", maxRedirects = 0)
    
    // Follow up to 1 redirect
    val r1 = requests.get("http://www.github.com", maxRedirects = 1)
  11. Use client-side certificates (mTLS)

    master

    To use client-side certificates, provide a PKCS 12 archive via the cert parameter.

    • For a non-password protected file: cert = "path/to/cert.p12"
    • For a password protected file: cert = ("path/to/cert.p12", "password")

    For testing with self-signed certificates, you can combine cert with verifySslCerts = false. Alternatively, you can provide a custom SSLContext using the sslContext parameter.

    // Password protected P12
    requests.get(
      "https://client.badssl.com",
      cert = ("./badssl.com-client.p12", "password")
    )
    
    // Using a custom SSLContext
    val sslContext: SSLContext = // ...
    requests.get("https://client.badssl.com", sslContext = sslContext)