Requests: Python HTTP for Humans

repository·main·Indexed 13 days ago

https://github.com/psf/requests

A simple and elegant Python HTTP library for sending HTTP/1.1 requests without manual query string or form-encoding management. It supports Python 3.10+ and provides a high-level interface for GET, POST, PUT, PATCH, DELETE, and HEAD methods, along with session persistence via requests.Session, cookie management with RequestsCookieJar, and a comprehensive exception hierarchy starting with requests.RequestException.

Tokens
22.8K
Snippets
94
Records
123
Agent score
99%

What's inside Requests

  1. Overview of Requests features

    main

    Requests provides several high-level features for modern web interaction, including:

    • Connection Management: Keep-Alive & Connection Pooling, Connection Timeouts, and Chunked Requests.
    • Security: Browser-style SSL Verification, Basic/Digest Authentication, and HTTP(S) Proxy Support.
    • Data Handling: Automatic Content Decoding, Automatic Decompression, Unicode Response Bodies, and Multipart File Uploads.
    • Session & State: Sessions with Cookie Persistence and Elegant Key/Value Cookies.
    • URL & Domain Support: International Domains and URLs, and .netrc support.
    • Downloads: Streaming Downloads.
  2. Use Session objects to persist parameters and cookies

    main

    A requests.Session object allows you to persist certain parameters (like authentication or headers) and cookies across multiple requests. It also utilizes urllib3 connection pooling to reuse underlying TCP connections, which improves performance when making multiple requests to the same host.

    Key Behaviors:

    • Cookie Persistence: Cookies received in one request are automatically sent in subsequent requests made via the same session.
    • Parameter Merging: Parameters passed to a request method (e.g., s.get(..., headers=...)) are merged with session-level parameters. Method-level parameters override session-level ones.
    • Non-Persistence of Method Parameters: Parameters passed directly to a request method (like cookies or headers) are not persisted to the session for future requests.
    • Removing Session Values: To omit a key that exists at the session level from a specific request, set its value to None in the method call.
    • Context Manager: Use with requests.Session() as s: to ensure the session is closed automatically when the block exits.
    import requests
    
    # Persisting cookies
    s = requests.Session()
    s.get('https://httpbin.org/cookies/set/sessioncookie/123456789')
    r = s.get('https://httpbin.org/cookies')
    print(r.text)  # '{"cookies": {"sessioncookie": "123456789"}}'
    
    # Providing default data
    s = requests.Session()
    s.auth = ('user', 'pass')
    s.headers.update({'x-test': 'true'})
    
    # Method-level headers are merged with session headers
    s.get('https://httpbin.org/headers', headers={'x-test2': 'true'})
    
    # Using as a context manager
    with requests.Session() as s:
        s.get('https://httpbin.org/cookies/set/sessioncookie/123456789')
  3. How Response Encoding is Determined

    main

    When accessing Response.text, Requests attempts to guess the encoding:

    1. It first checks the HTTP header for an encoding.
    2. If absent, it uses charset_normalizer or chardet to guess.

    Special Case (RFC 2616): If no explicit charset is present in headers AND the Content-Type header contains text, Requests defaults to ISO-8859-1.

    To override this, you can manually set Response.encoding or use Response.content for raw bytes.

  4. Use netrc for automatic authentication

    main

    If no auth argument is provided, Requests attempts to retrieve credentials for the URL's hostname from the user's .netrc file.

    Requests searches for the file at:

    • ~/.netrc (Unix/macOS)
    • ~/_netrc (Unix/macOS)
    • %USERPROFILE%/.netrc (Windows)
    • The path specified by the NETRC environment variable.

    If credentials are found, the request is sent using HTTP Basic Auth. This behavior overrides raw HTTP authentication headers set via headers=.

    To disable this behavior in a session, set trust_env to False.

    import requests
    
    # Disable netrc/environment variable lookup
    s = requests.Session()
    s.trust_env = False
    s.get('https://httpbin.org/basic-auth/user/pass')
  5. How Requests handles encoded data

    main

    Requests automatically decompresses gzip-encoded responses and attempts to decode response content to Unicode whenever possible.

    To enable automatic decoding of Brotli-encoded responses, you must have either the brotli or brotlicffi package installed in your environment.

    If you require low-level access, you can access the raw response or the underlying socket directly.

  6. Access Request and Response objects

    main

    Every call to requests.get() (or similar methods) involves two main components:

    1. A Request object: The object constructed to query the resource.
    2. A Response object: The object generated containing the server's response and the original Request object.

    You can access the headers sent to the server by inspecting response.request.headers, and the headers received from the server via response.headers.

    import requests
    
    r = requests.get('https://en.wikipedia.org/wiki/Monty_Python')
    
    # Access headers sent to the server
    print(r.request.headers)
    
    # Access headers received from the server
    print(r.headers)
  7. Use Transport Adapters to customize service interaction

    main

    Transport Adapters allow you to define per-service configuration by mounting an adapter to a Session object. When a request URL matches the adapter's prefix (using longest prefix match), that adapter is used.

    Note: To avoid accidental matches (e.g., http://localhost matching http://localhost.other.com), it is recommended to terminate hostnames with a / (e.g., https://github.com/).

    s = requests.Session()
    s.mount('https://github.com/', MyAdapter())
  8. Quickstart with Requests

    main

    Requests is an elegant and simple HTTP library for Python that allows you to send HTTP/1.1 requests easily. It automates tasks like adding query strings to URLs, form-encoding POST data, and managing keep-alive and connection pooling via urllib3.

    >>> import requests
    >>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
    >>> r.status_code
    200
    >>> r.headers['content-type']
    'application/json; charset=utf8'
    >>> r.encoding
    'utf-8'
    >>> r.text
    '{"type":"User"...'
    >>> r.json()
    {'private_gists': 419, 'total_private_repos': 77, ...}
  9. Contribute documentation to Requests

    main

    Documentation improvements are welcome. Documentation files are located in the docs/ directory and are written in reStructuredText using Sphinx for generation.

    Style Guidelines:

    • Width: Maintain a soft-limit of 79 characters per line.
    • Tone: Use a semi-formal, friendly, and approachable prose style.
    • Python Code: When presenting Python code snippets, use single-quoted strings (e.g., 'hello' instead of "hello").
  10. Configure Proxies for Requests

    main

    You can configure proxies in three ways:

    1. Per-request: Pass a proxies dictionary to any request method (e.g., get, post).
    2. Session-wide: Configure a requests.Session object. Note that setting session.proxies may be overwritten by environment variables.
    3. Environment Variables: Requests automatically uses http_proxy, https_proxy, no_proxy, and all_proxy (and their uppercase variants).

    To use HTTP Basic Auth with a proxy, use the http://user:password@host/ syntax in your proxy URL.

    To provide a proxy for a specific scheme and host, use the scheme://hostname form as the key in the dictionary. Proxy URLs must include the scheme.

    import requests
    
    # Per-request proxy
    proxies = {
      'http': 'http://10.10.1.10:3128',
      'https': 'http://10.10.1.10:1080',
    }
    requests.get('http://example.org', proxies=proxies)
    
    # Session-wide proxy
    session = requests.Session()
    session.proxies.update(proxies)
    session.get('http://example.org')
    
    # Specific host proxy
    proxies = {'http://10.20.1.128': 'http://10.10.1.10:5323'}
  11. Submit feature requests to Requests

    main
    Requests is currently in a perpetual feature freeze. The maintainers consider the software to be feature-complete. While you are welcome to raise feature requests via GitHub issues, please be aware that they are highly unlikely to be accepted or approved.