Requests Toolbelt

The Requests Toolbelt is a collection of utilities designed for users of the python-requests library. It provides specialized functionality that is not included in the core requests package but is frequently requested by the community. The library is maintained by the core requests development team.

Key capabilities include:

• multipart/form-data encoding for complex form uploads.
• Custom User-Agent construction.
• SSLAdapter for specialized SSL configurations.
• ForgetfulCookieJar for managing cookie behavior.
• Various utilities for uploading data, handling sessions, and managing threading.

The Requests Toolbelt is a collection of useful utilities for use with the requests library. It provides extended functionality for common tasks such as multipart/form-data encoding, custom User-Agent construction, SSL adapter configuration, and specialized cookie management. To use these utilities, you must have the requests library installed.

pip install requests-toolbelt

The toolbelt provides a threading implementation designed around the concept of using one requests.Session per thread. This approach avoids common thread-safety issues with shared sessions.

Key Characteristics:

• Naïve Implementation: The Pool does not synchronize attributes like authentication or cookies across threads. Each thread operates on its own session.
• No Domain Affinity: The Pool does not attempt to direct requests for the same domain to the same thread. If you are hitting multiple domains, requests to the same domain may be distributed across different threads.
• Core Abstractions:
  • Pool: The main manager for executing a collection of requests.
  • ThreadResponse: A wrapper around a successful response that behaves like a standard requests.Response.
  • ThreadException: A wrapper for exceptions raised during a request.

The tee utilities allow you to consume a streaming response while simultaneously performing an action on the data chunks (like writing to a file or a bytearray). This effectively 'splits' the stream so you can process it in multiple ways without making multiple network requests.

Available utilities:

• tee: General purpose teeing.
• tee_to_file: Duplicates the stream into a specified file.
• tee_to_bytearray: Duplicates the stream into a bytearray object.

from requests_toolbelt.downloadutils.tee import tee_to_file, tee_to_bytearray

# Example: Teeing to a file
tee_to_file(r, 'output.bin')

# Example: Teeing to a bytearray
ba = bytearray()
tee_to_bytearray(r, ba)

The maintainers provide security support for the following:

• The default branch
• The most recent release

When a security bug is identified, the project endeavors to fix it and issue a new release as quickly as possible.

Some utility functions previously located in requests.utils have been deprecated in the main requests library. To continue using these utilities, you should import them from requests_toolbelt.utils.deprecated instead.

This migration ensures continued access to these specific helper functions as they are no longer maintained within the core requests package.

If you identify a security vulnerability, do not report it via the normal GitHub issue tracker. Instead, use the official security advisory page to ensure the issue is handled sensitively and privately.

1. Navigate to the security advisory page.
2. Submit your report. You should receive an acknowledgment and potential follow-up from the maintainers.

To check if your project is using a version of requests-toolbelt with known vulnerabilities, you can consult the osv.dev database or use automated scanning tools in your CI/CD pipeline or local environment:

• pip-audit
• osv-scan

To get started with the Requests Toolbelt, install it via pip. The minimum tested version of requests is 2.1.0.

pip install requests-toolbelt

You can use StreamingIterator with file-like objects (such as sys.stdin) to stream data while providing a known length, preventing the use of Transfer-Encoding: chunked.

import requests
from requests_toolbelt.streaming_iterator import StreamingIterator
import sys

# Assuming size and content_type are known
stream = StreamingIterator(size, sys.stdin)
r = requests.post(url, data=stream, 
                  headers={'Content-Type': content_type})

You can use MultipartEncoder for requests that do not require file uploads, only standard key-value fields.

from requests_toolbelt import MultipartEncoder
import requests

m = MultipartEncoder(fields={'field0': 'value', 'field1': 'value'})

r = requests.post('http://httpbin.org/post', data=m, 
                  headers={'Content-Type': m.content_type})