python-multipart

The BaseParser is the foundation for all streaming parsers in this library. It uses a callback-based system to notify the consumer of parsing events without copying data unnecessarily.

Callbacks are registered via set_callback(name, func). There are two types of callbacks:

1. Notification callbacks: Called with no arguments (e.g., on_start, on_end).
2. Data callbacks: Called with (data, start, end), where data[start:end] represents the slice of interest. This avoids expensive data copying.

Common callback names include start, data, end, field_start, field_name, field_data, field_end, part_begin, part_data, part_end, header_begin, header_field, header_value, header_end, headers_finished, and on_end.

from python_multipart.multipart import BaseParser

class MyParser(BaseParser):
    def __init__(self, callbacks):
        super().__init__()
        self.callbacks = callbacks

# Example usage pattern for a consumer of a parser:
def my_data_handler(data, start, end):
    print(f"Found data: {data[start:end]}")

# The parser would then be used by calling .write(chunk) in a loop

For more control or streaming requirements (e.g., computing hashes without loading files into memory), use the MultipartParser class. This allows you to feed data into the parser incrementally using the .write() method.

Setup:

1. Extract the boundary from the Content-Type header (use python_multipart.multipart.parse_options_header to assist).
2. Define a callbacks dictionary containing:
   • on_part_begin: Called when a new part starts.
   • on_part_data: Called with (data, start, end) when a chunk of data is available.
   • on_part_end: Called when a part is fully processed.
3. Initialize MultipartParser(boundary, callbacks).
4. Feed data to the parser using parser.write(data) in a loop.

import python_multipart
from python_multipart.multipart import parse_options_header

# 1. Extract boundary
content_type, params = parse_options_header(environ['CONTENT_TYPE'])
boundary = params.get(b'boundary')

# 2. Define callbacks
callbacks = {
    'on_part_begin': lambda: print("Part started"),
    'on_part_data': lambda data, start, end: print(f"Data chunk: {len(data)}"),
    'on_part_end': lambda: print("Part ended"),
}

# 3. Initialize parser
parser = python_multipart.MultipartParser(boundary, callbacks)

# 4. Stream data
while True:
    chunk = input_stream.read(8192)
    if not chunk:
        break
    parser.write(chunk)

The FormParser uses a configuration dictionary to control behavior. You can pass a custom dictionary to the config argument during initialization to override these values.

Default Configuration Keys

custom_config = {
    "MAX_BODY_SIZE": 5 * 1024 * 1024,  # 5MB
    "UPLOAD_DIR": "/tmp/uploads",
    "UPLOAD_ERROR_ON_BAD_CTE": True
}

parser = FormParser(
    content_type="multipart/form-data",
    boundary="boundary",
    on_field=my_field_cb,
    on_file=my_file_cb,
    config=custom_config
)

When handling file uploads, you can provide a FileConfig dictionary to control how files are stored and managed. This is useful for preventing memory exhaustion or managing disk space.

Key configuration options include:

• UPLOAD_DIR: A directory path where files will be stored. If None, system temporary locations are used.
• UPLOAD_DELETE_TMP: If True (default), automatically created temporary files are deleted.
• UPLOAD_KEEP_FILENAME: If True, the uploaded filename is used (stripped to its basename). If False, a temporary name is used.
• UPLOAD_KEEP_EXTENSIONS: If True, the original file extension is maintained. If False, a .tmp extension is used.
• MAX_MEMORY_FILE_SIZE: The threshold (in bytes) after which the file is moved from memory to a temporary file on disk. Default is 1 MiB.

config = {
    'UPLOAD_DIR': '/tmp/uploads',
    'UPLOAD_DELETE_TMP': True,
    'UPLOAD_KEEP_FILENAME': True,
    'UPLOAD_KEEP_EXTENSIONS': True,
    'MAX_MEMORY_FILE_SIZE': 1024 * 1024  # 1 MiB
}

To parse multipart data manually, instantiate a MultipartParser. You must provide a boundary and a callbacks dictionary containing functions to handle various stages of the parsing lifecycle.

Supported callback keys include:

• on_part_begin: Triggered when a new part starts.
• on_part_data: Triggered when part data is received.
• on_part_end: Triggered when a part ends.
• on_header_field: Triggered for each header field name.
• on_header_value: Triggered for each header field value.
• on_header_end: Triggered when headers for a part are finished.
• on_headers_finished: Triggered when all headers are processed.
• on_end: Triggered when the entire stream is finished.

You can also pass a config dictionary to set limits like MAX_BODY_SIZE, MAX_HEADER_COUNT, and MAX_HEADER_SIZE.

parser = MultipartParser(
    boundary,
    callbacks={
        "on_part_begin": on_part_begin,
        "on_part_data": on_part_data,
        "on_part_end": on_part_end,
        "on_header_field": on_header_field,
        "on_header_value": on_header_value,
        "on_header_end": on_header_end,
        "on_headers_finished": on_headers_finished,
        "on_end": _on_end,
    },
    max_size=MAX_BODY_SIZE,
    max_header_count=MAX_HEADER_COUNT,
    max_header_size=MAX_HEADER_SIZE,
)