ijson Documentation

The ijson routines are chained: basic_parse $\rightarrow$ parse $\rightarrow$ kvitems/items. You can intercept this chain by passing the output (an iterable of events) of one function as the input to another. This allows for custom filtering or injection before full object construction.

import io
import ijson

# Create a parser from a byte stream
parse_events = ijson.parse(io.BytesIO(b'["skip", {"a": 1}, {"b": 2}, {"c": 3}]'))

# Intercept: skip everything until we find the value 'skip'
while True:
    prefix, event, value = next(parse_events)
    if value == "skip":
        break

# Pass the remaining events to items()
for obj in ijson.items(parse_events, 'item'):
    print(obj)

When using the low-level ijson.parse function, the parser generates three-element tuples: (prefix, event, value).

Event Types

• start_map / end_map: Beginning and end of a JSON object. Value is None.
• start_array / end_array: Beginning and end of a JSON array. Value is None.
• map_key: The name of a field in a JSON object. The associated value is the name itself.
• null, boolean, integer, double, number, string: Content events. The associated value is the actual data.

Prefixes

A prefix represents the path to the current context within the JSON document. It is a string where parts are separated by dots (.):

• Object members append <name> to the prefix.
• Array elements append item to the prefix.
• Example: A field name inside an object that is an element of an array might have a prefix like item.name.

In version 3.x, ijson transitioned from a pull model (where the library requests data from the file) to a push model. In this model, the user is responsible for holding data chunks and pushing them into the parsing pipeline.

This architecture decouples I/O from the core parsing logic. The pipeline is composed of chained generator-based coroutines. Data flows through the pipeline as follows:

1. The user sends a chunk of data into the first stage (e.g., basic_parse_basecoro).
2. Each stage processes the data and uses .send() to push the resulting events or values to the next stage in the chain.
3. The final stage sends the processed results to the user's target.

This design allows for greater control over data feeding and enables efficient chaining of parsing stages.

ijson uses different backends for parsing. You can select them via the IJSON_BACKEND environment variable or by importing them directly.

Available Backends

• yajl2_c: Fastest; uses YAJL 2.x C extension. May require YAJL development files.
• yajl2_cffi: Uses YAJL 2.x via CFFI.
• yajl2: Uses YAJL 2.x via ctypes.
• yajl: Deprecated YAJL 1.x via ctypes.
• python: Pure Python parser; recommended for PyPy.

Usage

To use a specific backend, import it directly or use ijson.get_backend(name).

To see all available backends, inspect ijson.ALL_BACKENDS.

import ijson.backends.yajl2_cffi as ijson

for item in ijson.items(...):
    # ...

# OR
backend = ijson.get_backend('yajl2_c')
for item in backend.items(...):
    # ...

To maximize performance when using ijson, follow these steps in order of impact:

1. Use the fastest backend available (typically yajl2_c).
2. Use use_float=True if your JSON data contains only "well-behaved" numbers and you can tolerate potential precision loss.
3. Feed binary data to ijson instead of text data (e.g., open files in 'rb' mode).
4. Tune buf_size to find an optimal buffer size for your specific data source and system.

Install the ijson package from PyPI using pip. Binary wheels are provided for major platforms and Python versions.

pip install ijson

When using low-level parsing, ijson yields events as tuples of (prefix, type, value).

Prefixes represent the path to the nested element from the root. For example, in {"array": [1, 2]}, the number 1 has the prefix array.item.

Available event types and values:

• ('null', None)
• ('boolean', <True or False>)
• ('number', <int or Decimal>)
• ('string', <unicode/str>)
• ('map_key', <str>)
• ('start_map', None)
• ('end_map', None)
• ('start_array', None)
• ('end_array', None)

The yajl2_c backend is a C-based wrapper for the YAJL2 extension, designed for high-performance JSON parsing. It provides three primary interfaces for interacting with JSON data:

1. Generator interface (*_gen): Returns a standard Python generator. Use this for synchronous, memory-efficient iteration over large JSON files.
2. Async interface (*_async): Provides asynchronous parsing capabilities.
3. Coroutine interface (*_basecoro): Uses Python coroutines (via target.send) for low-level event-driven parsing.

All functions accept a buf_size keyword argument to control the internal buffer size (defaults to 64 * 1024 bytes).

# Example of using the generator interface for synchronous parsing
import ijson.backends.yajl2_c as yajl2

with open('large_data.json', 'rb') as f:
    for item in yajl2.items(f, 'item_prefix'):
        print(item)

The 2.x design relies on a blocking, file-like object that ijson pulls data from. This creates limitations in two main scenarios:

1. asyncio environments: Since asyncio relies on non-blocking I/O where data is pushed to the user, the pull model requires emulating a blocking file-like object, which is error-prone.
2. Multi-stage socket conversations: In scenarios where a program must parse JSON from a socket and then immediately send a reply over that same socket before receiving more data, the pull model's requirement to exhaust the stream can conflict with the application's communication flow.