pysimdjson Documentation

pysimdjson provides Python bindings for the simdjson project, which is a SIMD-accelerated JSON parser. It is designed to be safe for all environments: if SIMD instructions are unavailable on the host CPU, the library automatically falls back to a standard parser.

Supported platforms include OS X, Linux, and Windows, with support for Python versions 3.9 through 3.12.

The Native API in pysimdjson provides significant performance improvements over the builtin-compatible API when you only need to access specific parts of a JSON document.

Instead of immediately converting the entire JSON structure into Python objects, the Native API returns Object and Array types. These act as 'fake' dicts and lists that delay the creation of actual Python objects until the specific elements are accessed. This lazy evaluation minimizes overhead for large documents.

import simdjson

parser = simdjson.Parser()
# Parsing returns Object or Array types which delay Python object creation
doc = parser.parse('{"key": "value"}')
print(doc['key'])

If you need to extract a specific property from a document but want to forward or store the rest of the document as a raw string (e.g., in a message router or database), use the .mini property on Object or Array instances.

This returns the entire content of that object/array as a minified Python str. This allows you to access metadata (like a routing key) without ever converting the large payload into a heavy Python object structure.

import simdjson

# Example: Extracting a key and getting the rest as a minified string
parser = simdjson.Parser()
doc = parser.parse(request_data)

key = doc['key']          # Only 'key' is turned into a Python object
payload = doc.mini       # The rest is returned as a minified string

The simdjson module provides load and loads functions designed as drop-in replacements for the built-in Python json module. These functions allow you to parse JSON data with minimal code changes. While they are slightly slower than using the explicit Parser interface, they are optimized for ease of use and

import simdjson

# Parsing from a string
data = simdjson.loads('{"key": "value"}')

# Parsing from a file-like object
with open('data.json', 'r') as f:
    data = simdjson.load(f)

When processing many JSON documents, do not create a new simdjson.Parser() for every document. Instead, instantiate a single parser and reuse it for multiple calls to .parse().

Reusing the parser allows it to reuse its internal buffers, drastically reducing the number of memory allocations. If a new document is larger than the existing buffer, the parser will automatically grow the buffer to fit.

import simdjson

# Efficient: Reuse the same parser instance
parser = simdjson.Parser()

for i in range(100):
    # The parser reuses its internal buffer across iterations
    doc = parser.parse(b'{"a": "b"}')
    # Explicitly delete doc if necessary to free the reference to the parsed content
    del doc

When working with Cython, you can use additional environment variables to modify the build process:

• Debug/Coverage: To build with support for linetracing and coverage, use BUILD_FOR_DEBUG=1 along with BUILD_WITH_CYTHON=1.
• Force Rebuild: To prevent the build from reusing an existing .so file and force Cython to rebuild it, use FORCE_REBUILD=1 along with BUILD_WITH_CYTHON=1.

# Build for debug/coverage
BUILD_WITH_CYTHON=1 BUILD_FOR_DEBUG=1 python setup.py develop

# Force a full rebuild
BUILD_WITH_CYTHON=1 FORCE_REBUILD=1 python setup.py develop

If you want to force an installation from source (for example, to use a newer compiler than the one used for the pre-compiled binary wheels), use the --no-binary :all: flag.

Note: When installing from source, a C++11 (or better) compiler is required to build the underlying simdjson library.

pip install pysimdjson --no-binary :all:

You can install pysimdjson using pip. If binary wheels are available for your platform and Python version, no additional requirements are needed.

Binary wheels are provided for CPython 3.9 through 3.12 on the following architectures:

• x86_64: OS X, Windows, Linux
• ARM64: Linux
• PowerPC: Linux

pip install pysimdjson

To set up a development environment with all necessary testing dependencies, install the package in editable mode using the [test] extra:

pip install -e ".[test]"

To run the test suite, use pytest. For slow integration tests, use the --runslow flag:

pytest
# or for slow integration tests
pytest --runslow

By default, setup.py uses the pre-generated csimdjson.cpp to avoid requiring Cython as an install-time dependency. If you need to regenerate csimdjson.cpp from the .pyx and .pxd files, set the BUILD_WITH_CYTHON=1 environment variable when running setup.py develop.

BUILD_WITH_CYTHON=1 python setup.py develop