DOCSEARCHbyAlgolia
Submit Docs

JSON Machine

repository·master·Indexed 20 days ago

https://github.com/halaxa/json-machine

A memory-efficient, stream-based JSON parser for PHP that allows developers to iterate over large JSON files or streams using standard foreach loops with a constant memory footprint. It supports JSON Pointer (RFC 6901) for targeting specific subtrees, recursive iteration for complex structures via RecursiveItems, and custom decoders like ErrorWrappingDecoder to handle malformed items without stopping the parsing process.

Tokens
4.1K
Snippets
10
Records
24
Agent score
80%

What's inside json-machine

  1. Key features of JSON Machine

    master

    JSON Machine is designed for unpredictably long JSON streams or documents with the following characteristics:

    • Constant memory footprint: Handles large documents without increasing memory usage.
    • Ease of use: Iterate through JSON using standard foreach loops without needing events or callbacks.
    • Efficient subtree iteration: Use JSON Pointer to target specific parts of a document.
    • High performance: Optimized for speed and uses native json_decode for item decoding by default.
    • Flexible input: Parses streams, files, or any iterable that produces JSON chunks.

  2. Iterate nested values in arrays using wildcards

    master
    The JSON Pointer specification allows using a hyphen (-) as a wildcard for any array index. This is useful for extracting specific fields from every object within an array without loading the entire array into memory. For example, '/results/-/color' will iterate over the color property of every item in the results array.

  3. Parse a specific JSON subtree using JSON Pointer

    master

    To iterate over a specific subtree rather than the root level, use the pointer option with a valid JSON Pointer string. This allows you to target a specific object or array within the document. Memory consumption remains constant because only one item in the target subtree is loaded into memory at a time.

    <?php

use \JsonMachine\Items;

$fruits = Items::fromFile('fruits.json', ['pointer' => '/results']);
foreach ($fruits as $name => $data) {
    // Iterates through the 'results' subtree
}

  4. Handle errors and malformed JSON items

    master

    All exceptions in the library extend JsonMachineException.

    To prevent a single malformed item from stopping the entire parsing process, use the ErrorWrappingDecoder. When using this decoder, malformed items are yielded as DecodingError objects within your foreach loop, allowing you to continue to the next item.

  5. Use RecursiveItems for complex or deep JSON structures

    master

    When JSON structures are too complex for standard JSON Pointers, or when individual items are too large to handle, use JsonMachine\RecursiveItems.

    Key Characteristics:

    • It never returns a PHP array or object; it only returns scalar values or new RecursiveItems instances.
    • It is slower than Items but provides deep, lazy iteration.
    • Warning: If you break an iteration of a deeper level (e.g., skipping a nested loop) and try to advance to the next item in the parent loop, you may encounter a 'closed generator exception' because the parser must iterate in the background to maintain state.
    <?php

use JsonMachine\RecursiveItems

$users = RecursiveItems::fromFile('users.json');
foreach ($users as $user) {
    /** @var $user RecursiveItems */
    foreach ($user['friends'] as $friend) { // or $user->advanceToKey('friends')
        /** @var $friend RecursiveItems */
        $friendArray = $friend->toArray();
        // $friendArray is now a plain PHP array
    }
}

  7. Install and use JSON Machine for memory-efficient JSON parsing

    master

    JSON Machine is a drop-in replacement for json_decode when dealing with large JSON files or streams. While json_decode(file_get_contents(...)) loads the entire file into memory (often causing Allowed Memory Size Exhausted errors), JSON Machine uses generators to load items one by one, maintaining a constant and small memory footprint regardless of file size.

    To use it, use JsonMachine\Items::fromFile() to create an iterable object that you can loop over with foreach.

  8. Parse streaming HTTP responses

    master

    To parse a stream API response or any other JSON stream, use Items::fromStream($streamResource).

    GuzzleHttp

    Convert Guzzle streams to PHP streams using \GuzzleHttp\Psr7\StreamWrapper::getResource() before passing them to Items::fromStream.

    Symfony HttpClient

    Since Symfony HttpClient responses work as iterators and JSON Machine is based on iterators, you can use them directly.

  9. Track parsing progress

    master

    For large documents, you can track progress by calling Items::getPosition() inside your foreach loop. This returns the current count of processed bytes from the beginning.

    Note: You must set the debug option to true for getPosition() to work. If debug is disabled, getPosition() always returns 0.

    <?php

use JsonMachine//

$fileSize = filesize('fruits.json');
$fruits = Items::fromFile('fruits.json', ['debug' => true]);
foreach ($fruits as $name => $data) {
    echo 'Progress: ' . intval($fruits->getPosition() / $fileSize * 100) . ' %'; 
}

  10. Use recursive parsing to descend into nested structures

    master
    By setting the $recursive parameter to true in the Parser constructor, the parser can handle nested objects or arrays by creating new Parser instances for those subtrees. This is useful when you want to treat nested structures as individual parsable units within a larger stream.

  11. Configure JSON Machine options

    master

    Options are passed as an array in the second parameter of all Items::from* functions. Available options include:

    • pointer: A JSON Pointer string that specifies which part of the document to iterate.
    • decoder: An instance of the ItemDecoder interface.
    • debug: A boolean (true or false). When true, data such as line, column, and position are available during parsing or in exceptions. Disabling debug mode provides a slight performance advantage.

  12. Troubleshoot memory exhaustion

    master

    If you encounter "Allowed memory size ... exhausted" errors, check the following:

    1. Missing JSON Pointer: Ensure you are using a pointer option if the items you want to iterate are nested under a specific key (e.g., "results").
    2. Large Individual Items: If a single item in the iteration is too large to be decoded at once, consider using Recursive iteration.
    3. Massive Scalar Strings: If a single JSON scalar string (like a massive base64-encoded file) is larger than your memory limit, it may not be possible to parse with current support.