DOCSEARCHbyAlgolia
Submit Docs

jsongrep

repository·main·Indexed 20 days ago

https://github.com/micahkepe/jsongrep

A high-performance command-line tool and Rust library (v0.9.0) for querying JSON, YAML, and TOML using a JSONPath-inspired query language. It utilizes a Deterministic Finite Automaton (DFA) for efficient declarative path matching, supporting operators such as sequences, unions, wildcards, and repetitions. The project includes a WASM implementation for browser-based queries and a programmatic API via QueryBuilder and QueryDFA.

Tokens
10.8K
Snippets
40
Records
48
Agent score
71%

What's inside jsongrep

  1. Understand jsongrep benchmark groups

    main

    The benchmark suite is divided into four distinct measurement groups to isolate different performance characteristics:

    1. document_parse: Measures the time taken to parse raw JSON into the tool's in-memory representation (e.g., serde_json_borrow::Value for jsongrep).
    2. query_compile: Measures the cost of constructing the query engine (e.g., DFA construction for jsongrep or expression compilation for jmespath).
    3. query_search: The core performance metric. It measures only the traversal/execution time by using pre-compiled queries and pre-parsed documents.
    4. end_to_end: Simulates a real CLI usage by running the full pipeline (parse + compile + search) without any caching.

  2. How jsongrep path matching works

    main

    Unlike jq, which uses a filter pipeline to transform data, jsongrep is declarative. You describe the sets of paths you want to match using regular expression operators. The engine compiles your query into a Deterministic Finite Automaton (DFA) to find matching paths efficiently.

    Common path patterns include:

    • **.name: Kleene star; matches name under any nested object.
    • users[*].email: Wildcard; matches all email fields in the users array.
    • (error|warn).*: Disjunction; matches any field starting with error or warn.
    • (* | [*])*.name: Matches name through any combination of objects and arrays at any depth.
    # Example of disjunction matching multiple fields
curl -s https://api.nobelprize.org/v1/prize.json | jg 'prizes[0].(year|category)'

  3. Understand jsongrep Query Syntax

    main

    jsongrep queries are regular expressions applied to JSON paths. You can use sequences, unions, wildcards, and repetitions to navigate JSON structures. Queries can be nested using parentheses.

    Supported Operators

    OperatorExampleDescription
    Sequencefoo.bar.bazConcatenation: match path foo $\rightarrow$ bar $\rightarrow$ baz
    Disjunctionfoo | barUnion: match either foo or bar
    Kleene star**Match zero or more field accesses
    Repetitionfoo*Repeat the preceding step zero or more times
    Wildcards* or [*]Match any single field or array index
    Optionalfoo?.barOptional foo field access
    Field accessfoo or "foo bar"Match a specific field (quote if spaces)
    Array index[0] or [1:3]Match specific index or slice (exclusive end)

    Advanced Patterns

    • Recursive Descent: To find a field at any depth, use (* | [*])*. For example, (* | [*])*.foo finds all paths where foo exists at any level.
    • Nesting: Use parentheses for grouping, e.g., foo.(bar|baz).qux matches foo.bar.qux or foo.baz.qux.
    WARNING

    The /regex/ syntax for matching field names by pattern is currently reserved and will result in an "Unsupported feature" error.

    | Operator     | Example              | Description                                                                                  |
| ------------ | -------------------- | ---------------------------------------------------------------------------------------------|
| Sequence     | `foo.bar.baz`        | **Concatenation**: match path `foo` $\rightarrow$ `bar` $\rightarrow$ `baz`                         |
| Disjunction  | `foo \| bar`         | **Union**: match either `foo` or `bar`                                                        |
| Kleene star  | `**`                 | Match zero or more field accesses                                                            |
| Repetition   | `foo*`               | Repeat the preceding step zero or more times                                                  |
| Wildcards    | `*` or `[*]`         | Match any single field or array index                                                         |
| Optional     | `foo?.bar`           | Optional `foo` field access                                                                  |
| Field access | `foo` or `"foo bar"` | Match a specific field (quote if spaces)                                                    |
| Array index  | `[0]` or `[1:3]`     | Match specific index or slice (exclusive end)                                                  |

  4. Query multiple formats with jsongrep

    main

    jsongrep supports multiple serialization formats. It auto-detects the format from the file extension or allows you to specify it explicitly using the -f, --format flag. Non-JSON formats are converted to JSON internally before querying.

    Supported Formats:

    FormatExtensionsFeature flagNotes
    JSON.json—Default
    JSONL/NDJSON.jsonl, .ndjson—Wrapped into JSON array
    YAML.yaml, .ymlyamlIncluded by default
    TOML.tomltomlIncluded by default
    CBOR.cborcborIncluded by default
    MessagePack.msgpack, .mpmsgpackIncluded by default

    Examples:

    • Querying TOML: jg 'dependencies.*.version' Cargo.toml
    • Querying YAML via STDIN: cat config.yaml | jg -f yaml 'database.host'
    • Querying JSONL: jg '[*].email' users.jsonl
    cat config.yaml | jg -f yaml 'database.host'

  5. Quick start with jsongrep

    main

    Use jg to extract specific paths from JSON data via STDIN or files. The tool returns the matched value along with its path header by default in a terminal.

    Extracting from an API response:

    curl -s https://api.nobelprize.org/v1/prize.json | jg 'prizes[0].laureates[*].firstname'

    Extracting from inline JSON:

    echo '{"users": [{"name": "Alice"}, {"name": "Bob"}]}' | jg 'users.[*].name'

  8. Use jsongrep as a Rust Library

    main

    To use jsongrep in your Rust project, add it to your Cargo.toml. You can query JSON using a simple one-liner, or for better performance in repeated queries, compile a QueryDFA once and reuse it. You can also build queries programmatically using QueryBuilder.

    // Add to Cargo.toml
[dependencies]
jsongrep = "0.9.0"

// One-liner usage
let json: jsongrep::Value = serde_json::from_str(r#"{"users": [{"name": "Alice"}]}"#)?;
let results = jsongrep::grep(&json, "users[*].name")?;

for result in &results {
    println!("{:?}: {}", result.path, result.value);
}

// Optimized usage for repeated queries
use jsongrep::query::QueryDFA;
let dfa = QueryDFA::from_query_str("users[*].name")?;
let results = dfa.find(&json);

// Programmatic query building
use jsongrep::query::{QueryBuilder, QueryDFA};
let query = QueryBuilder::new()
    .field("users")
    .array_wildcard()
    .field("name")
    .build();
let dfa = QueryDFA::from_query(&query);
let results = dfa.find(&json);

  9. Generate Shell Completions for jsongrep

    main

    You can generate shell completion scripts for bash, zsh, or fish using the jg generate shell <SHELL> command.

    # Bash
jg generate shell bash > /etc/bash_completion.d/jg.bash

# Zsh
jg generate shell zsh > ~/.zsh/completions/_jg

# Fish
jg generate shell fish > ~/.config/fish/completions/jg.fish

  10. Run jsongrep benchmarks

    main

    You can run the jsongrep benchmark suite using cargo bench or the provided justfile. The benchmarks use the Criterion library to provide statistical rigor and compare jsongrep against other tools like jsonpath-rust, jmespath, jaq, and jql.

    To view the results, open the generated HTML report in your browser.

    # Using cargo
cargo bench --bench query

# Using just
just bench

  11. Download XLarge test data for benchmarking

    main

    While Small, Medium, and Large test datasets are included in the repository via include_str!, the XLarge dataset (citylots.json, ~190 MB) is loaded from disk at runtime. If the file is missing, the benchmark will silently skip it. Use the following command to download the required data:

    just bench-download