DOCSEARCHbyAlgolia
Submit Docs

Miller Documentation

repository·main·Indexed 27 days ago

https://github.com/johnkerl/miller

A powerful, streaming command-line tool for manipulating structured data formats including CSV, TSV, JSON, and JSON Lines. Miller provides a format-aware replacement for traditional Unix text processing tools, featuring a domain-specific language (DSL) for record manipulation, an interactive REPL, and a streaming architecture built with Go. It supports complex data structures via Mlrval and Mlrmap, and offers advanced capabilities for format conversion and record transformation.

Tokens
149.6K
Snippets
353
Records
965
Agent score
91%

What's inside Miller

  1. Overview of Miller Verbs

    main

    Miller uses verbs (subcommands) to process data, similar to how the Unix toolkit uses separate executables like cat, sort, or cut. Verbs are invoked as mlr <verb> [options] [files].

    Verbs are categorized into several functional groups:

    • Unix-toolkit analogs: cat, cut, grep, head, join, sort, tac, tail, top, uniq.
    • awk-like functionality: filter, put, sec2gmt, sec2gmtdate, step, tee.
    • Statistical orientation: bar, bootstrap, decimate, histogram, least-frequent, most-frequent, rank, sample, shuffle, sparkline, stats1, stats2.
    • Record Heterogeneity orientation: group-by, group-like, having-fields.
    • Other specialized verbs: check, count-distinct, label, merge-fields, nest, nothing, regularize, rename, reorder, reshape, seqgen.

    You can explore available verbs directly from the command line using:

    • mlr -l (list verbs)
    • mlr -L (list verbs with descriptions)

  2. Overview of Miller capabilities

    main

    Miller is a command-line tool designed for querying, shaping, and reformatting data files. It supports multiple formats including CSV, TSV, JSON, JSON Lines, YAML, and DCF.

    Key capabilities include:

    • Format conversion: Converting between formats (e.g., CSV to JSON) or pretty-printing data horizontally/vertically.
    • Data manipulation: Removing or creating columns.
    • Data processing: Cleaning data for database ingestion, post-processing database query output, or interactive data exploration.
    • Verbs and Logic: Using compact 'verbs' (analogous to standard Unix tools like sort or head) or the put verb to execute custom programming-language expressions for complex logic.

  3. Overview of Miller record transformers

    main
    Miller uses a transformer system to manipulate input records into requested output formats. This includes operations such as sorting, filtering, and reordering fields. The system is built around the IRecordTransformer interface and supports chaining multiple transformations together using the ChainTransformer mechanism, which pipes the output of one transformer into the next via Go channels.

  4. Overview of Miller features and data formats

    main

    Miller is a high-performance, streaming data processing tool designed for structured data. It functions similarly to Unix tools like awk, sed, cut, join, and sort, but is specifically optimized for key-value-pair data.

    Supported Data Formats

    • CSV (Comma-Separated Values)
    • TSV (Tab-Separated Values)
    • JSON
    • JSON Lines
    • Positionally-indexed (Unix-style integer-indexed fields)
    • Tabular pretty-printing

    Key Capabilities

    • Format Conversion: Easily convert between the formats listed above.
    • Named Fields: Operate on data using field names rather than counting positional indices.
    • Streaming Processing: Most operations process one record at a time, allowing you to handle files larger than available RAM and use Miller in tail -f contexts.
    • Record Heterogeneity: Miller can handle data where interleaved records have different schemas (different field names).
    • Format-Awareness: Operations like sort and tac respect format-specific structures (e.g., keeping CSV header lines at the top).
    • Zero Runtime Dependencies: Written in Go, it is a single portable binary.

  5. Overview of Miller features and capabilities

    main

    Miller is a data processing tool designed for name-indexed data (key-value pairs) rather than positional indices. It functions similarly to awk, sed, cut, join, and sort but operates on structured data formats.

    Supported Data Formats

    Miller supports a variety of formats, including:

    • CSV
    • TSV
    • JSON
    • JSON Lines
    • YAML
    • DCF
    • Tabular pretty-printing
    • Positionally-indexed data (Unix-style)

    Key Capabilities

    • Format Conversion: Easily convert between different data formats (e.g., JSON to CSV).
    • Streaming Processing: Most operations are streaming, meaning they process one record at a time. This allows Miller to handle files larger than available RAM and work in tail -f contexts.
    • Record Heterogeneity: Miller can process data where interleaved records have different schemas (different field names).
    • Format Awareness: Operations like sort and tac are aware of the data format (e.g., keeping CSV header lines in place).
    • High Performance: Written in Go, Miller offers high-throughput performance comparable to the Unix toolkit and has no runtime dependencies, making it a portable single binary.

  6. Understand Miller data types and variables

    main
    In Miller, a value is the data indexed by a key in a map. Values follow Miller's specific data types. Variables allow you to access data by name within the Miller programming language. The keyword var can be used for type declarations, allowing you to declare a new variable in an inner scope with the same name as one in an outer scope.

  7. Understand Miller's error propagation and exit behavior

    main

    Miller is transitioning from scattered os.Exit calls to a centralized error propagation model. This ensures that library code can be used safely without abruptly terminating the process, allows for proper deferred cleanup (like pprof), and enables features like the DSL exit statement and strict mode.

    Key behaviors:

    • Centralized Exit: os.Exit is primarily handled in entrypoint.Main. Errors propagate up the stack to this point.
    • Exit Mapping:
      • nil error $\rightarrow$ exit code 0.
      • cli.ErrHelpRequested $\rightarrow$ exit code 0.
      • cli.ErrUsagePrinted $\rightarrow$ exit code 1.
      • All other errors $\rightarrow$ print error and exit code 1.
    • Structured Errors: The --errors-json flag provides structured error emission. The transition to returning errors ensures that stream-time errors are correctly captured and categorized.

  8. Core Capabilities of Miller

    main

    Miller is a command-line tool designed for processing tabular data across multiple formats. Its core workflow follows a three-step paradigm:

    1. Ingest: Read a stream of records where each record is a collection of key-value pairs (supporting formats like CSV, TSV, JSON, etc.).
    2. Transform: Apply transformations using built-in verbs or a Domain-Specific Language (DSL).
    3. Emit: Output the transformed stream in the original format or convert it to a different format.

    Key technical characteristics include:

    • Streaming: Processes data one record at a time, allowing it to handle files larger than available RAM and work with piped input from other processes.
    • Multi-format: Interoperable support for various structured data formats.
    • Performance: Built in Go for high-speed processing of large datasets.
    • Hybrid Interface: Combines high-level verbs (for common tasks like stats) with a powerful DSL (for arbitrary logic).

  9. Miller Error Handling and Exit Behavior

    main

    Miller is transitioning from scattered os.Exit calls to a centralized error propagation model. For end-users, the following behavioral invariants apply:

    • Exit Codes: Standard errors (unknown verbs, flags, or syntax errors) continue to result in an exit code of 1.
    • Successful Commands: Commands like mlr --version or mlr put -e "valid" return exit code 0.
    • DSL Runtime Errors: Errors occurring within the Domain Specific Language (DSL) during execution (e.g., misuse of Higher Order Functions) will surface as mlr: <message> via error propagation.
    • REPL Behavior: In the mlr repl environment, DSL errors will continue to use the 'print-and-continue' behavior, allowing the user to correct the command without the session terminating.
    • Broken Pipes: Miller maintains existing behavior for broken pipes (e.g., mlr cat big | head -1).

  10. Understand Miller DSL variable types

    main

    Miller uses several types of variables within its Domain Specific Language (DSL):

    • Fields of stream records ($): Refer to fields in the current data record (e.g., $x, $y). Use $* to refer to the entire record. These are read-write and their extent is limited to the current record.
    • Out-of-stream variables (@): Persist across records (e.g., @sum). They are stored in nested maps and can be used in begin and end blocks to track values like sums or counters. Their extent is the entire record stream.
    • Local variables: Limited to the current statement's scope (e.g., function arguments, loop variables).
    • Built-in variables: Read-only constants like NF, NR, FILENAME, M_PI, and M_E.

  11. Internationalization support in Miller

    main

    Miller supports ASCII and UTF-8 strings. While Miller's internal commands, verbs, and help text are in English, you can use UTF-8 for filenames, field names, string literals, and variable names.

    Key internationalization features include:

    • Correct Alignment: Tabular output formats like pprint and xtab align UTF-8 characters correctly.
    • UTF-8 Aware Functions:
      • strlen counts UTF-8 codepoints rather than bytes.
      • toupper, tolower, and capitalize operate using Go's standard library capabilities for UTF-8.
    • UTF-8 Data: You can process CSV/TSV files containing non-ASCII characters in headers and data rows.