DOCSEARCHbyAlgolia
Submit Docs

csv-diff

repository·main·Indexed 43 days ago

https://github.com/simonw/csv-diff

A tool and Python library for viewing and comparing differences between two CSV, TSV, or JSON files. It provides a command-line utility for human-readable summaries and a programmatic interface for data comparison, supporting unique row identification via a key column and machine-readable JSON output.

Tokens
1.5K
Snippets
10
Records
10
Agent score
6%

What's inside csv-diff

  1. Understand the JSON output format

    main

    When using the --json flag, csv-diff returns a JSON object with the following structure:

    • added: An array of objects representing rows present in the second file but not the first.
    • removed: An array of objects representing rows present in the first file but not the second.
    • changed: An array of objects representing rows where values differ. Each object contains a key (the value of the unique identifier) and a changes object mapping column names to an array of [old_value, new_value].
    {
    "added": [
        { "id": "3", "name": "Bailey", "age": "1" }
    ],
    "removed": [
        { "id": "2", "name": "Pancakes", "age": "2" }
    ],
    "changed": [
        {
            "key": "1",
            "changes": {
                "age": ["4", "5"]
            }
        }
    ],
    "columns_added": [],
    "columns_removed": []
}

  2. Run csv-diff using Docker

    main

    You can run csv-diff in a containerized environment. You must mount the directory containing your files to /files inside the container.

    Build the image:

    docker build -t csvdiff .

    Run the container:

    # Run against files in the current directory
docker run --rm -v $(pwd):/files csvdiff one.csv two.csv
    docker run --rm -v $(pwd):/files csvdiff one.csv two.csv

  3. Use the csv-diff CLI to compare files

    main

    Run csv-diff followed by the paths to your two files. You must provide a --key option to specify the unique column used to identify records across files.

    Example usage:

    csv-diff one.csv two.csv --key=id

  4. Install csv-diff via pip

    main

    Install the csv-diff CLI tool using pip to compare differences between CSV, TSV, or JSON files.

    pip install csv-diff

  5. Use csv-diff as a Python library

    main

    You can integrate csv-diff into your Python projects by importing load_csv and compare. The compare function returns a dictionary matching the structure of the CLI's --json output.

    from csv_diff import load_csv, compare

diff = compare(
    load_csv(open("one.csv"), key="id"),
    load_csv(open("two.csv"), key="id")
)

  6. Configure csv-diff CLI options

    main

    Use the following flags to control the behavior and output of the csv-diff command:

    • --key=<column_name>: Specifies the unique column to identify records.
    • --format=<format>: Forces the input format. Supported values: csv, tsv, json. The tool attempts automatic detection otherwise.
    • --show-unchanged: Includes full details of unchanged values for rows that have at least one change.
    • --json: Outputs the difference in a machine-readable JSON format.
    • --extra <name> "<template>": Adds custom templated information to the human-readable output. The template is a Python format string where variables correspond to row values (e.g., {id}).
    # Example: Using --show-unchanged
csv-diff one.csv two.csv --key=id --show-unchanged

# Example: Using --json for machine-readable output
csv-diff one.csv two.csv --key=id --json

# Example: Adding templated extras
csv-diff one.csv two.csv --key=id --extra latest "https://news.ycombinator.com/latest?id={id}"

  7. Output changes as JSON with csv-diff

    main

    To get machine-readable output, use the --json flag. Note that when using --json, you cannot use the --extra option, as extra fields are only supported in human-readable text mode.

    csv-diff previous.csv current.csv --key id --json

  8. Customize human-readable output with --extra

    main

    You can define additional fields to display in the human-readable output using the --extra flag. This flag accepts a key:format_string pair. This feature is incompatible with the --json output mode.

    csv-diff previous.csv current.csv --key id --extra name:{name} --extra email:{email}

  9. Reference: csv-diff CLI options

    main

    The following options are available for the csv-diff command:

    OptionTypeDescription
    previousargumentPath to the previous file (must exist).
    currentargumentPath to the current file (must exist).
    --keystringColumn to use as a unique ID for each row.
    --formatchoiceExplicitly specify input format: csv, tsv, or json. (Defaults to auto-detecting).
    --jsonflagOutput changes as JSON instead of human-readable text.
    --singularstringSingular word to use in human-readable output (e.g., 'tree' for '1 tree').
    --pluralstringPlural word to use in human-readable output (e.g., 'trees' for '2 trees').
    --show-unchangedflagShow unchanged fields for rows that have at least one change.
    --extrakey:valueDefine extra fields to display using a key and a format string. (Note: Not supported when --json is used).
    csv-diff previous.csv current.csv --key id --format csv --json