DOCSEARCHbyAlgolia
Submit Docs

xsv

repository·master·Indexed 27 days ago

https://github.com/burntsushi/xsv

A high performance CSV command line toolkit for indexing, slicing, analyzing, splitting, and joining CSV files. Version 0.13.0 provides a suite of CLI tools including cat, count, fixlengths, flatten, fmt, frequency, headers, index, join, partition, sample, reverse, search, select, slice, sort, split, stats, and table, designed for composability and handling very large CSV datasets.

Tokens
12.5K
Snippets
31
Records
57
Agent score
95%

What's inside xsv

  1. Speed up xsv operations with an index

    master

    Creating an index for a CSV file makes operations like stats and slice significantly faster. For example, slice operations become instantaneous because only the sliced portion is parsed.

    To create an index:

    xsv index filename.csv
    xsv index worldcitiespop.csv

  2. Install xsv

    master

    You can install xsv using various package managers or by compiling from source.

    macOS (Homebrew):

    brew install xsv

    macOS (MacPorts):

    sudo port install xsv

    Nix/NixOS:

    nix-env -i xsv

    Cargo (Rust package manager):

    cargo install xsv

    Build from source:

    git clone git://github.com/BurntSushi/xsv
cd xsv
cargo build --release

    Note: The binary will be located in ./target/release/xsv after building.

    brew install xsv

  3. Format CSV output with xsv table

    master

    The xsv table command takes any CSV data and formats it into aligned columns using elastic tabstops, which handles Unicode character alignment correctly.

    Example usage:

    xsv stats data.csv --everything | xsv table
    xsv stats worldcitiespop.csv --everything | xsv table

  4. Join CSV files with xsv join

    master

    Perform inner, outer, or cross joins. By default, xsv join performs an inner join. You can use the --no-case flag for case-insensitive joins.

    Syntax: xsv join [options] <column_in_file1> <file1> <column_in_file2> <file2>

    Example (joining sample.csv with countrynames.csv on the Abbrev column):

    xsv join --no-case Country sample.csv Abbrev countrynames.csv
    xsv join --no-case Country sample.csv Abbrev countrynames.csv

  5. Select and re-order columns with xsv select

    master

    Use xsv select to pick specific columns or re-order them. You can use special syntax to remove columns:

    • !column_name: Removes the specified column.
    • column_name[index]: Selects a specific occurrence of a column name (e.g., Country[1] selects the second occurrence of 'Country').

    Example to remove Abbrev and the second Country column:

    xsv select '!Abbrev,Country[1]' data.csv

  6. Reference the xsv available commands

    master

    The following commands are available in the xsv CLI:

    • cat - Concatenate CSV files by row or by column.
    • count - Count the rows in a CSV file. (Instantaneous with an index.)
    • fixlengths - Force a CSV file to have same-length records by either padding or truncating them.
    • flatten - A flattened view of CSV records. Useful for viewing one record at a time.
    • fmt - Reformat CSV data with different delimiters, record terminators or quoting rules.
    • frequency - Build frequency tables of each column in CSV data.
    • headers - Show the headers of CSV data or the intersection of headers between files.
    • index - Create an index for a CSV file for constant time indexing.
    • input - Read CSV data with exotic quoting/escaping rules.
    • join - Inner, outer and cross joins using a hash index.
    • partition - Partition CSV data based on a column value.
    • sample - Randomly draw rows using reservoir sampling.
    • reverse - Reverse order of rows.
    • search - Run a regex over CSV data, applying it to each field individually.
    • select - Select or re-order columns.
    • slice - Slice rows from any part of a CSV file (extremely fast with an index).
    • sort - Sort CSV data.
    • split - Split one CSV file into many CSV files of N chunks.
    • stats - Show basic types and statistics (mean, median, etc.) of each column.
    • table - Show aligned output using elastic tabstops.

  9. Configure CSV parsing with the Config struct

    master

    The Config struct is used to manage settings for CSV reading and writing in xsv. You can initialize it using Config::new(path) and then chain configuration methods to set delimiters, quoting styles, headers, and more.

    Key Configuration Methods:

    • delimiter(Option<Delimiter>): Sets the field delimiter. Use the Delimiter type to handle special cases like tabs.
    • no_headers(bool): Specifies whether the data contains headers. Note: if the environment variable XSV_TOGGLE_HEADERS is set to 1, this value will be toggled.
    • flexible(bool): Enables flexible parsing (allows varying number of fields per record).
    • crlf(bool): Sets the line terminator to CRLF if true, otherwise defaults to \n.
    • quote(u8): Sets the quote character.
    • quote_style(csv::QuoteStyle): Sets the quoting style.
    • double_quote(bool): Enables/disables double quoting.
    • escape(Option<u8>): Sets an escape character.
    • quoting(bool): Enables/disables quoting.
    • select(SelectColumns): Configures column selection logic.

  10. Create CSV readers and writers from Config

    master

    Once a Config is initialized, you can use it to generate csv::Reader and csv::Writer instances configured with your settings.

    • reader(): Returns an io::Result<csv::Reader<Box<io::Read+'static>>> using the configured input (file or stdin).
    • writer(): Returns an io::Result<csv::Writer<Box<io::Write+'static>>> using the configured output (file or stdout).
    • from_reader<R: Read>(rdr: R): Creates a csv::Reader from any type implementing Read using the Config settings.
    • from_writer<W: io::Write>(wtr: W): Creates a csv::Writer from any type implementing Write using the Config settings.

  11. Use the Indexed type for random access CSV reading

    master

    The Indexed<R, I> struct allows for efficient random access to CSV records by composing a csv::Reader<R> with a RandomAccessSimple<I> index. It implements Deref and DerefMut to csv::Reader<R>, meaning you can use it directly as a standard CSV reader.

    To use it, you must provide both a CSV reader and an index reader that implement io::Read + io::Seek.

  12. Handle indexed CSV files

    master

    The Config struct supports working with indexed CSV files to speed up operations.

    • index_files(): Attempts to find and open an index file associated with the configured CSV path. It checks if the CSV file was modified after the index file; if so, it returns an error requiring the index to be re-created.
    • indexed(): Returns an Option<Indexed<fs::File, fs::File>> if an index is found and valid.