DOCSEARCHbyAlgolia
Submit Docs

CleverCSV

repository·master·Indexed 23 days ago

https://github.com/alan-turing-institute/clevercsv

A Python package for handling messy CSV files, designed as a robust replacement for the standard csv module. It provides improved dialect detection via the Sniffer and Detector classes, high-level wrappers for loading data into lists or Pandas DataFrames, and a command-line interface for detecting dialects, generating import code, and standardizing files to RFC-4180.

Tokens
14.3K
Snippets
15
Records
113
Agent score
79%

What's inside clevercsv

  1. Explore the CleverCSV module structure

    master

    CleverCSV is organized into several specialized modules for handling different aspects of CSV processing. Key modules include:

    • clevercsv.read and clevercsv.write: Primary interfaces for reading from and writing to CSV files.
    • clevercsv.detect: Tools for detecting CSV dialects and formats.
    • clevercsv.dialect: Logic for handling CSV dialects.
    • clevercsv.dict_read_write: Reading and writing CSV data using dictionary-like interfaces.
    • clevercsv.encoding: Handling file encodings.
    • clevercsv.exceptions: Custom exception classes for error handling.
    • clevercsv.consistency: Modules for ensuring data consistency.
    • clevercsv.break_ties: Logic for resolving ambiguities during detection.
    • clevercsv.detect_pattern and clevercsv.detect_type: Specialized detection for patterns and data types.
    • clevercsv.normal_form: Tools for normalizing CSV data.
    • clevercsv.potential_dialects: Identifying possible dialects in a file.

  2. Use CleverCSV CLI commands for data processing

    master

    The clevercsv.console.commands package provides a suite of command-line interface (CLI) tools for handling problematic CSV files. The available command submodules include:

    • detect: Used to detect the dialect (delimiter, quoting, etc.) and structure of a CSV file.
    • explore: Used to explore the contents and structure of a CSV file.
    • standardize: Used to transform a problematic CSV file into a standardized, clean format.
    • view: Used to view the contents of a CSV file in a formatted way.
    • code: Used to generate Python code to import and process a specific file.

  3. Understand the behavior of the date regular expression

    master
    The date regular expression used in CleverCSV is designed to validate whether a string matches a recognized date format, rather than validating if the date itself is chronologically valid. For example, a string like 2019-02-31 will be considered a valid match because it follows a recognized date format, even though February 31st is not a real date. This design choice prioritizes execution speed and simplicity.

  5. Use CleverCSV as a drop-in replacement for the Python CSV module

    master

    CleverCSV is designed to be a drop-in replacement for Python's built-in csv module. You can import it and use its reader and Sniffer classes similarly to the standard library. For large files, you can optimize dialect detection by passing a sample of the file to the sniffer instead of the entire content.

    import clevercsv

with open("data.csv", "r", newline="") as fp:
  # You can use verbose=True to see what CleverCSV does
  dialect = clevercsv.Sniffer().sniff(fp.read(), verbose=False)
  fp.seek(0)
  reader = clevercsv.reader(fp, dialect)
  rows = list(reader)

  6. Integrate CleverCSV standardization into git via pre-commit

    master

    To ensure CSV files in your repository always conform to RFC-4180, you can use the clevercsv-standardize pre-commit hook.

    1. Install pre-commit.
    2. Add the following to your .pre-commit-config.yaml:
    repos:
  - repo: https://github.com/alan-turing-institute/CleverCSV-pre-commit
    rev: v0.6.6   # or any later version
    hooks:
      - id: clevercsv-standardize
    1. Run pre-commit install.

  7. Quick Start with the CleverCSV Python package

    master

    You can use CleverCSV to load messy CSV files as lists of rows or Pandas DataFrames. It also provides a Sniffer class that acts as a drop-in replacement for the standard Python csv.Sniffer to detect dialects in files that the standard library might fail to parse.

    # Import the package
import clevercsv

# Load the file as a list of rows
rows = clevercsv.read_table('./imdb.csv')

# Load the file as a Pandas Dataframe
# Note that df = pd.read_csv('./imdb.csv') would fail here
df = clevercsv.read_dataframe('./imdb.csv')

# Use CleverCSV as drop-in replacement for the Python CSV module
# This follows the Sniffer example: https://docs.python.org/3/library/csv.html#csv.Sniffer
# Note that csv.Sniffer would fail here
with open('./imdb.csv', newline='') as csvfile:
    dialect = clevercsv.Sniffer().sniff(csvfile.read())
    csvfile.seek(0)
    reader = clevercsv.reader(csvfile, dialect)
    rows = list(reader)

  8. Detect and handle file encodings in clevercsv-standardize

    master

    By default, clevercsv-standardize uses chardet to automatically detect the encoding of the input CSV file.

    If detection fails or is incorrect, you can use the following flags:

    • Use -e or --encoding to specify the input encoding (and the output encoding if -E is not provided).
    • Use -E or --target-encoding to specify a specific encoding for the resulting standardized file.

    If you are processing multiple files and providing manual encodings via -e, the number of provided encodings must match the number of input files.

  10. Explore a CSV file with clevercsv-explore

    master

    The clevercsv explore command allows you to quickly inspect a CSV file by dropping you into an interactive Python shell (REPL) with the file already loaded. The command automatically detects the file's dialect.

    By default, the file is loaded as a list of lists.

    To start exploring a file as a list of lists, run:

    clevercsv explore FILE

  12. Use CleverCSV wrapper functions for automatic detection

    master

    CleverCSV provides several high-level wrapper functions that automatically detect the dialect and encoding, making CSV handling easier:

    • detect_dialect(path): Returns the detected dialect for a given file path.
    • read_table(path): Returns the data as a list of rows.
    • stream_table(path): Returns a generator that yields rows.
    • read_dataframe(path): Returns a Pandas DataFrame (requires pandas installed).
    • read_dicts(path): Returns rows as dictionaries (assumes first row is header).
    • stream_dicts(path): A streaming version of read_dicts.
    • write_table(path, data): Writes a list of lists to a file using the RFC-4180 dialect.
    • write_dicts(path, data): Writes a list of dictionaries to a file using the RFC-4180 dialect.