FastCSV Documentation

repository·main·Indexed 20 days ago

https://github.com/osiegmar/fastcsv

A high-performance, lightweight, and RFC 4180 compliant CSV library for Java. FastCSV provides high-speed reading and writing with zero runtime dependencies and is compatible with Java 17+, Android 34+, GraalVM Native Image, and OSGi. It features flexible configuration via CsvReaderBuilder and CsvWriterBuilder for handling field separators, quote characters, encoding, null values, and comment strategies.

Tokens
14.8K
Snippets
44
Records
79
Agent score
70%

What's inside FastCSV

  1. Understand FastCSV's CSV interpretation grammar

    main

    FastCSV supports two modes of CSV interpretation based on whether comment handling is enabled.

    1. Default Mode (Comment handling disabled): FastCSV adheres to the ABNF grammar of RFC 4180-bis. In this mode, fields are either escaped (enclosed in DQUOTE) or non-escaped. Line breaks are recognized as CR, LF, or CRLF.

    2. Comment Mode (Comment handling enabled): FastCSV adheres to an extended grammar that allows for comments starting with a HASH (#) character. In this mode, a line starting with a HASH is treated as a comment. Fields within a record can contain data that includes a HASH character, provided they are handled according to the specific grammar rules for data-with-hash.

    ; Default Mode (RFC 4180-bis)
    file = [header] *(record)
    header = [field] *(COMMA field) linebreak
    record = [field] *(COMMA field) linebreak
    field = (escaped / non-escaped)
    escaped = DQUOTE *(textdata / COMMA / CR / LF / 2DQUOTE) DQUOTE
    non-escaped = *(textdata)
    
    ; Comment Mode
    file = *((comment / record) linebreak)
    comment = HASH *comment-data
    record = first-field *(COMMA field)
  2. Handle null values and empty fields

    main

    Since CSV does not natively distinguish between empty fields and null values, FastCSV provides the following mechanisms:

    On Writing

    Depending on your QuoteStrategy, null values are written as:

    • Empty fields (default).
    • Quoted empty fields (using QuoteStrategies.EMPTY). This is useful for PostgreSQL compatibility.
    • To write a specific string for a null value, pass that string directly to CsvWriter.writeRecord().

    On Reading

    To distinguish between an empty field and a null value, you can implement a custom CsvRecord that utilizes the quoted parameter provided to CsvCallbackHandler.addField() to check if the field was enclosed in quotes.

    1,,fooCRLF
    2,"",barCRLF
  3. Read CSV files with comments using CommentStrategy

    main

    When reading CSV files that contain comments, you can control how FastCSV treats those lines using the CommentStrategy enum. FastCSV provides three main behaviors:

    1. Treat as data (Default): Comments are treated as part of the data to ensure maximum compatibility with the RFC 4180 standard and prevent data loss.
    2. Skip comments: The comment lines are ignored during the reading process.
    3. Read as comments: The lines are explicitly interpreted as comments.

    To use a specific strategy, configure the CsvReaderBuilder.

  4. Field separators and encapsulation constraints

    main

    FastCSV allows you to configure the field separator (e.g., semicolon or tab) and the encapsulation character (e.g., single quotes) to support various real-world CSV formats.

    However, the following are not supported:

    • Multiple characters: Field separators and encapsulation characters must be a single character.
    • Mixed separators/encapsulation: You cannot use different separators or different encapsulation characters within the same file. The configured character is used for the entire file.
  5. Detect if a field is quoted when reading CSV

    main

    FastCSV does not provide a built-in way to check if a field was enclosed in quotes during the reading process. To distinguish between quoted and unquoted fields (which can be useful for differentiating between empty strings and null values), you must implement a custom callback handler using QuotableFieldHandler and QuotableField.

    // Implementation pattern for detecting quoted fields
    // Requires implementing a custom handler to intercept field metadata
    CsvReader reader = CsvReader.builder()
        .fieldHandler(new MyQuotableFieldHandler()) 
        .build(inputStream);
  6. Manage internal buffer flushing in CsvWriter

    main

    In FastCSV 4.x, CsvWriter (for both OutputStream and Writer) no longer flushes the internal buffer after every record. It only flushes when the buffer is full, or when flush() or close() is called.

    Best Practices:

    • Always call close() on the CsvWriter at the end of the process to ensure all data is written.
    • Call flush() manually if you need to write to the underlying Writer or OutputStream directly before closing.
    • You no longer need to wrap your Writer in a BufferedWriter for performance unless you have explicitly disabled FastCSV's internal buffer using CsvWriterBuilder.bufferSize(0).
  7. How CsvReader works

    main

    The CsvReader is the central entry point for reading CSV data. It uses a builder pattern (CsvReaderBuilder) to configure format specifications (like separators and quote characters) and parsing behavior.

    Internally, the CsvReader orchestrates a CsvParser to handle low-level data parsing and a CsvCallbackHandler to materialize fields into records. The CsvReader implements the Iterable interface, allowing you to iterate over parsed records (such as CsvRecord objects) using standard loops or forEach.

    try (CsvReader<CsvRecord> csv = CsvReader.builder().ofCsvRecord(file)) {
        csv.forEach(System.out::println);
    }
  8. How indexed reading works in FastCSV

    main

    FastCSV provides IndexedCsvReader for random access and paginated reading of large CSV files. Since CSV files lack a native index, FastCSV builds one in memory while reading the file. This index allows you to navigate to specific rows or pages without reading the entire file sequentially.

    Key Characteristics

    • Non-blocking Indexing: The indexing process runs in the background. You can start reading the file while the index is still being generated. A status monitor is available to track progress.
    • Memory Usage: The index is stored in the heap and scales with the record count, not the file size. It holds one page entry (approximately 40 bytes) per pageSize records.
      • Formula: index heap ≈ 40 bytes × ceil(recordCount / pageSize)
      • A smaller pageSize provides finer-grained random access but increases heap consumption.
    • Persistence: The index can optionally be stored in a file to avoid re-indexing the CSV on subsequent reads.
    // Index the CSV file with up to 5 records per page
    IndexedCsvReader<CsvRecord> csv = IndexedCsvReader.builder()
        .pageSize(5)
        .ofCsvRecord(file);