DOCSEARCHbyAlgolia
Submit Docs

SmarterCSV Documentation

repository·main·Indexed 23 days ago

https://github.com/tilo/smarter_csv

A high-performance Ruby library for CSV ingestion and generation. It produces Rails-ready hashes with symbol keys and automatic type conversions, utilizing a C extension for accelerated parsing. Key features include memory-efficient batch processing via chunk_size, support for non-seekable streaming inputs (STDIN, HTTP responses), high-precision decimal handling with BigDecimal, and robust error management through bad row quarantine options.

Tokens
46.6K
Snippets
127
Records
220
Agent score
79%

What's inside SmarterCSV

  1. Introduction to SmarterCSV

    main

    SmarterCSV is a Ruby gem designed for high-performance and convenient CSV importing and exporting. Unlike Ruby's built-in CSV library which returns arrays of arrays, SmarterCSV returns 'Rails-ready' hashes with symbol keys, automatic numeric conversion, and whitespace stripping. This makes it ideal for direct use with ActiveRecord, insert_all, Sidekiq, or parallel processing workflows.

    Key capabilities include:

    • High Performance: Uses a C extension to accelerate parsing, hash construction, and value conversion.
    • Robustness: Auto-detects row/column separators and handles common issues like BOMs and inconsistent whitespace.
    • Batch Processing: Supports memory-efficient chunked processing via chunk_size.
    • Data Transformation: Provides built-in support for header remapping, column selection, and custom value converters.

  2. Use post-mapping keys for column selection

    main

    When using key_mapping: to rename headers, the headers: { only: } and headers: { except: } options must use the post-mapping names (the symbols that appear in the final result hash), not the original CSV header names.

    # CSV has header "First Name"; key_mapping renames it to :given_name
data = SmarterCSV.process('contacts.csv', 
  key_mapping:  { first_name: :given_name },
  headers:      { only: [:given_name] },   # Use the post-mapping name
)
# => [{given_name: "Alice"}, ...]

  3. Handle high-precision decimals without data loss

    main

    Unlike Ruby's standard CSV library which uses Float() and can cause silent precision loss for scientific or financial data, SmarterCSV uses a default decimal_precision: :auto setting.

    • Default behavior: Returns a BigDecimal for values exceeding 16 significant digits to ensure full precision.
    • Float behavior: Returns a Float for values within the 16-digit threshold. Floats are decoded using the Eisel-Lemire algorithm for bit-for-bit identity with String#to_f.
    • Explicit Float: If you require Float output even for high-precision numbers (matching Ruby's standard behavior), pass decimal_precision: :float in your options.

  4. How SmarterCSV's `:auto` quote escaping works

    main

    When quote_escaping is set to :auto (the default), SmarterCSV uses a two-step process to handle different escaping conventions:

    1. Multiline Detection: It performs a single pass using dual counting. It computes both a backslash-aware quote count and an RFC (plain) quote count. A line is only considered multiline if both counts are odd. This prevents false multiline stitching when a field ends with \".
    2. Parsing: It attempts the backslash-escape interpretation first. If the parser encounters a MalformedCSV error (indicating an unclosed quote), it retries using the RFC 4180 (doubled quotes) interpretation.

    This fallback mechanism is applied per-line, meaning a single file can contain rows using different escaping conventions and still be parsed correctly.

  5. Access errors via Class-level vs Reader API

    main

    Class-level API (SmarterCSV.errors)

    Returns errors from the most recent call to process, parse, each, or each_chunk on the current thread.

    • Warning: In multi-threaded environments (Puma, Sidekiq), errors are thread-local.
    • Warning: In fiber-based environments (Async, Falcon), SmarterCSV.errors uses Thread.current and may return stale results. Use the Reader API for fiber safety.

    Reader API (SmarterCSV::Reader)

    Use SmarterCSV::Reader directly for full control and thread/fiber safety. Errors are scoped to the specific reader instance.

    AttributeDescription
    reader.errors[:bad_row_count]Total bad rows encountered (all modes)
    reader.errors[:bad_rows]Array of error records (:collect mode only)
    # Reader API is safer for concurrent/fiber environments
reader = SmarterCSV::Reader.new('data.csv', on_bad_row: :collect)
reader.process
puts reader.errors[:bad_row_count]
puts reader.headers.inspect

  6. Understand the Value Transformation Pipeline

    main

    After a row is parsed, SmarterCSV applies transformations to field values in a specific sequence. Understanding this order is critical when using value_converters or nil_values_matching.

    StepOptionDefaultDescription
    1strip_whitespacetrueStrips leading/trailing whitespace from values and headers
    2nil_values_matchingnilSets values matching the regexp to nil
    3remove_empty_valuestrueRemoves keys whose value is nil or blank
    4remove_zero_valuesfalseRemoves keys whose value is numeric zero
    5convert_values_to_numerictrueConverts numeric-looking strings to Integer or Float
    6value_convertersnilApplies custom converter lambdas or classes
    7remove_empty_hashestrueDrops rows that are entirely empty after transformations

    Important: value_converters receive the value after numeric conversion (Step 5). Guard against Integer/Float input in your converters if necessary.

  7. Handle quoting, escaping, and delimiters

    main

    SmarterCSV is designed to handle the inconsistencies of real-world quoting and delimiters.

    Key Features:

    • Automatic Delimiter Detection: Use col_sep: :auto (default) to automatically detect separators like semicolons (common in Europe) or tabs (TSV files).
    • Quoting Conventions: The default :auto mode for quote_escaping handles both RFC 4180 (Excel style "") and backslash escaping (\") used by MySQL and PostgreSQL.
    • Mid-field Quotes: With quote_boundary: :standard (the default), quotes appearing in the middle of a field (e.g., 5'10") are treated as literal characters rather than field boundaries.
    • Multi-line Fields: Newlines inside quoted fields are automatically stitched into a single field.

  8. Use `required_keys` with `key_mapping`

    main

    Because required_keys validation runs after header transformations, you must use the mapped names when using key_mapping. If you map a CSV header acct_from to :source_account, your required_keys array should contain :source_account.

    options = {
  key_mapping:   { acct_from: :source_account, acct_to: :destination_account },
  required_keys: [:source_account, :destination_account, :amount],
}

  9. Apply per-key and global value converters

    main

    Use the value_converters option to transform data during serialization.

    Per-key Converters

    Pass a hash where keys match your data keys and values are lambdas. Each lambda receives the field value and returns the string to write.

    Global Converters (:_all)

    Use the special key :_all to define a transformation applied to every field. Global converters run after per-key converters.

    Note on Quoting: If you use :_all to manually handle quoting, you must also set the top-level option disable_auto_quoting: true to avoid double-quoting.

    # Per-key converter example
SmarterCSV.generate('output.csv', value_converters: { active: ->(v) { v ? 'YES' : 'NO' } }) do |csv|
  csv << { name: 'Alice', active: true  }
end

# Global converter example
SmarterCSV.generate('output.csv', value_converters: { _all: ->(_k, v) { v.is_a?(String) ? v.strip : v } }) do |csv|
  csv << { name: '  Alice  ', city: ' NYC ' }
end

# Combining per-key and global
options = {
  value_converters: {
    active:   ->(v) { v ? 'YES' : 'NO' },
    _all:     ->(_k, v) { v.to_s.upcase },
  }
}
SmarterCSV.generate('output.csv', options) do |csv|
  csv << { name: 'Alice', city: 'nyc', active: true }
end

  10. Understand the SmarterCSV Data Transformation Pipeline

    main

    SmarterCSV automatically normalizes values in each row through a configurable pipeline. Transformations run in a specific order for every row. Understanding this order is critical, especially when using custom converters, as they receive values after numeric conversion has already occurred.

    Transformation Order:

    1. strip_whitespace: Strips leading/trailing whitespace from headers and values.
    2. nil_values_matching: Converts values matching a regex to nil.
    3. remove_empty_values: Removes keys with nil or blank values.
    4. remove_zero_values: Removes keys with numeric zero.
    5. convert_values_to_numeric: Converts numeric-looking strings to Integer or Float.
    6. value_converters: Applies custom lambdas or classes per key.
    7. remove_empty_hashes: Drops rows that are entirely empty after all previous steps.

  11. Configure automatic detection of row and column separators

    main

    SmarterCSV can automatically detect column and row separators using the default settings col_sep: :auto and row_sep: :auto. This is useful for processing files where the format is unknown, such as user uploads.

    • Column Separator Detection: Considers ,, \t, ;, :, and |.
    • Row Separator Detection: Considers \n, \r\n, and \r.

    To control the initial scan size used for detection, use the :auto_row_sep_chars option. The default is 4096 bytes. Detection stops once a separator has a clear majority, up to a 64KB cap. If you provide an out-of-range value, nil, or 0, it will fall back to the default with a warning.

  12. Use Value Converters to transform CSV data

    main

    Value converters allow you to transform raw CSV strings into specific Ruby types (like Date, Float, or custom objects) during the parsing process. Converters run per-key after SmarterCSV has parsed the headers.

    Converters can be implemented in two ways:

    1. Lambdas: Best for simple, inline transformations.
    2. Classes: Best for reusable, complex, or independently testable logic. A class must implement a self.convert(value) method.

    Important: Interaction with key_mapping If you are using key_mapping:, your value_converters must use the mapped key name, not the original CSV header name, because mapping occurs before conversion.

    # Example using both key_mapping and value_converters
options = {
  key_mapping:      { membersince: :member_since },
  value_converters: { member_since: ->(v) { v ? Date.strptime(v, '%m/%d/%Y') : nil } },
}
data = SmarterCSV.process('records.csv', options)