csvlint

The Csvlint::Validator adapts its validation strategy based on the type of source provided:

1. URL (String): Uses Typhoeus to perform an HTTP request. It validates metadata from response headers (like content-type and link headers) and processes the body in chunks. It supports following redirects.
2. File: Parses the file extension to determine type and uses the file path to locate potential metadata/schemas.
3. IO / StringIO: Processes the content as a stream using each_line.

If a URL is provided, the validator also looks for a schema via the link header (e.g., rel="describedby") or by checking the /.well-known/csvm path on the host.

The validator performs heuristic analysis on column data to identify formats. This is used to check for consistency and schema compliance. Supported formats include:

• numeric: Integers and floats (e.g., 123, -0.45, 1e10).
• uri: Strings matching HTTP/HTTPS patterns.
• date_db: ISO-style dates (YYYY-MM-DD).
• date_short/long/rfc822: Various human-readable date formats.
• dateTime_...: Various combinations of date and time (ISO8601, HMS, etc.).
• string: The fallback format.

If a column's values are inconsistent (e.g., a column is 90% numeric but 10% strings), the validator will issue a :inconsistent_values warning.

The validator automatically attempts to locate a schema to validate the CSV structure and data types. It follows this priority:

1. Explicit Schema: If a schema object is passed to initialize.
2. Link Headers: If the source is a URL, it checks the link HTTP header for a rel="describedby" relationship pointing to a JSON schema.
3. Well-known URI: For HTTP(S) sources, it checks [source_url]/.well-known/csvm for a list of metadata paths.
4. Convention-based paths: It searches for {url}-metadata.json or csv-metadata.json relative to the source URL.

If a schema is found, it is used to validate headers, rows, and foreign key constraints.

The dialect is a hash used to define how the CSV should be parsed. If not provided, the validator uses a default dialect. The validator also attempts to merge dialect settings found in a linked schema if one is present.

Key dialect keys include:

• header: Boolean indicating if the first row is a header.
• headerRowCount: Integer specifying the number of header rows.
• delimiter: String used as the column separator (defaults to ,).
• skipInitialSpace: Boolean (defaults to true).
• lineTerminator: The character(s) used for line breaks (e.g., :auto, \n, \r\n).
• quoteChar: The character used for quoting fields (defaults to ").
• trim: Boolean or :true to strip whitespace from headers.

dialect = {
  "delimiter" => "|",
  "quoteChar" => "'",
  "header" => true,
  "lineTerminator" => "\r\n"
}
validator = Csvlint::Validator.new('data.csv', dialect)

Use Csvlint::Csvw::Column.from_json to instantiate a column object from a JSON description object (typically parsed from a CSVW metadata file). This method handles property checking, annotation extraction, and inheritance of properties from the parent schema.

Arguments:

• number: The column index.
• column_desc: A hash representing the column's JSON properties.
• base_url: (Optional) The base URL for resolving URIs.
• lang: (Optional) The language code (defaults to "und").
• inherited_properties: (Optional) A hash of properties to inherit from the table/file level.

column_desc = {
  "@type" => "Column",
  "name" => "Age",
  "datatype" => { "@id" => "http://www.w3.org/2001/XMLSchema#integer" }
}

column = Csvlint::Csvw::Column.from_json(0, column_desc)

After all rows in all relevant tables have been parsed and validated using validate_row(..., true), call validate_foreign_keys to perform cross-table referential integrity checks.

This method iterates through the table's defined foreign_keys, identifies the remote tables, and triggers validate_foreign_key_references on those remote tables to ensure that referencing values exist in the target tables and that constraints (like uniqueness) are respected.

# Assuming all tables have been processed with validate_row(..., true)
table.validate_foreign_keys

Use validate_header(header, source_url = nil, validate = true) to check if the provided header matches the field names defined in the loaded schema.

If the header does not match the expected schema fields, it will trigger a :malformed_header warning. The method returns true if the header is valid according to the schema, and false otherwise.

schema = Csvlint::Schema.load_from_uri('schema.json')
header = ['name', 'age', 'email']

if schema.validate_header(header)
  puts "Header is valid"
else
  puts "Header mismatch detected"
end

The Csvlint::Schema class provides methods to load schema definitions from various sources.

• load_from_uri(uri, output_errors = true): Fetches a schema from a remote URL or a local file path using URI.open.
• load_from_string(uri, string, output_errors = true): Parses a JSON string as a schema. If the JSON contains an @context key, it is treated as CSVW (CSV on the Web) metadata; otherwise, it is treated as a JSON Table schema.

Note: load_from_json is deprecated and should be replaced by load_from_uri.

# Loading from a URL
schema = Csvlint::Schema.load_from_uri('https://example.com/schema.json')

# Loading from a local file path
schema = Csvlint::Schema.load_from_uri('/path/to/schema.json')

# Loading from a JSON string
json_string = '{"title": "My Schema", "fields": []}'
schema = Csvlint::Schema.load_from_string('local_name', json_string)

Use Csvlint::Csvw::Table.from_json to instantiate a table object directly from a CSVW (CSV on the Web) table description hash. This method automatically parses the tableSchema, including columns, primary keys, foreign keys, and row titles, while handling property inheritance and annotations.

table_desc = {
  "@type" => "Table",
  "url" => "data.csv",
  "tableSchema" => {
    "columns" => [
      { "name" => "id", "type" => "integer" },
      { "name" => "name", "type" => "string" }
    ],
    "primaryKey" => ["id"]
  }
}

table = Csvlint::Csvw::Table.from_json(table_desc)

Use the match(value) method to check if a given string conforms to the defined NumberFormat. It returns true if the value matches the internal regular expression generated by the format, and false otherwise.

Note: The method handles nil values by returning false.

format = Csvlint::Csvw::NumberFormat.new("$"?"#,###.##")
format.match("$1,234.56") # => true
format.match("$1234.56")   # => false (missing grouping separator)
format.match(nil)           # => false