Sep .NET Library

Sep takes a pragmatic approach to CSV parsing, prioritizing performance and flexibility for use cases like machine learning pipelines over strict adherence to [RFC-4180].

Key Differences from RFC-4180:

• Line Endings: While RFC-4180 requires \r\n (CRLF), Sep supports \r\n, \n, and \r (similar to .NET). Environment.NewLine is used when writing.
• Separators: RFC-4180 specifies only a comma (,) as a separator. Sep defaults to a semicolon (;) when writing and auto-detects supported separators when reading.
• Quoting: RFC-4180 is strict about quoting. Sep supports matching pairs of quotes regardless of where they appear in a field, and allows characters like emojis which are restricted in the RFC's ABNF grammar.
• Escaping: In escaped fields, Sep allows any pair of quotes, whereas the RFC specifically requires double-quotes ("") to escape a quote.

Sep is designed for speed and real-world data rather than strict internet data exchange compliance.

Because SepReader is not LINQ-compatible due to its ref struct design, Sep provides specialized extension methods for efficient enumeration.

Enumerate: Converts the reader into an IEnumerable<T> by parsing rows into a target type T. This is the recommended way to bridge the gap between the high-performance ref struct API and standard LINQ/collection usage.

ParallelEnumerate: Provides multi-threaded enumeration using a batching strategy. It is built on top of AsParallel().AsOrdered().

Best Practices:

• Use ParallelEnumerate for parsing: It is optimized for the CPU-intensive task of turning raw text into typed data.
• Avoid expensive operations inside the delegate: If your per-row operation (like loading an image) takes more than 1ms, perform that work after the enumeration using standard LINQ AsParallel().
• Avoid LINQ .Where on the result of Enumerate if possible: Filtering after parsing can be inefficient if you are parsing many rows that will be discarded. Instead, use the Enumerate overload that accepts a RowTryFunc<T> to filter and parse in a single step.

// Efficiently filter and parse in one step using RowTryFunc
var actual = reader.Enumerate((SepReader.Row row, out (string Key, double Value) kv) =>
{
    var keyCol = row["Key"];
    if (keyCol.Span.StartsWith("B"))
    {
        kv = (keyCol.ToString(), row["Value"].Parse<double>());
        return true;
    }
    kv = default;
    return false;
}).ToArray();

// Using ParallelEnumerate for parsing, then AsParallel for expensive work
var results = reader.ParallelEnumerate(ParseRow)
                    .AsParallel().AsOrdered()
                    .Select(LoadData) // Expensive load happens here
                    .ToList();

The library follows a fluent API pattern where each step flows into the next:

Reading Pattern: Sep/Spec $\rightarrow$ SepReaderOptions $\rightarrow$ SepReader $\rightarrow$ Row $\rightarrow$ Col(s) $\rightarrow$ Span/ToString/Parse

Writing Pattern: Sep/Spec $\rightarrow$ SepWriterOptions $\rightarrow$ SepWriter $\rightarrow$ Row $\rightarrow$ Col(s) $\rightarrow$ Set/Format

Important Performance Note: SepReader.Row and SepReader.Col are ref structs. They act as lightweight facades pointing to internal state to avoid allocations. Because they are ref structs:

• You cannot use LINQ directly on the reader (e.g., reader.ToArray() is not possible).
• You cannot store rows in arrays or lists.
• To use LINQ, you must first parse/transform the rows into standard types (e.g., using yield return in an iterator or the provided Enumerate methods).

Sep provides efficient asynchronous support using ValueTask.

Requirements

Because SepReader.Row and SepWriter.Row are ref structs, using async/await requires C# 13.0+ (to support "ref and unsafe in iterators and async methods") and .NET 9.0+ (for IAsyncEnumerable<SepReader.Row> support via the allows ref struct annotation).

Usage Patterns

• SepReader: Creation methods like FromTextAsync are asynchronous because the reader must parse the first row to determine the separator and header columns. Use await when creating the reader. To iterate asynchronously, use await foreach.
• SepWriter: Creation methods (e.g., ToText) are synchronous. However, because disposing a row may involve flushing data to the underlying TextWriter, you must use await using for the writer and the rows.
• Cancellation: Many methods, including From*Async and NewRow, accept a CancellationToken to support cancellation.
• Context Configuration: Both SepReaderOptions and SepWriterOptions include an AsyncContinueOnCapturedContext boolean option, which controls ConfigureAwait behavior.

var text = """
           A;B;C;D;E;F
           Sep;🚀;1;1.2;0.1;0.5
           CSV;✅;2;2.2;0.2;1.5
           
           """; // Empty line at end is for line ending

using var reader = await Sep.Reader().FromTextAsync(text);
await using var writer = reader.Spec.Writer().ToText();
await foreach (var readRow in reader)
{
    await using var writeRow = writer.NewRow(readRow);
}
Assert.AreEqual(text, writer.ToString());