CSV.jl

CSV.jl is a fast and flexible delimited file reader and writer designed for Julia. It is optimized for performance, especially on large files, and integrates with the Tables.jl ecosystem, allowing it to work seamlessly with containers like DataFrames.

Key characteristics:

• Performance: Significantly faster than the standard library's DelimitedFiles.jl for large datasets.
• Compatibility: Returns Tables.jl-style containers rather than simple Matrix objects, making it highly compatible with the modern Julia data science stack.

Standard Types

When column types are not manually provided, CSV.jl automatically detects the following types:

• Int64
• Float64
• Date
• DateTime
• Time
• Bool
• String

Cardinality

Cardinality refers to the ratio of unique values to total values in a column:

• Low Cardinality: A low percentage of unique values (many repeated values).
• High Cardinality: A high percentage of unique values (e.g., ID-like columns where most values are unique).

Depending on your specific use case, you might consider these alternatives to CSV.jl:

CSV.Chunks allows you to process extremely large files by dividing them into smaller, manageable pieces.

• Mechanism: By passing the ntasks::Integer keyword argument, the input file is split into ntasks number of chunks.
• Iteration: Each iteration of a CSV.Chunks object returns a CSV.File representing the next parsed chunk.
• Compatibility: It satisfies the Tables.partitions interface, meaning it can be passed directly to any sink function that supports partitioned input.

# Example: Processing a file in 10 chunks
for chunk in CSV.Chunks("huge_data.csv"; ntasks=10)
    # 'chunk' is a CSV.File object
    process_data(chunk)
end

To read a specific file from a zip archive, use a zip utility (like ZipArchives) to open the archive and then pass the specific entry to CSV.File.

using ZipArchives, Mmap, CSV, DataFrames

# ... (assuming a.zip exists) ...

# Read file from zip archive
z = ZipReader(mmap(open("a.zip")))

# Identify and parse the specific entry
a_copy = CSV.File(zip_openentry(z, "a.csv")) |> DataFrame

CSV.jl is a registered package in the Julia General registry. You can install it directly from the Julia REPL using the package manager.

To install, enter the package mode by pressing ] and run the add command.

# In the Julia REPL
add CSV

To read CSV files with non-UTF-8 encodings (like ISO-8859-1), use the StringEncodings package to open the file with the appropriate encoding and pass the resulting IO object to CSV.File.

By default, CSV.File reads the input into a temporary file on disk. To perform the encoding conversion entirely in memory for potentially faster performance, set buffer_in_memory=true.

using CSV, StringEncodings

# Default: reads into a temporary file
file = CSV.File(open("iso8859_encoded_file.csv", enc"ISO-8859-1"))

# Faster: performs conversion in memory
file = CSV.File(open("iso8859_encoded_file.csv", enc"ISO-8859-1"); buffer_in_memory=true)

You can read delimited data from the web using either HTTP.jl or the Downloads standard library.

Using HTTP.jl: Fetch the response and pass the body (as a Vector{UInt8}) to CSV.File. Using Downloads: Download the file to a temporary location and pass the resulting path to CSV.File.

# Option 1: Using HTTP.jl
using CSV, HTTP
http_response = HTTP.get(url)
file = CSV.File(http_response.body)

# Option 2: Using Downloads (Julia 1.6+)
using Downloads
http_response = Downloads.download(url)
file = CSV.File(http_response)

CSV.jl can automatically decompress gzipped files. Pass the filename of the .gz file directly to CSV.File.

By default, the data is decompressed to a temporary file and mmapped. To perform decompression in memory, use buffer_in_memory=true.

using CSV

# Decompress to a temporary file
file = CSV.File("data.gz")

# Decompress in memory
file = CSV.File("data.gz"; buffer_in_memory=true)

The input argument is required for reading. Data should be ASCII or UTF-8 encoded. Supported types include:

• File name (String or FilePath): Parsed via memory mapping. For .gz files, CodecZlib.jl is used to decompress to a temporary file. Use buffer_in_memory=true to decompress in memory instead of a temp file.
• Vector{UInt8} or SubArray{UInt8, 1, Vector{UInt8}}: Direct byte buffers. For strings, use CSV.File(IOBuffer(str)).
• IO or Cmd: Consumed into a temporary file and mmapped. Use buffer_in_memory=true to avoid temporary files and buffer in memory.
• Web files: Use HTTP.get(url).body to get a Vector{UInt8} or use Downloads.download(url) from the Julia standard library.

If you have a collection of delimited data inputs (e.g., a Vector of filenames, IO objects, AbstractVector{UInt8}, or IOBuffers) that share the same schema, you can pass the entire collection to CSV.File.

Each input is processed on a separate thread, and the results are vertically concatenated into a single CSV.File. The columns are lazily concatenated using the ChainedVector type. To parse directly into a DataFrame, use CSV.read with a mapping function.

using CSV

data = [
    "a,b,c\n1,2,3\n4,5,6\n",
    "a,b,c\n7,8,9\n10,11,12\n",
    "a,b,c\n13,14,15\n16,17,18",
]

# Vertically concatenate all inputs into one CSV.File
f = CSV.File(map(IOBuffer, data))

# Alternatively, parse directly into a DataFrame
# df = CSV.read(map(IOBuffer, data), DataFrame)