parquet-dotnet

A StructField acts as a container for other fields, effectively creating a table within a table cell. From a physical Parquet perspective, struct fields are stored as flat columns, but they are logically grouped.

Declaration

Use StructField and pass its member fields as children:

var schema = new ParquetSchema(
   new DataField<string>("name"),
   new StructField("address",
      new DataField<string>("line1"),
      new DataField<string>("postcode")
   ));

Reading and Writing

Even though the schema is nested, data is written and read using the individual, flat columns. The only difference is that the Path property of the field in the schema will include the struct name (e.g., address.line1).

// Writing to a struct
using var ms = new MemoryStream();
await using(ParquetWriter writer = await ParquetWriter.CreateAsync(schema, ms)) {
    using ParquetRowGroupWriter rgw = writer.CreateRowGroup();

    // Accessing fields by index or name from the schema
    await rgw.WriteAsync(schema.DataFields[0], new[] { "Joe" }); // 'name'
    await rgw.WriteAsync(schema.DataFields[1], new[] { "Amazonland" }); // 'address.line1'
    await rgw.WriteAsync(schema.DataFields[2], new[] { "AAABBB" }); // 'address.postcode'
}

In Parquet, an array (or repeatable field) is a column that can contain multiple values per row.

Declaration

You declare an array by specifying IEnumerable<T> as the type in a DataField:

var field = new DataField<IEnumerable<int>>("items");

Alternatively, you can set isArray: true in the DataField constructor.

Writing Arrays

When writing to an array field, you must provide both the flattened data values and the repetition levels to indicate where new lists start.

Warning: While supported, using arrays instead of lists is strongly discouraged due to poor compatibility with other systems and lack of support for schema evolution.

Checking for Arrays

You can check if a field is repeatable by inspecting the .IsArray boolean property.

// Declare a repeatable field
var field = new DataField<IEnumerable<int>>("items");

// Writing data with repetition levels
// Data: [1, 2, 3, 4, 5] representing [[1, 2, 3], [4, 5]]
// RL:   [0, 1, 1, 0, 1] (0 starts a new list, 1 continues current list)
await groupWriter.WriteAsync<int>(field, new int[] { 1, 2, 3, 4, 5 }, new int[] { 0, 1, 1, 0, 1 });

Lists are used to store collections of items. A ListField requires a name and a child Field representing the item type.

Lists of Primitive Types

To declare a list of integers, pass a DataField<int> as the second parameter to ListField:

new ListField("item", new DataField<int>("id"));

Lists of Structs

Lists can contain complex types like structs. When defining a list of structs, the schema follows a 3-level structure: the list name, a middle level called list, and a level called element containing the struct fields.

Note: ListField automatically handles the internal list level, so you only need to define the element level using the ListField.ElementName constant.

var schema = new ParquetSchema(
    new DataField<int>("TopLevelId"),
    new ListField("Structs",
        new StructField(ListField.ElementName,
            new DataField<int>("Id"),
            new DataField<string>("Name"))));

// Writing a list of structs
await using(ParquetWriter w = await ParquetWriter.CreateAsync(schema, ms)) {
    using ParquetRowGroupWriter gw = w.CreateRowGroup();

    await gw.WriteAsync(nameField, new string[] { "Joe", "Bob" });
    // Note: Writing list elements requires providing repetition levels
    await gw.WriteAsync(line1Field, new[] { "Amazonland", "Disneyland", "Cryptoland" }, new[] { 0, 1, 0 });
    await gw.WriteAsync(postcodeField, new[] { "AAABBB", "CCCDDD", "EEEFFF" }, new[] { 0, 1, 0 });
}

The High level API is designed for ease of use, mimicking class serialization similar to JSON processing. It is ideal when you want to work with C# objects (rows) rather than managing columnar data manually.

Key Characteristics:

• Uses ParquetSerializer to convert lists of objects to/from Parquet files.
• Uses compiled expression trees for high performance (significantly faster than reflection after the first serialization).
• Requirements for classes/structs:
  • Must have a parameterless constructor.
  • Properties/fields must be readable for serialization and writeable for deserialization.
  • Deserialization appends to existing list properties rather than overwriting them.
  • While you can serialize struct, deserialization is only supported into class due to optimization requirements.

// 1. Define your data model
class Event {
    public DateTime Timestamp { get; set; }
    public string EventName { get; set; }
    public double MeterValue { get; set; }
}

// 2. Generate data
List<Event> data = Enumerable.Range(0, 1_000_000).Select(i => new Event {
    Timestamp = DateTime.UtcNow.AddSeconds(i),
    EventName = i % 2 == 0 ? "on" : "off",
    MeterValue = i 
}).ToList();

// 3. Write to file
await ParquetSerializer.SerializeAsync(data, "/path/to/data.parquet");

// 4. Read from file
DeserializationResult<Event> result = await ParquetSerializer.DeserializeAsync<Event>("/path/to/data.parquet");
Event[] events = result.Data;

Parquet row groups are immutable, so you cannot append data to an existing row group. To append data, you must create a new row group at the end of the file.

To perform an append operation:

1. Open the ParquetWriter using ParquetWriter.CreateAsync and set the append parameter to true.
2. Create a new row group using writer.CreateRowGroup().
3. Write the new data to that row group.

Warning: Avoid creating very small row groups (e.g., 1-2 rows). Row groups are designed for large batches (averaging ~50,000 rows). Small row groups significantly increase file size and degrade read performance.

// Assume 'schema' and 'ms' (MemoryStream) are already initialized
// Append to this file by creating a new row group
await using(ParquetWriter writer = await ParquetWriter.CreateAsync(schema, ms, append: true)) {
    using(ParquetRowGroupWriter rg = writer.CreateRowGroup()) {
        await rg.WriteAsync<int>(id, new int[] { 3, 4 });
    }
}

The Low level API provides extreme performance and full control by interacting directly with Parquet's columnar structure. This is the preferred method for big data processing where you only need to read or write specific columns.

Core Concepts:

• Schema: Defined using ParquetSchema containing DataField objects.
• Row Groups: A file consists of one or more row groups. Each row group contains all columns from the schema but different data rows. Writing/reading is done at the row group level.
• Columnar Writing: Data must be prepared as arrays (chunks) per column.
• Columnar Reading: Requires pre-allocating buffers for the expected number of rows in a row group.

// --- WRITING --- 
var schema = new ParquetSchema(
    new DataField<DateTime>("Timestamp"),
    new DataField<string>("EventName"),
    new DataField<double>("MeterValue"));

using Stream fs = System.IO.File.OpenWrite("data.parquet"); 
await using(ParquetWriter writer = await ParquetWriter.CreateAsync(schema, fs));
using ParquetRowGroupWriter groupWriter = writer.CreateRowGroup();

// Write columns as arrays
await groupWriter.WriteAsync<DateTime>(schema.DataFields[0], new[] { DateTime.UtcNow });
await groupWriter.WriteAsync(schema.DataFields[1], new[] { "start" });
await groupWriter.WriteAsync<double>(schema.DataFields[2], new[] { 12.34 });

// --- READING --- 
using(Stream fs = System.IO.File.OpenRead("data.parquet"));
await using(ParquetReader reader = await ParquetReader.CreateAsync(fs));

for(int i = 0; i < reader.RowGroupCount; i++) {
    using(ParquetRowGroupReader rowGroupReader = reader.OpenRowGroupReader(i)) {
        DataField[] dataFields = reader.Schema.GetDataFields();

        // Pre-allocate buffers
        DateTime[] timestamps = new DateTime[rowGroupReader.RowCount];
        string[] eventNames = new string[rowGroupReader.RowCount];
        double[] meterValues = new double[rowGroupReader.RowCount];

        // Read data into buffers
        await rowGroupReader.ReadAsync<DateTime>(dataFields[0], timestamps);
        await rowGroupReader.ReadAsync(dataFields[1], eventNames);
        await rowGroupReader.ReadAsync<double>(dataFields[2], meterValues);
    }
}