ParquetSharp

The Parquet Map logical type is a special case of nested data. Because ParquetSharp's API works with leaf-level columns, .NET Dictionary objects cannot be represented as a single column. Instead, a Map is represented as two separate columns:

1. A column containing arrays of keys.
2. A column containing arrays of values.

For a valid Map, the arrays in both columns must have the same length for every row.

Writing Map data

To write a dictionary, you must split your Dictionary<K, V> collection into two arrays (keys and values) and define a schema with a GroupNode annotated with LogicalType.Map() containing a repeated key_value group.

Reading Map data

To read a Map, read the key column and the value column separately using LogicalReader, then use LINQ (e.g., Zip) to reconstruct the Dictionary objects in memory.

// Writing: Split dictionaries into keys and values
var dictionaries = new[] { new Dictionary<string, int>{{"a", 0}} };
var keys = dictionaries.Select(d => d.Keys.ToArray()).ToArray();
var values = dictionaries.Select(d => d.Values.ToArray()).ToArray();

// Schema for Map
using var keyNode = new PrimitiveNode("key", Repetition.Required, LogicalType.String(), PhysicalType.ByteArray);
using var valueNode = new PrimitiveNode("value", Repetition.Required, LogicalType.None(), PhysicalType.Int32);
using var keyValueNode = new GroupNode("key_value", Repetition.Repeated, new Node[] {keyNode, valueNode});
using var mapNode = new GroupNode("map_column", Repetition.Required, new Node[] {keyValueNode}, LogicalType.Map());
using var schema = new GroupNode("schema", Repetition.Required, new Node[] {mapNode});

// Reading: Reconstruct dictionaries
var readDictionaries = readKeys.Zip(
    readValues, (dictKeys, dictValues) => new Dictionary<string, int>(
        dictKeys.Zip(dictValues, (k, v) => new KeyValuePair<string, int>(k, v)))
    ).ToArray();

The row-oriented API provides a high-level abstraction over Parquet's column-oriented nature, allowing you to work with whole rows (e.g., using tuples or custom structs) instead of individual columns. This results in more readable code but comes with a performance trade-off: all values in a row group must be stored in memory simultaneously, and internal buffers are resized and copied as they grow.

When to use: Use this API for convenience and readability in non-performance-critical paths. For high-performance requirements, use the lower-level column-oriented API.

Key Operations:

• ParquetFile.CreateRowWriter<T>(path, columns): Creates a writer for a specific row type T.
• rowWriter.WriteRow(row): Writes a single row.
• rowWriter.StartNewRowGroup(): Manually starts a new row group.
• rowWriter.Close(): Finalizes the file.
• ParquetFile.CreateRowReader<T>(path): Creates a reader for a specific row type T.
• rowReader.ReadRows(rowGroupIndex): Reads all rows within a specific row group.

var columns = new[] {"Timestamp", "ObjectId", "Value"};
using var rowWriter = ParquetFile.CreateRowWriter<(DateTime, int, float)>("data.parquet", columns);

// Writing rows
rowWriter.WriteRow((DateTime.Now, 1, 0.5f));

// Starting a new row group
rowWriter.StartNewRowGroup();
rowWriter.WriteRow((DateTime.Now, 2, 0.7f));

rowWriter.Close();

ParquetSharp uses Type Factories to map .NET types (logical system types) to Parquet physical types and handle the associated conversions. This mechanism allows users to:

1. Read/Write custom types: By providing a mapping between a user-defined type and a Parquet physical type.
2. Override default mappings: Change how standard .NET types are handled globally or per-file.

Core Components

• @ParquetSharp.LogicalTypeFactory: Manages the mapping between logical and physical types. It can be subclassed for custom logic or configured via properties for simple changes.
• @ParquetSharp.LogicalReadConverterFactory: Used by ParquetFileReader to tell readers how to convert physical Parquet data into user-defined logical types.
• @ParquetSharp.LogicalWriteConverterFactory: Used by ParquetFileWriter to tell writers how to convert user-defined logical types into physical Parquet data.

When to use which factory

• LogicalTypeFactory is required if you use a custom type in a Column object during writing, or if you access a reader/writer via non-overriding methods (where the factory is needed to establish the full type mapping).
• Converter Factories (LogicalReadConverterFactory or LogicalWriteConverterFactory) are always required when using a custom type that the default converters do not support.

Writing encrypted files involves three main steps:

1. Configure KMS Connection: Set up a KmsConnectionConfig with your service details.
2. Configure Encryption: Create an EncryptionConfiguration specifying the footerKeyId. You can enable UniformEncryption (all columns use the same master key) or provide a ColumnKeys dictionary for per-column master keys. You can also toggle DoubleWrapping or set PlaintextFooter = true to allow schema reading without decryption.
3. Generate Properties: Use a CryptoFactory to generate FileEncryptionProperties, then pass these to a WriterPropertiesBuilder to create WriterProperties for your ParquetFileWriter.

Note: Use the Key Management Tools API if you require compatibility with PyArrow.

using var kmsConnectionConfig = new KmsConnectionConfig();
// Set fields required by your IKmsClient implementation
kmsConnectionConfig.KmsInstanceId = ...;

string footerKeyId = ...;
using var encryptionConfig = new EncryptionConfiguration(footerKeyId);
// Optional: encryptionConfig.UniformEncryption = true;
// Optional: encryptionConfig.DoubleWrapping = false;

using var fileEncryptionProperties = cryptoFactory.GetFileEncryptionProperties(
    kmsConnectionConfig, encryptionConfig);

using var writerPropertiesBuilder = new WriterPropertiesBuilder();
using var writerProperties = writerPropertiesBuilder
                .Compression(Compression.Snappy)
                .Encryption(fileEncryptionProperties)
                .Build();

Column[] columns = ...;
using var fileWriter = new ParquetFileWriter(parquetFilePath, columns, writerProperties);
// Write data with fileWriter