Apache Parquet Java

Parquet-Jackson is a specialized module used to shade Jackson artifacts. This prevents dependency conflicts (such as version mismatches) when running Parquet alongside other libraries like Apache Hadoop that might use different versions of Jackson.

Key characteristics:

• It is not a fork of Jackson.
• It contains the same classes as Jackson, but they are relocated to the parquet.com.fasterxml.jackson.core namespace.
• The parquet-jackson module provides a single shared location for these relocated classes to prevent duplication across different Parquet artifacts.

The build produces a shaded Jar that can be executed using the hadoop command. Replace 1.16.0 with your specific version.

hadoop jar parquet-cli-1.16.0-runtime.jar org.apache.parquet.cli.Main

Parquet-Java supports the Java Vector API to accelerate reading. This is an experimental feature and is not part of the standard distribution.

Requirements:

• Java 17+ (64-bit)
• CPU supporting avx512vbmi and avx512_vbmi2 instruction sets.

To build the jars with vector support:

./mvnw clean package -P vector-plugins

To enable in Apache Spark:

1. Build Parquet with the vector profile.
2. Replace parquet-encoding-{VERSION}.jar in the Spark jars folder with the newly built jar.
3. Copy parquet-encoding-vector-{VERSION}.jar to the Spark jars folder.
4. Modify Spark's VectorizedRleValuesReader.readNextGroup to refer to ParquetReadRouter.readBatchUsing512Vector.
5. Build Spark with Maven and replace spark-sql_2.12-{VERSION}.jar in the Spark jars folder.

To avoid typing the full hadoop jar command every time, you can add an alias to your shell configuration. Note that the --dollar-zero flag is used to ensure arguments are passed correctly.

alias parquet="hadoop jar /path/to/parquet-cli-1.16.0-runtime.jar org.apache.parquet.cli.Main --dollar-zero parquet"

You can pass JMH (Java Microbenchmarking Harness) arguments through the run.sh script to control the rigor and output of the benchmarks.

Examples:

• Quick run: Run every benchmark once using -wi 0 -i 1 -f 1 (~20 minutes).
• Rigorous run: Run with more iterations and save a report to a JSON file using -wi 5 -i 5 -f 3 -rff <path>.
• Help: View available JMH options by running ./parquet-benchmarks/run.sh all -help.

# Run every benchmark once (~20 minutes).
./parquet-benchmarks/run.sh all -wi 0 -i 1 -f 1

# A more rigourous run of all benchmarks, saving a report for comparison.
./parquet-benchmarks/run.sh all -wi 5 -i 5 -f 3 -rff /tmp/benchmark1.json

You can configure ParquetInputFormat and ParquetOutputFormat using either a Hadoop Configuration object or by using programmatic setters. This allows you to tune parameters such as page size and block size for Hadoop MapReduce jobs.

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.Job;

Configuration conf = new Configuration();
conf.set("parquet.page.size","128");

Job writeJob = new Job(conf);
ParquetOutputFormat.setBlockSize(writeJob, 1024);

The ./parquet-benchmarks/run.sh script is a wrapper for the JMH tool that allows you to launch various benchmark configurations, suites, or specific benchmarks using regex.

Common usage patterns include:

• Running all benchmarks with default settings.
• Running a specific benchmark suite (e.g., checksum).
• Running a single benchmark class using a regex pattern.
• Cleaning up state from previous runs.

You can build the Parquet CLI project using Maven. Use the following command to build the project while skipping tests:

./mvnw clean install -DskipTests

To use Apache Parquet in your Java project, add the following dependencies to your pom.xml. The current release version is 1.17.0.

  <dependencies>
    <dependency>
      <groupId>org.apache.parquet</groupId>
      <artifactId>parquet-common</artifactId>
      <version>1.17.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.parquet</groupId>
      <artifactId>parquet-encoding</artifactId>
      <version>1.17.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.parquet</groupId>
      <artifactId>parquet-column</artifactId>
      <version>1.17.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.parquet</groupId>
      <artifactId>parquet-hadoop</artifactId>
      <version>1.17.0</version>
    </dependency>
  </dependencies>

Once Thrift and Protobuf are available in your PATH, use Maven to build the project.

Full Build:

LC_ALL=C ./mvnw clean install

Build Jars only:

./mvnw package

Run Unit Tests:

./mvnw test

Parquet-Java depends on the Thrift compiler. You can install it manually from source or via Homebrew on OSX.

Manual Installation (Linux/Generic):

wget -nv https://archive.apache.org/dist/thrift/0.23.0/thrift-0.23.0.tar.gz
tar xzf thrift-0.23.0.tar.gz
cd thrift-0.23.0
chmod +x ./configure
./configure --disable-libs
sudo make install -j

OSX (Homebrew):

brew install thrift
export PATH="/usr/local/opt/thrift@0.23.0/bin:$PATH"