ParSeq Documentation

ParSeq is a Java framework designed to simplify writing asynchronous code. It provides primitives for managing asynchronous operations, allowing developers to handle complex workflows with ease.

Key capabilities include:

• Parallelization: Running asynchronous operations (like IO) in parallel.
• Serialized Execution: Performing non-blocking computations sequentially.
• Task Composition: Reusing code by composing different tasks.
• Error Handling: Simple propagation and recovery mechanisms.
• Tracing: Execution tracing and visualization of task plans.
• Batching: Support for batching asynchronous operations.
• Retry Policies: Defining retry logic for tasks.

ParSeq Batching automatically groups individual asynchronous operations into batches to improve efficiency, especially when dealing with I/O.

Instead of executing every task immediately, the system identifies tasks that can be combined based on a defined strategy. When tasks are batched, their descriptions in traces are prefixed with batch:.

Key benefits include:

• Automatic Batching: Converts individual calls into batch calls without breaking modularity.
• De-duplication: If multiple tasks in a batch request the same resource (e.g., the same ID), the batching mechanism can de-duplicate these requests so the underlying API is only called once for that resource.
• Efficiency: Reduces the number of individual I/O operations by leveraging batch-capable APIs (like BATCH_GET).

Parseq Lambda Names transforms task names from opaque runtime identifiers into actionable debugging information.

Standard Parseq behavior: Uses generated Lambda class names which are unique to a specific service instance but cannot be mapped back to source code. Example: andThen: com.linkedin.voyager.jobs.services.JobPostingsService$$Lambda$2760/928179328

With Parseq Lambda Names: Provides the method name, the class name, the line number, and (where applicable) the function call within the lambda. Example: andThen: fetchJobPostings(JobPostingsService:112) Example with function inference: map: MapHelpers.mergeMaps(_,_) fetchTreatments(LixServiceImpl:124)

You can fine-tune the benchmark execution by specifying regex filters for benchmark methods and configuring the number of forks, iterations, and warmup iterations.

Commonly used flags:

• -t <threads>: Number of worker threads.
• -f <forks>: Number of forks.
• -i <iterations>: Number of measurement iterations.
• -wi <warmup_iterations>: Number of warmup iterations.
• <regex>: A regular expression to filter which benchmark methods to run.

Example: Running IdGeneratorBenchmark with 4 threads, 3 forks, 10 measurement iterations, and 5 warmup iterations:

java -jar build/libs/benchmarks.jar -t 4 -f 3 -i 10 -wi 5 ".*IdGeneratorBenchmark.*"

To enable batching in your application, you must register BatchingSupport as a PlanDeactivationListener to your Engine during setup. Then, register your custom BatchingStrategy with the BatchingSupport instance.

// 1. Set up BatchingSupport in the Engine
final BatchingSupport _batchingSupport = new BatchingSupport();
engineBuilder.setPlanDeactivationListener(_batchingSupport);

// 2. Register your strategy
MyBatchingStrategy myBatchingStrategy = new MyBatchingStrategy();
_batchingSupport.registerStrategy(myBatchingStrategy);

After building, you can run the visualizer by following these steps:

1. Extract the parseq-tracevis.tar.gz package.
2. Open trace.html in any web browser.

Alternatively, the tool can be hosted from a web server. For development, coding, or debugging purposes, the visualizer can also be run directly from the directory containing this README.

# Example workflow
./gradlew makeDist
tar -xzf build/distributions/parseq-tracevis.tar.gz
# Then open build/distributions/parseq-tracevis/trace.html in a browser

Parseq Lambda Names improves Parseq traces by replacing runtime-generated, non-deterministic Lambda class names (e.g., com.linkedin...$$Lambda$123) with human-readable task names that include the source code location and function signatures.

To enable this feature, include the shaded JAR of parseq-lambda-names on your classpath alongside the standard parseq JAR. When a Lambda is executed for the first time, the library will use ASM to analyze the bytecode and provide a meaningful description. If the parseq-lambda-names JAR is missing, Parseq will default to its standard behavior of using the generated Lambda class name.

<!-- Ensure both JARs are on the classpath -->
- parseq.jar
- parseq-lambda-names-shaded.jar

To build the trace visualization server from source, use the Gradle wrapper. This generates a runnable JAR file containing all necessary dependencies located at build/libs/parseq-tracevis-server-jar-with-dependencies.jar.

./gradlew build

ParSeq uses Gradle for building.

To build and test the entire project:

./gradlew clean build

To build a specific subproject (module):

./gradlew :<module_name>:build

MacOS Catalina (>=10.15) Setup: If you are building on MacOS Catalina or later, ensure you have installed the required Xcode Command Line Tools. You may also need to set the following environment variables:

export LDFLAGS="-mmacosx-version-min=10.13"
export CXXFLAGS="-mmacosx-version-min=10.13"

To run the benchmarks in the parseq-benchmark subproject, use Maven to build the project and then execute the generated JAR file. You can specify a regex pattern to filter specific benchmarks and set the number of threads using the -t flag.

Note: The benchmark output uses JMH (Java Microbenchmark Harness) format, reporting results in operations per second (ops/s).

# Build the benchmark project
mvn clean install

# Run benchmarks matching a regex pattern (e.g., ".*") with 4 threads
java -jar target/benchmarks.jar ".*" -t 4