ZenDNN (Zen Deep Neural Network)

ZenML Profiler is a log analysis tool designed to extract performance insights from ZenDNN or OneDNN primitive-level logs. It automates the process of detecting iteration boundaries, slicing logs for specific iterations, and generating detailed summaries including grouped operations, FLOPs, and efficiency metrics.

Key Features:

• Multi-backend support: Handles ZenDNN_5.2 (default), ZenDNN_5.1 (legacy positional), ZenDNN_4.2, and OneDNN.
• Automatic Iteration Detection: Uses pattern analysis to find boundaries, or accepts a user-specified iteration count.
• Comparative Analysis: Supports comparing multiple logs (e.g., different versions or algorithms) in a single report.
• Rich Reporting: Generates text reports, Excel workbooks (with charts), and CSV exports.
• Integrations: Supports PyTorch Profiler (JSON/log) and BenchDNN.
• Performance Metrics: Calculates % Impact against end-to-end time (if provided) or total library execution time.

The LowOHA Normalization Operator is a high-performance, function-based interface designed for latency-sensitive inference workloads. Unlike standard operator factory patterns, it provides a single entry point, normalization_direct, which minimizes per-call overhead by avoiding operator/context object lifecycles on the hot path.

It supports five normalization variants:

• LayerNorm: Normalizes across trailing dimensions (e.g., Transformer encoder blocks).
• RMSNorm: Root Mean Square normalization (e.g., LLaMA, Mistral).
• FusedAddRMSNorm: Fuses residual addition with RMSNorm; the residual buffer is updated in-place.
• FusedLayerNormAdd: Applies LayerNorm and then adds a read-only residual to the result (norm-then-add); the residual buffer is not modified.
• BatchNorm (Inference): Normalizes per channel using pre-computed running statistics (e.g., CNN inference).

Key features include native AVX-512 and AVX-512-FP16 vectorized kernels and support for FP32, BF16, and F16 data types.

// Example of the primary entry point
status_t status = normalization_direct(
  input,         // const void *
  output,        // void *
  gamma,         // const void *
  beta,          // const void *
  running_mean, // const void *
  running_var,  // const void *
  residual,      // void *
  params         // norm_params &
);

The group_matmul gtests provide a self-contained testing environment for the group_matmul operator family, including basic group matmul, fused-MoE, scheduling algorithms, quantization, and the prepack module.

Key Testing Areas

• Public Dispatcher: Tests the group_matmul_direct(...) entry point and all reachable scheduling ALGOs.
• Fused-MoE: Covers Op1 + activation + Op2, gated activations, MoE post-op (weighted reduce), and per-expert contracts.
• Quantization: Tests WOQ, INT8, and dynamic quantization.
• Prepack Module: Validates prepacking logic, cache behavior, and correctness across different algorithms.
• Custom Kernels: Direct-surface tests for the CK stack (dispatch, resolve, prepare, pack, and microkernels) for BF16, INT8, and FP16.

Running Tests

All tests compile into a single gtests executable. You can filter tests using the --gtest_filter flag:

• python3 gtests --gtest_filter=*Prepack*
• python3 gtests --gtest_filter=*FusedMoE*

The LowOHA Reorder Operator is a high-performance, low-overhead function-based interface designed for quantization, dequantization, and data type conversion. Unlike the standard Reorder operator which uses a factory pattern, LowOHA provides a direct API optimized for minimal execution overhead.

Key Capabilities

• Quantization/Dequantization: Convert between BF16/FP32/FP16 and INT8/UINT8 formats.
• Data Type Conversion: Convert among FP32, BF16, and FP16 in any direction.
• Dynamic Quantization: Compute scale and zero-point from source data at runtime (supports BF16, FP32, and FP16 sources).
• Granularity Support: Supports per-tensor, per-channel (row/column), and per-group (row/column) quantization.
• Memory Support: Supports strided (non-contiguous) source memory.
• Weight Prepack: Use is_prepack = true to produce a backend-blocked weight buffer once at model load and reuse it across matmul calls.

Performance Note: FP16 Path Selection

For per-tensor FP16 source/destination, the operator auto-selects between two AVX-512 backends at dispatch time via can_use_f16_fma_kernel():

1. F32-FMA: Uses AVX-512F + F16C convert.
2. FP16-FMA: Uses native __m512h (requires AVX512-FP16 ISA).

To force the F32-FMA path, build with -DZENDNNL_NATIVE_F32_ACCUM=ON.

// Primary interface for LowOHA Reorder
status_t reorder_direct(
  const void *src,                      // Pointer to source data buffer
  void *dst,                            // Pointer to destination data buffer (nullptr allowed for dynamic quant compute-only)
  reorder_params_t &params              // Reorder parameters (non-const reference)
);

Dynamic quantization computes the scale (and zero point for asymmetric mode) from the input data at runtime.

1. Set params.dynamic_quant = true.
2. Provide a buffer in params.quant_params.scale.buff to receive the computed scale(s).
3. For Symmetric mode (e.g., dst_dtype = data_type_t::s8): Set params.quant_params.zero_point.buff = nullptr. The scale is computed as max(|min|, |max|) / 127.
4. For Asymmetric mode (e.g., dst_dtype = data_type_t::u8): Provide a buffer in params.quant_params.zero_point.buff. The scale is (max - min) / 255 and zero point is round(-min / scale).
5. To compute only (without writing output): Set the destination pointer in reorder_direct to nullptr.

// Example: Symmetric Per-Tensor Dynamic Quantization (BF16 -> S8)
reorder_params_t params;
params.src_dtype = data_type_t::bf16;
params.dst_dtype = data_type_t::s8;
params.src_shape = {M, N};
params.dst_shape = {M, N};
params.dynamic_quant = true;

float computed_scale = 0.0f;
params.quant_params.scale.buff = &computed_scale;
params.quant_params.scale.dt = data_type_t::f32;
params.quant_params.scale.dims = {1, 1};
params.quant_params.zero_point.buff = nullptr; // Symmetric

status_t status = reorder_direct(input_bf16.data(), output_s8.data(), params);

The LOWOHA Conv2D Operator is a high-performance, function-based interface designed for latency-sensitive inference workloads. Unlike standard operator factory patterns, it provides direct control over execution parameters and is optimized for minimal overhead, repeated weight reuse (via weight caching), and backend-native post-operation fusion.

It supports standard, grouped, and depthwise convolutions and operates on tensors in NHWC format.

// The primary interface is the conv_direct function
status_t conv_direct(
  const void *input,           // Input tensor [N, H, W, C] (NHWC format)
  const void *filter,          // Filter tensor [KH, KW, C_in, C_out]
  const void *bias,            // Optional bias vector [C_out] (can be nullptr)
  void *output,                // Output tensor [N, H_out, W_out, C_out]
  const bool is_weights_const, // Whether filters are constant (enables caching)
  conv_params &params          // LOWOHA parameters (dims, strides, padding, dtypes, post-ops)
);

ZenDNN GTest validates numerical accuracy by comparing optimized implementations against reference results using dynamic tolerance algorithms. The tolerance scales based on the matrix dimension k (the inner dimension) to account for accumulation error.

Tolerance Formulas

BF16/F16 Operations: allowed_error = (k * epsilon_bf16) + (rtol_bf16 * |reference_value|)

F32 Operations: allowed_error = (((20 + log2(k)/4) * k + 15) * epsilon_f32) + (rtol_f32 * |reference_value|)

Key Constants

• epsilon_f32: 1.19e-7 (Base F32 tolerance)
• epsilon_bf16: 9.76e-4 (Base BF16 tolerance)
• rtol_f32: 1e-5 (F32 relative tolerance)
• rtol_bf16: 1e-2 (BF16 relative tolerance)

Relaxation and Scaling

• Adaptive Scaling: For large matrices, logarithmic scaling is used to prevent excessive tolerance growth.
• Relaxation: A 9e-4 buffer is added for F32 operations. Relaxation can be enabled via the ENABLE_F32_RELAXATION macro in gtest_utils.hpp and is enabled by default for LIBXSMM backends.