DOCSEARCHbyAlgolia
Submit Docs

Intel Open Image Denoise

repository·master·Indexed 24 days ago

https://github.com/renderkit/oidn

A high-performance, deep-learning-based denoising library for ray-traced images, part of the Intel Rendering Toolkit. It provides a C99 API and a C++11 wrapper to reduce rendering times by allowing lower samples per pixel. The library supports multiple hardware backends including CPU, Intel GPU (SYCL), NVIDIA GPU (CUDA), AMD GPU (HIP), and Apple GPU (Metal).

Tokens
29.6K
Snippets
45
Records
109
Agent score
84%

What's inside Intel Open Image Denoise

  1. Overview of Intel® Open Image Denoise

    master
    Intel® Open Image Denoise (OIDN) is a library designed for denoising images, typically produced by Monte Carlo path tracing. For detailed information on features, changes, and the latest updates, refer to the project's official website or the repository's changelog.

  2. Overview of Intel Open Image Denoise

    master

    Intel Open Image Denoise (OIDN) is an open-source library providing high-performance, high-quality denoising filters for ray-traced images. It is designed to reduce rendering times by filtering out Monte Carlo noise from stochastic ray tracing methods like path tracing.

    Key features include:

    • Deep Learning Based: Uses efficient filters trained to handle a wide range of samples per pixel (spp), from 1 spp to near-convergence.
    • Flexible API: A C/C++ API allows for easy integration into existing or new rendering solutions.
    • Feature Buffers: Supports denoising using only the noisy color (beauty) buffer, or utilizing auxiliary feature buffers (e.g., albedo, normal) to preserve more detail.
    • Custom Training: While pre-trained models are provided, users can train custom models using the included training toolkit and their own datasets.
    • Hardware Acceleration: Exploits modern instruction sets (SSE4, AVX2, AVX-512, Intel AMX, NEON) and AI acceleration on various CPUs and GPUs (Intel, NVIDIA, AMD, and Apple silicon).

  3. Overview of the OIDN Training Toolkit

    master

    The OIDN source distribution includes a Python-based neural network training toolkit located in the training directory. This toolkit allows users to train custom denoising filter models using their own image datasets.

    Warning: This is an advanced feature requiring background knowledge in machine learning and familiarity with frameworks like PyTorch or TensorFlow.

    Available Scripts:

    • preprocess.py: Preprocesses training and validation datasets.
    • train.py: Trains a model using preprocessed datasets.
    • infer.py: Performs inference on a dataset using a specified training result.
    • export.py: Exports a training result to the runtime model weights format.
    • find_lr.py: Finds optimal minimum and maximum learning rates.
    • visualize.py: Invokes TensorBoard for visualizing training statistics.
    • split_exr.py: Splits multi-channel EXR images into separate feature images.
    • convert_image.py: Converts a feature image to a different format.
    • compare_image.py: Compares two feature images using specified quality metrics.

  4. Denoising input buffer types

    master

    Intel Open Image Denoise can utilize different types of auxiliary buffers to improve denoising quality. Based on the gallery examples, the library supports denoising using:

    • Color buffer: The primary noisy image.
    • Albedo buffer: Can be provided as either prefiltered (clean) or noisy.
    • Normal buffer: Can be provided as either prefiltered (clean) or noisy.

    Using prefiltered albedo and normal buffers (as seen in the 'Cabins' example) is a valid configuration for denoising low sample counts (e.g., 16 spp).

  5. Format Datasets for OIDN Training

    master

    Datasets must consist of noisy images and corresponding noise-free reference images stored in .exr format.

    Directory Structure

    All datasets (e.g., train, valid) must reside in the same parent directory (e.g., data). Each dataset is in its own subdirectory. All versions of a specific image (noisy and reference) must be in the same subdirectory.

    Filename Convention

    Filenames must follow a specific pattern using a base name, a suffix indicating samples per pixel (spp) or reference status (ref, reference, gt, target), a feature type extension, and the .exr extension.

    Regex pattern: .+_([0-9]+(spp)?|ref|reference|gt|target)\.(hdr|ldr|sh1[xyz]|alb|nrm)\.exr

    Note: spp numbers should be zero-padded to a fixed number of digits.

    Supported Image Features

    FeatureDescriptionChannelsExtension
    hdrcolor (HDR)3.hdr.exr
    ldrcolor (LDR)3.ldr.exr
    sh1color (normalized L1 spherical harmonics)3 × 3 images.sh1x.exr, .sh1y.exr, .sh1z.exr
    albalbedo3.alb.exr
    nrmnormal3.nrm.exr

  6. Use custom trained model weights

    master
    Instead of using the built-in trained models for filtering, you can specify user-trained models at runtime. This is done by passing a model weights blob (of type Data) to the filter parameters. The weights blob must correspond to the specific set of input features and filter parameters used during training. See the [Training] section for details on how to produce these weights using the included training tool.

  7. Manage device lifecycle and synchronization

    master

    Open Image Denoise uses reference counting for all objects.

    • Synchronization: Some functions execute asynchronously. To ensure all operations are complete and to catch any errors that occurred during async execution, call oidnSyncDevice(device). Errors are only reported when synchronization is triggered explicitly or implicitly (e.g., via oidnExecuteFilter).
    • Reference Counting: Use oidnRetainDevice(device) to increase the reference count. Use oidnReleaseDevice(device) to decrease it. When the count reaches 0, the device is automatically deleted.
    • Best Practice: Create only one OIDNDevice per physical device (e.g., one for all CPUs, one per GPU) to avoid high memory overhead and expensive creation costs.

  8. Limitations of the RT filter

    master

    The RT filter has specific constraints regarding input data:

    1. Ray Tracing Requirement: It cannot denoise images that were not rendered with ray tracing.
    2. Anti-aliasing/Sampling: While it supports high-quality pixel reconstruction filters (e.g., Gaussian, Blackman-Harris), it does not support weighted pixel sampling (splatting). Splatting introduces correlation between neighboring pixels that causes denoising to fail (noise will not be filtered). Use importance sampling instead.
    3. Auxiliary Noise: If cleanAux is enabled, any noise present in the albedo or normal images will be treated as signal and will appear as residual noise in the output image.

  9. Dataset requirements and format for OIDN training

    master

    A dataset must consist of a collection of noisy images and their corresponding noise-free reference images.

    Directory Structure: All datasets (e.g., training, validation) must reside in the same parent directory (e.g., data). Each dataset is stored in its own subdirectory (e.g., train, valid).

    Image Requirements:

    • Format: Images must be in OpenEXR (.exr) format.
    • Channels: Each feature (color, albedo, etc.) must be stored in a separate image file. Multi-channel EXR files are not supported (use split_exr.py to split them).
    • Grouping: All versions of an image (noisy and reference) must be in the same subdirectory.

    Filename Convention: Filenames must follow a specific pattern: [base_name]_[suffix].[feature_type].exr.

    The suffix must indicate the samples per pixel (padded with leading zeros) or whether it is the reference image.

    Regex for valid filenames:

    .+_([0-9]+(spp)?|ref|reference|gt|target)\.(hdr|ldr|sh1[xyz]|alb|nrm)\.exr

    Supported Features:

    FeatureDescriptionChannelsExtension
    hdrcolor (HDR)3.hdr.exr
    ldrcolor (LDR)3.ldr.exr
    sh1color (normalized L1 spherical harmonics)3 × 3 images.sh1x.exr, .sh1y.exr, .sh1z.exr
    albalbedo3.alb.exr
    nrmnormal3.nrm.exr

  10. Use Asynchronous Execution and Filter Quality modes

    master

    Asynchronous Execution

    Certain operations can be executed asynchronously to improve performance. These include:

    • Filtering: oidnExecuteFilterAsync, oidnExecuteSYCLFilterAsync
    • Data Copying: oidnReadBufferAsync, oidnWriteBufferAsync

    Important: When using asynchronous functions, the application is responsible for synchronization using oidnSyncDevice.

    Filter Quality

    You can trade off image quality for performance by setting the quality parameter on filters.

    Quality Modes:

    • OIDN_QUALITY_HIGH (Default): Highest image quality.
    • OIDN_QUALITY_BALANCED: Balanced quality and performance.
    • OIDN_QUALITY_FAST: Highest performance.

    Recommendation: Use OIDN_QUALITY_BALANCED or OIDN_QUALITY_FAST for interactive and real-time use cases.

  11. Interop with Compute and Graphics APIs

    master

    Open Image Denoise supports interoperability with external graphics and compute APIs (SYCL, CUDA, HIP, DX, Vulkan, Metal) to avoid copying texture data.

    Texture Sharing: Native texture sharing is not supported, but you can avoid copies by using a linear texture layout (e.g., VK_IMAGE_TILING_LINEAR in Vulkan) and sharing the underlying buffer. Ensure the row stride is correctly set.

    Synchronization: For efficient synchronization of shared memory, import external semaphores from your graphics API using:

    • oidnNewSharedSemaphoreFromFD
    • oidnNewSharedSemaphoreFromWin32Handle

    Query supported external memory types via the externalSemaphoreTypes device parameter. Use oidnSignalSemaphoresAsync and oidnWaitSemaphoresAsync for asynchronous signaling/waiting.

    Fallback: If external semaphores are unavailable, use oidnSyncDevice alongside your graphics API for host-side synchronization, though this may incur performance overhead.

  12. How filters work in Open Image Denoise

    master

    Filters are the primary objects responsible for denoising. Each filter is optimized for specific image types and use cases.

    Lifecycle and Performance:

    1. Creation: Create a filter using oidnNewFilter. This is an expensive operation.
    2. Reuse: It is strongly recommended to reuse the same filter object for multiple images if they share the same size, format, and features. Reusing a filter for different resolutions provides no benefit.
    3. Setup: Specify input/output images and parameters.
    4. Commit: You must call oidnCommitFilter to apply parameter changes. Committing major changes (like resolution) is expensive and should be avoided per-frame.
    5. Execution: Run the filter using oidnExecuteFilter (blocking) or oidnExecuteFilterAsync (non-blocking).
    6. Cleanup: Retain and release filter objects using oidnRetainFilter and oidnReleaseFilter.