RTX Neural Texture Compression (NTC) SDK

Inference on Sample is a mode where NTC decompression logic is executed directly within a pixel or ray tracing shader to decompress only the specific texels needed for a view.

Key Characteristics:

• Performance: Decompressing texels is computationally expensive compared to regular sampling. It returns one unfiltered texel with all material channels at a time.
• Hardware Requirements: It should ideally be used on high-performance GPUs supporting Cooperative Vector (CoopVec) extensions. While a DP4a fallback exists, it is significantly slower and intended only for functional validation.
• Filtering: Because simulating trilinear or anisotropic filtering is prohibitively expensive, Inference on Sample should be used in conjunction with Stochastic Texture Filtering (STF), followed by denoising or DLSS after shading.

NTC can compress full MIP chains (from highest LOD down to 1x1).

To achieve this, texture MIP levels are grouped into sets of 2-4 levels. Each group is compressed into a pair of latent tensors (one high-res and one low-res).

Note on quality: Due to grouping and sampling, compression quality may vary across MIP levels. This difference is most noticeable in LODs 3 and above, while high-detail levels remain largely unaffected.

NTC files may use GDeflate lossless compression for latent data. You can decompress this data using one of three methods:

1. GPU Decompression (Vulkan): Use IContext::DecompressGDeflateOnVulkanGPU(...) which utilizes the VK_NV_memory_decompression extension. The application must still manage data uploads and memory.
2. DirectStorage: Consume the compressed buffers directly using the DSTORAGE_COMPRESSION_FORMAT_GDEFLATE format. This requires implementing the DirectStorage API calls on the application side.
3. CPU Decompression: Use IContext::DecompressBuffer(...). This is the simplest method but the slowest.

Note: GDeflate compression is optional and can be configured via ITextureSet::ConfigureLosslessCompression(...) or the --gdeflate <option> flag in ntc-cli.

MIP chains can be handled in three ways:

1. Loading from directory: When using --loadImages and --loadMips, the tool scans for a mips/ subdirectory next to the base textures. It expects files named <base-name>.<mip>.png (e.g., albedo.01.png for MIP 1).
2. Loading from manifest: Exact file names for MIP levels can be specified within the JSON manifest.
3. Generating Mips: Using --generateMips automatically creates MIP levels 1 and above from mip 0 before compression. If --saveMips is also used, these generated levels are saved using the same directory layout described in point 1.

# Example directory structure for --loadMips
material-name/
    albedo.png # MIP 0
    normal.png
    mips/
        albedo.01.png
        normal.01.png
        albedo.02.png
        normal.02.png

NTC acts as an adjustable quality/constant bitrate lossy compression scheme. The per-texel memory footprint is determined by the Latent Shape, which is a composite of:

• The number of high- and low-resolution latent channels.
• The bit-depth of those channels.
• The scale factor between them.

To achieve a specific bitrate, you must select a compatible Latent Shape. For an approximate constant quality/variable bitrate approach, you can perform pre-analysis of the texture bundle to determine the required formats. See the Settings and Quality Guide for detailed configuration.

The NTC container format is a binary format modeled after GLB (Binary GLTF). It stores data in little-endian format and consists of a file header followed by two primary chunks: a JSON descriptor and a binary data chunk.

File Header Structure

• Signature: 4 bytes, NTEX
• Container version: 4 bytes (Current: 0x100)
• JSON chunk offset: 8 bytes
• JSON chunk size: 8 bytes
• Data chunk offset: 8 bytes
• Data chunk size: 8 bytes

Key Constraints

• Alignment: Chunk offsets and sizes must be multiples of 4 bytes to ensure efficient access (especially for GPU memory loading). If data is not a multiple of 4 bytes, it must be padded with zeros.
• Indexing: The JSON descriptor uses offsets relative to the start of the binary data chunk to locate specific data segments.

Image quality is reported as Peak Signal-to-Noise Ratio (PSNR) in decibels (dB).

• 35-40 dB: Generally considered sufficient for most uses.
• 50 dB: Perceptually lossless.

Note that for HDR images (EXR), PSNR values are not correctly normalized and should only be compared against other runs of the same type, not against other image comparison methods.

NTC supports High Dynamic Range (HDR) images (more than 8 bits per channel, values > 1.0).

To ensure compatibility with the neural decoder, the library automatically converts HDR images to the Hybrid Log-Gamma (HLG) color space before compression and linearizes them after decompression. This is enabled automatically in the CLI for all .exr images.

Consequences of HLG conversion:

1. Channel values with high magnitude are represented with lower accuracy.
2. Reported PSNR values for HDR images are not correctly normalized for comparison with standard SDR methods.

NTC manifest files use the JSON format to define a texture set. The root object specifies the textures to be included and can optionally define custom dimensions for the resulting texture set.

Root Document Fields

The Pixel Inspector (located in the top-right of the UI) allows for precise per-pixel analysis. It displays the raw RGBA values for the pixel currently under the mouse pointer for both the left and right image slots, regardless of whether you are in 2D or 3D view mode.

Value Formats:

• If the active image uses the UNORM8 pixel format, values are displayed as integers.
• For all other pixel formats, values are displayed as floating-point numbers.