UpscalerJS

UpscalerJS is an open-source library for enhancing images using AI. It is compatible with Browser, Node (both CPU and GPU), and Workers environments.

Key features include:

• Pretrained Models: Support for increasing resolution, denoising, and deblurring.
• UI-Focused Enhancement: Uses patch-based processing to maintain performance without sacrificing quality.
• Custom Models: Ability to integrate your own pretrained models.
• Cross-Device Compatibility: Works consistently across desktops, tablets, and phones.

UpscalerJS is an open-source JavaScript tool for enhancing images using AI. It is built on top of TensorFlow.js and is designed to perform super-resolution (increasing image size while maintaining or enhancing quality) using neural networks.

Key features include:

• Multi-environment support: Runs in the browser, Node.js, and Worker environments (such as Cloudflare Workers).
• Out-of-the-box models: Ships with pre-trained models so users can start upscaling immediately without manual model installation.
• Client-side processing: By running inference on the user's device, it offers improved privacy (images never leave the device), reduced server costs, and lower latency by avoiding server round-trips.

The MAXIM Experiments models in this repository are TensorFlow ports of the MAXIM family of models.

Key Characteristics:

• Performance: They run significantly faster than the Jax port.
• Trade-offs: They exhibit more artifacting and have inferior performance compared to the original models.
• Limitations: They operate only on fixed-size image chunks. Using these models on certain image sizes can introduce severe artifacting.

If you encounter significant artifacting, refer to the known issues in the UpscalerJS repository.

The MAXIM Dehazing Outdoor model is unquantized and supports dynamic input sizes.

Key constraints:

• Image Sizes: Must be a multiple of 64. UpscalerJS handles padding and trimming automatically.
• Patch Sizes: If providing a custom patch size, it must be a multiple of 64. If it is not, UpscalerJS will emit a warning and increase the patch size to the nearest multiple of 64.

The MAXIM models in UpscalerJS have specific requirements regarding input dimensions:

• Input Size: The model has a dynamic input size. Image dimensions must be a multiple of 64. UpscalerJS handles padding and trimming automatically to satisfy this.
• Patch Size: If you provide a custom patchSize, it must also be a multiple of 64. If it is not, UpscalerJS will emit a warning and automatically increase the patch size to the nearest multiple of 64 to ensure compatibility.
• Quantization: These models are unquantized.

When you provide a tensor as input or request a tensor as output, you take over responsibility for memory management. You must explicitly call .dispose() on these tensors to prevent memory leaks.

Important: While TensorFlow.js provides tf.tidy() for synchronous operations, most UpscalerJS methods are asynchronous. Therefore, you should manually call .dispose() on tensors once they are no longer needed within your .then() blocks or after await calls.

// Example of manual memory management
upscaler.upscale(tensor, {
  output: 'tensor',
}).then(upscaledTensor => {
  // 1. Dispose of the original input tensor
  tensor.dispose()

  // 2. Use the upscaled tensor
  upscaledTensor.print()

  // 3. Dispose of the upscaled tensor when finished
  upscaledTensor.dispose()
})

The standard MAXIM Deblurring model is unquantized and supports dynamic input sizes.

Requirements:

• Image Sizes: Must be a multiple of 64. UpscalerJS handles padding and trimming automatically.
• Patch Sizes: If providing a custom patch size, it must be a multiple of 64. If it is not, UpscalerJS will emit a warning and increase the patch size to the nearest multiple of 64.

Performance Note: The model is based on the MAXIM (Multi-Axis MLP for Image Processing) architecture, designed for efficient image enhancement tasks like deblurring, denoising, and deraining.

The standard MAXIM Retouching model is unquantized and supports dynamic input sizes.

Key constraints:

• Image Dimensions: Input image sizes must be a multiple of 64. UpscalerJS handles padding and trimming automatically to satisfy this.
• Patch Size: If you provide a custom patchSize, it must also be a multiple of 64. If it is not, UpscalerJS will emit a warning and automatically increase the patch size to the nearest multiple of 64.