Transformers.js

Transformers.js supports a wide variety of pre-trained models across different tasks, including language modeling, vision, audio, and time-series forecasting. Key model families supported include:

• Language Models: MT5, NanoChat, Nemotron, Nemotron-H, NeoBERT, OLMo (including OLMo2, OLMo3, and Olmo Hybrid), OpenELM, OPT, Phi (including Phi3 and Phi3V).
• Vision & Multimodal: OWL-ViT, OWLv2, PaliGemma, PVT.
• Audio: NLLB (Translation), Nougat (Document Understanding), Parakeet (ASR), PyAnnote (Speaker Diarization).
• Time Series: PatchTSMixer, PatchTST.

For specific model details, refer to the individual model documentation links provided in the official repository.

When integrating Transformers.js into a React application, follow these two architectural patterns to ensure a smooth user experience:

1. Use Web Workers: ML inference is computationally intensive and can block the main UI thread. Offload the loading and running of pipelines to a separate Web Worker thread.
2. Lazy Loading: Since models can be very large (often >1 GB), do not load them on application startup. Instead, trigger the model download and initialization only when the user performs a specific action (e.g., clicking a "Translate" button).

To implement this, you can use a singleton pattern within your worker script to ensure the pipeline instance is created only once and reused for subsequent calls.

To start the development server for the Next.js AI Chatbot project, use the following command:

npm run dev

When running the application for the first time, the model will be downloaded and cached in your browser. Subsequent visits will load the model from the browser cache, avoiding repeated downloads.

To add machine learning capabilities to your React application, install the @huggingface/transformers package via npm.

npm install @huggingface/transformers

To build a React application for Transformers.js, use Vite to set up the project environment. This provides a fast development server and build tool.

1. Run the Vite initializer:

   npm create vite@latest react-translator -- --template react

2. Navigate to the directory and install dependencies:

   cd react-translator
   npm install

3. Start the development server:

   npm run dev

The pipeline API is the easiest way to run models. It groups a pretrained model with the necessary preprocessing and postprocessing. The API is functionally equivalent to the Hugging Face Python transformers library.

To use a default model for a specific task (e.g., 'sentiment-analysis'), await the pipeline function and then call the resulting object with your input.

import { pipeline } from '@huggingface/transformers';

// Allocate a pipeline for sentiment-analysis
const pipe = await pipeline('sentiment-analysis');

const out = await pipe('I love transformers!');
// Output example: [{'label': 'POSITIVE', 'score': 0.999817686}]

To deploy a server-side Next.js application to Hugging Face Spaces:

1. Create a Dockerfile in your project root (use a template suitable for Next.js).
2. Create a new Space on Hugging Face and select the Docker SDK.
3. Upload your project files (excluding node_modules and .next).
4. Add a YAML configuration block to the top of your README.md to define the Space metadata:

---
title: Next Server Example App
emoji: 🔥
colorFrom: yellow
colorTo: red
sdk: docker
pinned: false
app_port: 3000
---

To perform audio processing in a Node.js environment, you must manually handle audio decoding because the Web Audio API (AudioContext) is not available in Node.js. This guide uses the wavefile library to convert .wav files into the Float32Array format required by Transformers.js pipelines.

Prerequisites

• Node.js version 18+
• npm version 9+

Installation

Initialize your project and install the necessary dependencies:

npm init -y
npm i @huggingface/transformers
npm i wavefile

Note: Ensure you add "type": "module" to your package.json to use ECMAScript modules.

To use WebGPU for high-performance GPU-accelerated computations in the browser, pass { device: 'webgpu' } as an option when initializing a pipeline using the pipeline function. This leverages ONNX Runtime Web to interact directly with the system's GPU.

Note on Browser Support: As of late 2024, WebGPU support is approximately 70%. If WebGPU is not working, you may need to enable it via feature flags:

• Firefox: Enable dom.webgpu.enabled.
• Safari: Enable the WebGPU feature flag.
• Older Chromium browsers: Enable the enable-unsafe-webgpu flag.

import { pipeline } from "@huggingface/transformers";

const classifier = await pipeline(
  "image-classification",
  "onnx-community/mobilenetv4_conv_small.e2400_r224_in1k",
  { device: "webgpu" },
);