DOCSEARCHbyAlgolia
Submit Docs

Paddle.js Documentation

repository·release/v2.2.5·Indexed 22 days ago

https://github.com/paddlepaddle/paddle.js

A web-based deep learning framework for running PaddlePaddle models in browsers or mini-programs. It supports multiple hardware acceleration backends including CPU, WebAssembly (WASM), WebGL, WebGPU, and NodeGL for Node.js environments. The framework includes @paddlejs/paddlejs-core for inference management, a model converter (paddlejsconverter), and benchmarking tools.

Tokens
40K
Snippets
148
Records
193
Agent score
73%

What's inside Paddle.js

  1. Overview of Paddle.js

    release/v2.2.5

    Paddle.js is an open-source deep learning framework for the browser, based on Baidu PaddlePaddle. It allows developers to run deep learning models directly in web browsers or environments like Baidu Smartprogram and WX miniprogram.

    Key capabilities include:

    • Loading pre-trained models.
    • Transforming models from paddle-hub using provided tools.
    • Running inference via multiple backends (WebGL, WebGPU, WASM, CPU, and NodeGL).
    • Supporting various AI scenarios through specialized model libraries (OCR, Gesture, Human Segmentation, etc.).

  2. Use the NodeGL backend for Paddle.js in Node.js

    release/v2.2.5
    The @paddlejs/paddlejs-backend-nodegl package provides a computation solution for running Paddle.js models within a Node.js environment. It implements the same operators used in the WebGL backend but utilizes the gl (headless-gl) npm package to create a WebGL context directly in Node.js, eliminating the need for a browser environment.

  4. Perform data post-processing for face detection

    release/v2.2.5

    Because the face detection model performs better on small-sized faces, the input image is resized (shrunk) before prediction. To get the correct coordinates relative to the original image, you must apply a data post-processing step to the output.

    When transforming the predicted output (red line in documentation diagrams) to the original image scale (green line), you need to account for the transformation using known values: dx, dy, fw (factor width), and fh (factor height).

  5. How OCR detection and recognition work

    release/v2.2.5

    The OCR process follows a two-step pipeline:

    1. Detection (ocr_detection): Locates the regions of text within an image.
    2. Recognition (ocr_recognition): Recognizes characters within those detected areas. The recognition model expects an input shape of [1, 3, 32, 320].

    Image Processing Logic

    When processing a detected text area:

    • If the width-to-height ratio of the area is ≤ 10, the entire area is passed to the recognition model.
    • If the width-to-height ratio is > 10, the area is cropped along its width. The recognition model processes these cropped parts separately, and the final results are spliced together to form the complete text string.

  6. Understand RNN and LSTM implementation in Paddle.js

    release/v2.2.5

    In Paddle.js, Recurrent Neural Network (RNN) operations are primarily implemented using LSTM (Long Short Term Memory) to mitigate the vanishing gradient problem and capture long-term dependencies.

    For sequence-based tasks like OCR (e.g., ch_ppocr_mobile_v2.0_rec_infer), Paddle.js often utilizes Bidirectional LSTM structures, where two LSTMs (one forward and one backward) are combined, and multiple layers can be stacked.

    RNN Operator Configuration Example

    When inspecting a converted model (like a PP-OCR model), the RNN operator configuration follows this structure:

    {
	Attr: {
		mode: 'LSTM'
		// Whether it is bidirectional; if true, both forward and backward directions are traversed
		is_bidirec: true
		// Number of hidden layers (number of recurrent iterations)
		num_layers: 2
	}
	
	Input: [
		transpose_1.tmp_0[25, 1, 288]
	]

	PreState: [
		fill_constant_batch_size_like_0.tmp_0[4, 1, 48],  
		fill_constant_batch_size_like_1.tmp_0[4, 1, 48]
	]

	WeightList: [
		lstm_cell_0.w_0[192, 288], lstm_cell_0.w_1[192, 48], 
		lstm_cell_1.w_0[192, 288], lstm_cell_1.w_1[192, 48], 
		lstm_cell_2.w_0[192, 96], lstm_cell_2.w_1[192, 48], 
		lstm_cell_3.w_0[192, 96], lstm_cell_3.w_1[192, 48],
		lstm_cell_0.b_0[192], lstm_cell_0.b_1[192],
		lstm_cell_1.b_0[192], lstm_cell_1.b_1[192],
		lstm_cell_2.b_0[192], lstm_cell_2.b_1[192], 
		lstm_cell_3.b_0[192], lstm_cell_3.b_1[192]
	]

	Output: [
	    lstm_0.tmp_0[25, 1, 96]
	]
}
    {
	Attr: {
		mode: 'LSTM'
		is_bidirec: true
		num_layers: 2
	}
	Input: [
		transpose_1.tmp_0[25, 1, 288]
	]
	PreState: [
		fill_constant_batch_size_like_0.tmp_0[4, 1, 48],  
		fill_constant_batch_size_like_1.tmp_0[4, 1, 48]
	]
	WeightList: [
		lstm_cell_0.w_0[192, 288], lstm_cell_0.w_1[192, 48], 
		lstm_cell_1.w_0[192, 288], lstm_cell_1.w_1[192, 48], 
		lstm_cell_2.w_0[192, 96], lstm_cell_2.w_1[192, 48], 
		lstm_cell_3.w_0[192, 96], lstm_cell_3.w_1[192, 48],
		lstm_cell_0.b_0[192], lstm_cell_0.b_1[192],
		lstm_cell_1.b_0[192], lstm_cell_1.b_1[192],
		lstm_cell_2.b_0[192], lstm_cell_2.b_1[192], 
		lstm_cell_3.b_0[192], lstm_cell_3.b_1[192]
	]
	Output: [
	    lstm_0.tmp_0[25, 1, 96]
	]
}

  7. Extend Paddle.js with custom operators and transformers

    release/v2.2.5

    Paddle.js Core provides extension points for customizing the inference engine:

    1. Custom Operators: Use the registerOp interface to register custom operators.
    2. Model Transformation: Use runnerConfig.plugins to modify the model structure. You can add, delete, or change layers (e.g., pruning unnecessary layers for speed or adding custom post-processing layers directly into the model topology).

  9. How the OCR module works

    release/v2.2.5

    The @paddlejs-models/ocr module provides text recognition capabilities using two distinct models:

    1. ocr_detection: Detects the specific regions (text boxes) containing text within an image.
    2. ocr_recognition: Recognizes characters (Chinese, English, and numbers) within those detected text areas.

    Processing Logic: The ocr_recognition model expects an input shape of [1, 3, 32, 320]. To handle varying aspect ratios, the module processes selected text areas as follows:

    • If the width-to-height ratio of the text box is ≤ 10, the entire area is passed to the recognition model.
    • If the width-to-height ratio is > 10, the area is cropped along its width into multiple parts. Each part is processed by the model, and the resulting character sequences are spliced together to form the final recognition result.

  10. Understand Facedetect post-processing requirements

    release/v2.2.5

    The Facedetect model is optimized for recognizing small-sized faces. To achieve this, the input image is shrunk before prediction.

    Important: Because the prediction happens on a shrunken version of the image, the output coordinates (bounding boxes) must be transformed back to the original image dimensions to be useful for the actual image. If you use the shrink parameter, you must apply this coordinate transformation to the resulting left, top, width, and height values.

  11. How to use Paddle.js in a WebWorker

    release/v2.2.5

    To offload Paddle.js computations from the main thread and prevent UI blocking, you can run it within a WebWorker. This implementation relies on several browser APIs to transfer data and rendering capabilities efficiently:

    1. OffscreenCanvas: Allows the WebWorker to render graphics without direct access to the DOM.
    2. createImageBitmap(): Used to create efficient, transferrable bitmap images.
    3. postMessage with Transferables: To avoid expensive data copying between the main thread and the worker, use Transferable objects. This project supports transferring ArrayBuffer, MessagePort, and ImageBitmap.

    By using these APIs, you can achieve high-performance machine learning inference in the background while keeping the main thread responsive.

    // Example of transferring an ArrayBuffer to a worker
worker.postMessage(arrayBuffer, [arrayBuffer]);

  12. Understand the Paddle.js Ecosystem

    release/v2.2.5

    The Paddle.js ecosystem is divided into core engines, backends, and specialized model SDKs:

    Core & Tools

    • paddlejs-core: The central inference engine responsible for the inference process.
    • paddlejs-converter: A tool to convert PaddlePaddle (fluid) models into browser-compatible formats.
    • paddlejs-models: A library of pre-packaged model SDKs providing easy-to-use APIs for common AI tasks.
    • paddlejs-mediapipe: A utility library for data stream processing (WebRTC video streams, lightweight OpenCV, etc.).

    Computation Backends

    • paddlejs-backend-webgl: The most mature backend with the widest operator support.
    • paddlejs-backend-webgpu: An experimental backend for WebGPU (currently in draft stage).
    • paddlejs-backend-wasm: WebAssembly-based computation.
    • paddlejs-backend-cpu: Standard CPU computation.
    • paddlejs-backend-nodegl: Enables prediction in Node.js environments using WebGL-based operators.