DOCSEARCHbyAlgolia
Submit Docs

TensorFlow.js

repository·master·Indexed 12 days ago

https://github.com/tensorflow/tfjs

An open-source, hardware-accelerated JavaScript library for training and deploying machine learning models in the browser and Node.js. It supports model inference performance measurement via multi-device and local benchmark tools, and provides integration and e2e testing with tags like #SMOKE, #REGRESSION, and #GOLDEN.

Tokens
50.3K
Snippets
174
Records
231
Agent score
98%

What's inside TensorFlow.js

  1. Overview of TensorFlow.js APIs

    master

    TensorFlow.js is composed of several specialized packages. You can import them individually to optimize bundle size.

    Core APIs

    • @tensorflow/tfjs-core: A flexible low-level API for neural networks and numerical computation.
    • @tensorflow/tfjs-layers: A high-level API implementing functionality similar to Keras.
    • @tensorflow/tfjs-data: An API to load and prepare data (analogous to tf.data).
    • @tensorflow/tfjs-converter: Tools to import a TensorFlow SavedModel to TensorFlow.js.
    • @tensorflow/tfjs-vis: In-browser visualization for models.
    • @tensorflow/tfjs-automl: APIs to load and run models produced by AutoML Edge.

  2. Overview of TensorFlow.js Packages

    master

    TensorFlow.js is a monorepo containing several specialized packages for machine learning in the browser and Node.js.

    Core APIs

    • TensorFlow.js Core: Low-level API for neural networks and numerical computation.
    • TensorFlow.js Layers: High-level API similar to Keras.
    • TensorFlow.js Data: API for loading and preparing data (analogous to tf.data).
    • TensorFlow.js Converter: Tools to import TensorFlow SavedModels to TensorFlow.js.
    • TensorFlow.js Vis: In-browser visualization tools.
    • TensorFlow.js AutoML: APIs for running models from AutoML Edge.

    Backends and Platforms

    • CPU Backend: Pure-JS backend for Node.js and the browser.
    • WebGL Backend: WebGL-accelerated backend for the browser.
    • WASM Backend: WebAssembly backend for the browser.
    • WebGPU Backend: WebGPU backend for the browser.
    • Node.js: Node.js platform via a TensorFlow C++ adapter.
    • React Native: React Native platform via an expo-gl adapter.

  3. Convert TensorFlow models to a web-friendly format

    master

    The TensorFlow.js converter is a Python library used to convert pretrained models (such as TensorFlow SavedModel, TensorFlow Hub modules, Keras HDF5, tf.keras SavedModel, or Flax/JAX models) into a web-friendly format for use with the TensorFlow.js JavaScript API.

    The conversion process follows two main steps:

    1. Conversion (Python): Use the tensorflowjs pip package to convert your model to a web-friendly format.
    2. Inference (JavaScript): Use the TensorFlow.js JavaScript API to load the converted model and run inference in the browser.

    Note: The Session bundle format is deprecated.

  4. Use the Headless WebGL backend for Node.js

    master
    The @tensorflow/tfjs-backend-nodegl package provides a lightweight, headless WebGL runtime for TensorFlow.js designed to run in Node.js environments. It is powered by the node-gles module, which utilizes ANGLE to interface with the system's GL runtime. This backend is intended as an acceleration engine for IoT, desktop, and Node.js applications where CUDA is unavailable due to size constraints or OS compatibility issues.

  5. Explore TensorFlow.js Demos and Applications

    master

    The TensorFlow.js Gallery contains a wide variety of real-world applications, research experiments, and demos. These projects demonstrate the capabilities of the library across different domains such as:

    • Computer Vision: Pose estimation (PoseNet, FaceMesh), object tracking, image segmentation (medical and general), and facial landmark detection.
    • Audio & Music: Hand-based music generation, neural drum machines, and keyword spotting.
    • Natural Language Processing (NLP): Textual similarity analysis, next-word prediction, and named entity recognition.
    • Generative Models: GAN visualizations and music generation.
    • Specialized Domains: Medical imaging (X-ray analysis, MRI segmentation), reinforcement learning (self-driving car environments), and plant disease identification.

    You can browse these demos to find inspiration or study their source code to understand how to implement similar features in your own applications.

  6. Overview of TensorFlow.js Backends and Platforms

    master

    TensorFlow.js supports various backends to accelerate computation depending on your environment:

    Browser Backends

    • @tensorflow/tfjs-backend-cpu: Pure-JS backend for the browser and Node.js.
    • @tensorflow/tfjs-backend-webgl: WebGL backend for hardware acceleration in the browser.
    • @tensorflow/tfjs-backend-wasm: WebAssembly backend for the browser.
    • @tensorflow/tfjs-backend-webgpu: WebGPU backend for the browser.

    Native/Mobile Platforms

    • @tensorflow/tfjs-node: Node.js platform via a TensorFlow C++ adapter.
    • @tensorflow/tfjs-react-native: React Native platform via an expo-gl adapter.

  7. What is the TensorFlow.js Core API?

    master

    The @tensorflow/tfjs-core package provides the foundational low-level capabilities of the TensorFlow.js ecosystem. It includes:

    • Hardware-accelerated linear algebra operations: High-performance math operations like matrix multiplication (matMul).
    • Eager API for automatic differentiation: Allows for immediate execution of operations and gradient computation.

    While the full @tensorflow/tfjs package includes high-level abstractions like the Layers API for neural network construction, the Core API is the engine that powers those higher-level modules.

  8. Understand the Nightly Build Pipeline

    master

    The automated nightly build process follows this sequence:

    1. Trigger: Cloud Scheduler writes to the nightly_tfjs topic at 4am.
    2. Execution: The nightly_tfjs function is triggered, which starts a Cloud Build programmatically.
    3. Status Reporting: The build runs and writes its status to the cloud-builds topic.
    4. Notification: The send_email function is triggered by the status update and sends an email and chat notification with the build status.

  9. Input data format for tfjs-inference

    master

    To perform inference, the inputs_dir must contain a set of files that collectively represent an array of tensors. The following files are required:

    1. inputs_data_file (e.g., data.json)
    2. inputs_shape_file (e.g., shape.json)
    3. inputs_dtype_file (e.g., dtype.json)
    4. tf_input_name_file (e.g., tf_input_name.json)

    These files must be present in the directory specified by --inputs_dir for the tool to function correctly.

  10. Difference between *Serialization and *Args types

    master

    In the TensorFlow.js Keras format implementation, two distinct type patterns are used to handle the transition from JSON to live objects:

    • *Serialization: Describes the raw, on-disk JSON representation. Nested objects in the config field are still in their serialized form (e.g., containing class_name and config).
    • *Args: Describes the actual arguments used to call a constructor. In this type, all nested serialized objects have already been deserialized into their actual class instances.

    Example Transformation:

    A FooSerialization containing a QuxSerialization is deserialized into FooArgs containing a Qux instance, which is then passed to the Foo constructor.

    // 1. The raw JSON structure (Serialization)
FooSerialization {
  class_name: 'Foo';
  config: {
    bar: string;
    baz: number[];
    qux: QuxSerialization;
  }
}

// 2. The resolved arguments for the constructor (Args)
FooArgs {
  bar: string;
  baz: number[];
  qux: Qux; // Note: QuxSerialization has become Qux
}

  11. Understand the Keras JSON serialization format

    master

    The Keras JSON format is used to serialize model components (like layers and models) into a JSON structure that mirrors the Python Keras API. Each serialized object is represented by a *Serialization type, which consists of two primary fields:

    1. class_name: A string identifying the class to be instantiated.
    2. config: An object containing the constructor arguments required to reconstruct the instance.

    When a configuration argument is itself an object, it is represented as a nested *Serialization object. During deserialization, these nested objects are recursively resolved into their actual class instances (e.g., converting a QuxSerialization into a Qux instance) before being passed to the parent constructor.

    // Example of the on-disk JSON structure represented by a *Serialization type
FooSerialization {
  class_name: 'Foo';
  config: {
    bar: string;
    baz: number[];
    qux: QuxSerialization;
  }
}

  12. Load AutoML models manually for non-browser platforms

    master

    If you cannot load models via HTTP (e.g., in non-browser environments), you can manually instantiate the model classes using a GraphModel and a dictionary loaded via your platform's specific IO handlers.

    import * as automl from '@tensorflow/tfjs-automl';
import * as tf from '@tensorflow/tfjs';

// For Image Classification
const graphModel = await tf.loadGraphModel('model.json');
const dict = loadDictionary("path/to/dict.txt");
const model = new automl.ImageClassificationModel(graphModel, dict);

// For Object Detection
const graphModelDet = await tf.loadGraphModel('model.json');
const dictDet = readDictionary("path/to/dict.txt");
const modelDet = new automl.ObjectDetectionModel(graphModelDet, dictDet);