TensorFlow.js Models

This repository provides a collection of pre-trained models ported to TensorFlow.js. These models are available via NPM and unpkg, allowing them to be used out-of-the-box in any web project.

Key characteristics:

• Ease of Use: Most models are designed to hide low-level tensor operations, making them accessible to developers who are not machine learning experts.
• Versatility: Models can be used directly for inference or integrated into transfer learning workflows using TensorFlow.js.
• Extensibility: Models serve as building blocks for more complex applications.

The Speech Command Recognizer is a JavaScript module built on TensorFlow.js that enables the recognition of spoken commands consisting of simple, isolated English words. It utilizes the WebAudio API for audio input and leverages WebGL GPU acceleration for inference and transfer learning directly in the browser.

Default Vocabulary:

• Digits: zero through nine
• Directions: up, down, left, right
• Actions: go, stop
• Affirmations/Negations: yes, no
• Special categories: unknown word, background noise

The PoseNet demos provide two primary use cases for pose estimation:

• Camera Demo: Estimates poses in real-time using a webcam video stream.
• Coco Images Demo: Estimates poses in static images from the COCO dataset. This demo also demonstrates the differences between single-person and multi-person pose detection algorithms.

There are two primary ways to interact with the face-landmarks-detection models via web demos:

• Live Camera Demo: Uses your device's camera for real-time face tracking. It is compatible with laptops, iPhones, and Android phones. You can switch between different runtimes to compare performance.
• Upload a Video Demo: Allows you to upload an .mp4 video file. After processing, the demo automatically downloads the video with the detected face landmarks applied.

The pose-detection package provides three primary model options:

The depth-estimation models power several interactive demos that showcase different capabilities:

• 3D Photo Demo: Upload a portrait image to create a 3D photo animation. Works on laptops, iPhones, and Android phones. View 3D Photo Demo

• Relighting Demo: Uses a live camera stream to allow placing light sources or point lights to change the scene's lighting. Works on laptops, iPhones, and Android phones. View Relighting Demo

• Depth Map Demo: Uses a live camera stream to visualize depth in real-time. Works on laptops, iPhones, and Android phones. View Depth Map Demo

BodyPix is a TensorFlow.js model used for person segmentation in the browser. It can identify pixels belonging to a person and can further segment specific body parts.

Note: This README is for the archived 1.0 version. Version 2.0 introduced multi-person support, a new ResNet model, and a new API. For the latest features, refer to the current documentation.

To use BodyPix, you follow a two-step process:

1. Loading the model: Use the provided loading function to initialize the model.
2. Making a prediction: Pass an image element (or tensor) to the model to receive segmentation data.

Available segmentation types:

• Person segmentation: Identifies which pixels belong to a person.
• Person body part segmentation: Identifies specific body parts (e.g., arms, legs, face) within the person segmentation.

BodyPix is a model for real-time person and body part segmentation in the browser.

Key capabilities:

• Person Segmentation: Segments an image into pixels that are part of a person and pixels that are not.
• Body Part Segmentation: Segments pixels into twenty-four distinct body parts.

Usage Note: The model is optimized for a single person centered in the input image or video. To segment multiple people, you must combine BodyPix with a person detector by cropping boxes for each detected person and running segmentation on those individual crops.

Unlike standard TensorFlow.js models that provide pre-trained weights, the KNN Classifier is a utility for constructing a K-Nearest Neighbors model using activations (tensors) from another model or any other tensors you associate with a label.

The workflow is typically:

1. Initialize a classifier using knnClassifier.create().
2. Extract features from an input (e.g., using a model like MobileNet) to get a tensor of activations.
3. Train by adding these activations as examples with a specific label using addExample().
4. Predict by passing new activations to predictClass().

const classifier = knnClassifier.create();
// ... extract logits from another model ...
classifier.addExample(logits, label);
// ... predict ...
const result = await classifier.predictClass(newLogits);

MediaPipe SelfieSegmentation offers two model variants optimized for different needs:

1. general: Operates on a 256x256x3 tensor and outputs a 256x256x1 segmentation mask. This is the default.
2. landscape: Operates on a 144x256x3 tensor. It has fewer FLOPs and is optimized for faster execution.