handtrack.js

You can include Handtrack.js directly in your HTML using a script tag from jsDelivr. Ensure that any images you use have appropriate CORS settings to allow the library to read them.

<!-- Load the handtrackjs model. -->
<script src="https://cdn.jsdelivr.net/npm/handtrackjs@latest/dist/handtrack.min.js"></script>

<!-- Replace this with your image. -->
<img id="img" src="hand.jpg"/>  

<script>
  const img = document.getElementById('img'); 
  const model =  await handTrack.load();
  const predictions = await model.detect(img); 
</script>

To run the included demo example from the demo folder, use the following commands. This will launch a Python webserver, and you can view the demo at http://localhost:3005/.

npm install
npm run start

To use Handtrack.js in your project, install the package using npm. Note that this library is designed for browser-based (frontend JavaScript) environments and is not intended for Node.js.

npm install --save handtrackjs

If you are using a module bundler, you can import Handtrack.js directly from the package.

import * as handTrack from 'handtrackjs';

const img = document.getElementById('img'); 
const model =  await handTrack.load();
const predictions = await model.detect(img); 

The current version (v0.1.x) supports the following hand pose labels:

• Open: All fingers are extended in an open palm position.
• Closed: All fingers are contracted in a ball in a closed fist position.
• Pinch: The thumb and index finger are together in a picking gesture.
• Point: Index finger is extended in a pointing gesture.
• Face: Used to disambiguate between the face and hands.

Handtrack.js supports multiple models (e.g., ssd320fpnlite, ssd640fpnlite) with different quantization levels to manage model weight size. Note that smaller models do not necessarily result in faster inference speed; all sizes yield approximately the same FPS.

• Large: The default fp32 version.
• Medium: fp16 quantized version.
• Small: Int8 quantized version (e.g., ssd320fpnlite small is ~3MB).

Once an ObjectDetection instance is loaded, use the detect(input) method to identify hands in a frame. The input can be an HTML <video> element, an <img> element, or a tf.Tensor.

Returns an array of detected objects. Each object contains:

• bbox: [minX, minY, width, height]
• class: The integer class ID
• label: The string label (e.g., 'open', 'closed', 'pinch')
• score: The confidence score as a string (e.g., '0.95')

You can update the configuration of an existing ObjectDetection instance using setModelParameters(params).

Available parameter keys (from defaultParams):

• flipHorizontal: boolean
• outputStride: number
• imageScaleFactor: number
• maxNumBoxes: number
• iouThreshold: number
• scoreThreshold: number
• modelType: string (e.g., 'ssd320fpnlite')
• modelSize: 'large' | 'medium' | 'small'
• bboxLineWidth: string
• fontSize: number
• basePath: string (URL to models)
• labelMap: object mapping IDs to labels

The library provides utility functions to manage the webcam stream:

• startVideo(video): Requests webcam access via getUserMedia and attaches the stream to the provided <video> element. Returns a Promise that resolves with { status: true, msg: '...' } on success or { status: false, msg: err } on failure.
• stopVideo(): Stops all tracks in the current local stream. Returns true if a stream was stopped, or false if no stream was active.

The library uses a specific mapping for hand states. You can use these labels when processing detection results or customizing your own UI.

// Labels used by the model
const labelMap = {
  1: "open",
  2: "closed",
  3: "pinch",
  4: "point",
  5: "face",
  6: "tip",
  7: "pinchtip",
};

// Default colors associated with labels
const colorMap = {
  open: "#374151",
  closed: "#B91C1C",
  pinch: "#F59E0B",
  point: "#10B981",
  face: "#3B82F6",
  tip: "#6366F1",
  pinchtip: "#EC4899",
};