ml5.js

ml5js uses a utility called handleArguments to allow developers to pass arguments to model constructors or methods in any order. Instead of strictly following a positional argument pattern, the library inspects the type of each argument provided and maps it to a named property.

Argument Mapping Logic

When you pass arguments, they are categorized as follows:

• string: Assigned to string (often used for model names or URLs).
• number: Assigned to number.
• function: Assigned to callback.
• Array: Assigned to array.
• HTMLVideoElement: Assigned to both video and audio.
• HTMLImageElement / HTMLCanvasElement / ImageData / tf.Tensor3D: Assigned to image.
• Other Objects: Any object that doesn't match the media types above is treated as an options object.

Usage Pattern

Because the library identifies arguments by type, you can often omit arguments or change their order. However, if you provide multiple arguments of the same type (e.g., two different objects), the last one provided will override the previous ones.

Example of flexible argument passing:

// Instead of: model.predict(image, options, callback)
// You can do:
model.predict(callback, image, options);

The HandPose model returns an array of hand objects. The raw TensorFlow.js output is enhanced for ease of use:

1. Confidence: The score property from the original model is renamed to confidence.
2. Keypoints: Each keypoint in the keypoints array is augmented with named properties. For each keypoint, you can access its coordinates via its name (e.g., if the keypoint is 'wrist', you can access hand.wrist).

Keypoint Structure: Each named keypoint object contains:

• x: 2D x-coordinate
• y: 2D y-coordinate
• x3D: 3D x-coordinate
• y3D: 3D y-coordinate
• z3D: 3D z-coordinate

const results = await myHandPose.detect(videoElement);

results.forEach(hand => {
  console.log(hand.confidence); // Renamed from score
  
  // Accessing augmented keypoints by name
  if (hand.wrist) {
    console.log('Wrist X:', hand.wrist.x, 'Wrist Y:', hand.wrist.y);
    console.log('Wrist 3D Z:', hand.wrist.z3D);
  }
});

ml5js models are designed to be flexible with input types. When providing data for image-based models, you can use any of the following formats:

• HTML Elements: HTMLImageElement, HTMLCanvasElement, HTMLVideoElement, or OffscreenCanvas.
• TensorFlow.js: A 3D Tensor (tf.Tensor3D).
• ImageData: Standard ImageData objects.
• p5.js: p5.Image, p5.Video, or any p5.Element (objects containing .elt or .canvas).

Note that HTMLVideoElement is treated as both an image and audio source by the internal argument handler.

// Example of the variety of inputs accepted by ml5 models:
// 1. HTML Image
const img = document.getElementById('my-image');
// 2. HTML Video
const video = document.getElementById('my-video');
// 3. p5.js Image
const p5Img = myP5Instance.loadImage('path/to/img.png');
// 4. TensorFlow Tensor
const tensor = tf.browser.fromPixels(img).expandDims(0); // (Note: models usually expect 3D tensors)

The NeuralNetwork class allows you to build, compile, and train custom TensorFlow.js models. You can define a sequential model, add layers, compile it with specific options, and train it using input and output tensors.

Workflow:

1. Instantiate NeuralNetwork (automatically calls createModel).
2. Use addLayer(layer) to add TensorFlow.js layers.
3. Use compile(options) to configure the model.
4. Use train(options) to start the training process.

Training Options: Pass an object to train() containing:

• inputs: The training input tensors.
• outputs: The training output tensors.
• batchSize: Number of samples per gradient update.
• epochs: Number of iterations over the dataset.
• shuffle: Boolean to shuffle data.
• validationSplit: Fraction of data to use for validation.
• whileTraining: Callbacks for the training lifecycle.

import NeuralNetwork from 'ml5/src/NeuralNetwork/NeuralNetwork';
import * as tf from '@tensorflow/tfjs';

const nn = new NeuralNetwork();

// 1. Add layers
nn.addLayer(tf.layers.dense({ units: 10, activation: 'relu', inputShape: [5] }));
nn.addLayer(tf.layers.dense({ units: 1 }));

// 2. Compile
nn.compile({ optimizer: 'adam', loss: 'meanSquaredError' });

// 3. Train
await nn.train({
  inputs: tf.tensor2d([[1, 2, 3, 4, 5]]),
  outputs: tf.tensor2d([[1]]),
  epochs: 10,
  batchSize: 1
});