DOCSEARCHbyAlgolia
Submit Docs

Stanford Nimble

repository·master·Indexed 19 days ago

https://github.com/keenon/nimblephysics

A differentiable physics engine designed for use as a non-linearity within neural networks. It provides PyTorch bindings that support analytical backwards passes through contact and friction, allowing physics simulations to be treated as differentiable PyTorch functions.

Tokens
29.1K
Snippets
76
Records
106
Agent score
67%

What's inside nimblephysics

  1. What is Nimble Physics?

    master

    Nimble is a toolkit designed for performing AI research on human biomechanics using physically simulated realistic human bodies. It is written in C++ for high performance and provides Python bindings for ease of use.

    Key features include:

    • Differentiable Physics: Built as a differentiable fork of the DART physics engine, allowing for gradient-based optimization.
    • Biological Joints: Support for specialized, differentiable joint types such as CustomJoint (e.g., knees), ConstantCurvatureJoint (e.g., spines), and EllipsoidJoint (e.g., shoulders).
    • OpenSim Integration: Ability to load, modify, and save OpenSim skeleton models.
    • Motion Capture: Support for handling raw motion capture data.
    • System Identification: Treats bone scales and optical marker offsets as first-class differentiable quantities.
    • Optimization: Includes various optimization algorithms and optimized computations for Jacobians and gradients through human body quantities.

  2. Understand the contents and processing passes of a B3D file

    master

    A .b3d file is an efficient binary format used by Nimble to store all processed trials for a single subject, including scaled/mass-tuned skeleton models and raw sensor inputs. Data is organized into "passes," which are recorded separately. This allows you to access different stages of the motion capture processing pipeline.

    Key processing pass types (accessible via nimble.biomechanics.ProcessingPassType):

    • kinematics: Uses marker data to solve for bone scales, marker offsets, and skeleton poses. This pass has not "seen" force data.
    • dynamics: Uses the output of previous passes (typically kinematics) to solve for mass distribution and fine-tune motion to match experimental force data.
    • lowPassFilter: Applies a low-pass filter to the output of a previous pass.

    Machine Learning Tip: If training a model to predict physical data (like ground reaction force) from motion, use the kinematics pass as input. Using the dynamics pass output may cause overfitting because that data has already been optimized to match the force plate data.

  3. Represent IMUs in Nimble

    master

    To represent accelerometers, gyroscopes, or magnetometers, you must define a sensor location relative to a bone. This is done using a list of tuples, where each tuple contains a nimble.dynamics.BodyNode (the bone the sensor is attached to) and a nimble.math.Isometry3 (the translation and rotation of the sensor within that bone's frame).

    For example, if you have an IMU that contains both an accelerometer and a gyroscope, you would create two separate lists (one for each sensor type) containing the same (BodyNode, Isometry3) pairs.

    import nimblephysics as nimble
import numpy as np
from typing import List, Tuple

# Define the bone and the offset (Isometry3)
right_wrist: nimble.dynamics.Joint = skeleton.getJoint("radius_hand_r")
translation: np.ndarray = np.array([0.0, 0.05, 0.0])
rotation: np.ndarray = np.eye(3)
watch_offset: nimble.math.Isometry3 = nimble.math.Isometry3(rotation, translation)

# Create the sensor list
sensors: List[Tuple[nimble.dynamics.BodyNode, nimble.math.Isometry3]] = [(right_wrist, watch_offset)]

  4. Understand Skeletons in Nimble

    master

    In Nimble, the primary unit of simulation is the nimble.dynamics.Skeleton. A Skeleton is a tree structure composed of nimble.dynamics.Joint objects that connect nimble.dynamics.BodyNode objects.

    Key Mental Models:

    • Root Attachment: Every Skeleton's root is rigidly attached to the world origin. To simulate a "free-floating" object (like a drone or a person walking), you must include a root joint that allows translation and rotation relative to the origin.
    • Generalized Coordinates: Nimble operates in generalized coordinates. The state of a skeleton (position, velocity, acceleration) is represented as a vector where each element corresponds to a specific degree of freedom (DOF) of a joint.

  5. Differentiable Physics with `nimble.timestep()`

    master

    Nimble is designed to work seamlessly with PyTorch. The core function nimble.timestep() is a fully differentiable operator. This allows you to perform backpropagation through physics timesteps to optimize initial conditions, control inputs, or other physical parameters.

    In a typical optimization loop, you:

    1. Concatenate initial position and velocity into a single state tensor.
    2. Iterate through timesteps using nimble.timestep(world, state, control_input).
    3. Calculate a loss based on the final state.
    4. Call loss.backward() to compute gradients through the entire simulation trajectory.
    # Example of a differentiable simulation loop
state: torch.Tensor = torch.cat((initial_position, initial_velocity), 0)

num_timesteps = 100
for i in range(num_timesteps):
    # timestep is differentiable
    state = nimble.timestep(world, state, torch.zeros((world.getNumDofs())))

# Calculate loss from the final state
final_position = state[:world.getNumDofs()]
loss = final_position.norm()

# Backpropagate through the physics
loss.backward()

  6. Efficiently load B3D data for Machine Learning

    master

    B3D files are designed for efficient random access, making them suitable for training ML models without overwhelming system RAM.

    When using nimble.biomechanics.SubjectOnDisk, the entire file is not loaded into memory. Instead, it maintains a lightweight index. You can instantiate many SubjectOnDisk objects simultaneously.

    To train your model:

    1. Instantiate SubjectOnDisk to index the file.
    2. Use readFrames(...) to load specific arrays of Frame objects into memory as needed.
    3. Use the data within the Frame objects (which contain information for all processing passes) to derive features for your ML system.

  7. Compose custom Jacobians using the chain rule

    master

    If a specific Jacobian is not provided by Nimble, you can compose it manually using the chain rule by multiplying existing Jacobians. For a composition $x = f(g(y))$, the Jacobian $\frac{\partial x}{\partial y}$ is calculated as the product of $\frac{\partial f}{\partial g}$ and $\frac{\partial g}{\partial y}$.

    d_x_d_g = ... # Get this from Nimble
d_g_d_y = ... # Get this from Nimble
d_x_d_y = d_x_d_g @ d_g_d_y

  8. Use physics as a differentiable PyTorch function

    master

    Nimble allows you to use physics simulations as a non-linearity within a neural network. Because everything is a PyTorch Tensor, the simulation is differentiable. A single timestep can be treated as a valid PyTorch function, supporting an analytical backwards pass that works through contact and friction.

    To perform a forward pass for a single timestep, use the timestep function.

    from nimble import timestep

# Everything is a PyTorch Tensor, and this is differentiable!!
next_state = timestep(world, current_state, control_forces)

  9. Understand marker mismatch info messages

    master

    During data ingestion, Nimble provides INFO messages regarding discrepancies between the motion capture (mocap) data and the physics model markers:

    • Marker in mocap but not on model: Markers like LPAT, P1, or x53 exist in the source data but are not defined in the target physics model. These are typically ignored.
    • Marker in model but not in mocap: Markers like BLFE, CHIN, or CLAV are defined in the model but absent from the source data. This is common when the model uses virtual markers (markers that are calculated rather than physically tracked). These messages are informational and do not indicate an error.

  10. Evaluate marker RMSE and error metrics

    master

    Each processed .trc file provides error metrics to assess the quality of the marker tracking relative to the model.

    • RMSE: Root Mean Square Error (e.g., RMSE: 1.17cm).
    • Max: The maximum error recorded (e.g., Max: 2.61cm).
    • Worst Markers: A ranked list of markers with the highest error (RMSE) is provided, allowing you to identify specific anatomical points that may require better sensor placement or cleaning.

  11. How the Nimble GUI hash table works

    master

    The Nimble GUI is conceptually a hash table where keys map to 3D objects.

    • Creation: Using gui.nativeAPI().createBox(key, ...) or higher-level methods like gui.nativeAPI().renderSkeleton(...) adds objects to the table. If a key already exists, the object is overwritten.
    • Updates: You can update specific attributes of an existing object without recreating it using methods like gui.nativeAPI().setObjectPosition(key, position).
    • Deletion:
      • Delete a single object: gui.nativeAPI().deleteObject(key)
      • Delete a group of objects: gui.nativeAPI().deleteObjectsByPrefix(prefix)
      • Clear everything: gui.nativeAPI().clear()
    # Example of updating an object instead of recreating it
gui.nativeAPI().setObjectPosition("my_box", [1.0, 2.0, 3.0])

# Example of deleting by prefix
gui.nativeAPI().deleteObjectsByPrefix("robot_arm_")

  12. Understand Worlds and Skeletons

    master

    In Nimble, a World is a collection of Skeletons. Each Skeleton is a tree of joints that connect rigid bodies together.

    Key concepts:

    • Skeletons: Represent a single articulated body (like a robot arm). Every Skeleton's root is rigidly attached to the world origin by default. To simulate a 'free-floating' object, you must include a root joint (like a FreeJoint) that allows movement relative to the origin.
    • Worlds: Act as a container for multiple Skeletons. The World's state (position and velocity) is a concatenation of all its Skeletons' states, ordered by when they were added.