NVIDIA Physical AI Autonomous Vehicles Developer Kit

To allow a custom dataclass to be interpolated using the Interpolator class, it must inherit from Interpolatable and implement a create_interpolator(self, timestamps) method.

If the fields within your dataclass are not themselves Interpolatable, you must specify the interpolation method using the interpolation_method metadata key on the field.

Interpolatable provides several metadata dictionaries to simplify this:

• Interpolatable.LINEAR
• Interpolatable.CUBIC_SPLINE
• Interpolatable.SLERP
• Interpolatable.ROTATION_SPLINE
• Interpolatable.DEFAULT_RIGID_TRANSFORM_INTERPOLATION

import dataclasses
from typing import Any
import numpy as np
from physical_ai_av.utils.interpolation import Interpolatable, Interpolator

@dataclasses.dataclass
class MyState(Interpolatable):
    # Field is Interpolatable (if MyState implements create_interpolator)
    position: np.ndarray 
    
    # Field is a raw type, so we must provide metadata
    velocity: np.ndarray = dataclasses.field(
        metadata=Interpolatable.LINEAR
    )

    def create_interpolator(self, timestamps: np.ndarray) -> Interpolator["MyState"]:
        return Interpolator(timestamps, self)

# Usage
timestamps = np.array([0, 100], dtype=np.int64)
state = MyState(position=np.array([0,0,0]), velocity=np.array([1,1,1]))
interpolator = state.create_interpolator(timestamps)

# Interpolate at t=50
new_state = interpolator(50)

The CameraModel abstract base class defines the interface for converting between 3D rays in the camera frame and 2D pixel coordinates.

Coordinate Systems:

• Camera Frame: Right-handed. Z-axis points out of the camera, X-axis points right, and Y-axis points down.
• Pixel Coordinates: Origin at the top-left corner. X-axis points right, and Y-axis points down.

Core Methods:

• ray2pixel(ray): Converts a ray vector [..., 3] in the camera frame to pixel coordinates [..., 2].
• pixel2ray(pixel): Converts pixel coordinates [..., 2] to a normalized unit-length ray vector [..., 3] in the camera frame.
• is_out_of_bounds(pixel): Returns a boolean mask indicating if pixels are outside the [0, width) and [0, height) range.

from physical_ai_av.utils.camera_models import CameraModel

# Assuming an implementation of CameraModel exists
# ray: [..., 3]
# pixel: [..., 2]
pixel = camera_model.ray2pixel(ray)
ray = camera_model.pixel2ray(pixel)

if camera_model.is_out_of_bounds(pixel):
    print("Pixel is outside image boundaries")

The physical_ai_av.utils.tf.TransformTree allows you to manage and query spatial relationships between different frames (e.g., from an 'anchor' to 'ego' to a specific 'sensor').

1. Initialize: Create a tf.TransformTree().
2. Add Transforms: Use add_transform(parent_frame, child_frame, transform_data) to build the tree. transform_data can be egomotion data.
3. Lookup: Use lookup_transform(target_frame_info, source_frame_info) to find the transformation between two frames at a specific timestamp. FrameInfo requires the feature ID and the timestamp in microseconds.
4. Apply Transformations: Once you have a transform (e.g., tf_target_source), you can use its .apply() method to transform 3D points or poses.

from physical_ai_av.utils import tf

tf_tree = tf.TransformTree()
tf_tree.add_transform("anchor", "ego", egomotion)

for sensor_id, sensor_pose in extrinsics.sensor_poses.items():
    tf_tree.add_transform("ego", sensor_id, sensor_pose)

# Lookup transform at a specific time
frame_time = actual_timestamps_us[-1]
anchor_to_camera_frame = tf_tree.lookup_transform(
    tf.FrameInfo(camera, frame_time), 
    tf.FrameInfo("anchor")
)

The Transformable base class allows you to create dataclasses that can be automatically transformed using a spt.RigidTransform. This is useful for complex objects containing points, poses, or vectors.

Supported Transformable Types

When defining fields in a dataclass, use the Transformable.TRANSFORMABLE_TYPE_KEY in the field metadata to specify how the field should be treated:

• TransformableType.POINT: The field is an np.ndarray representing a point. It is transformed using rigid_transform.apply().
• TransformableType.POSE: The field is an spt.RigidTransform. It is transformed via matrix multiplication: rigid_transform * field_value.
• TransformableType.VECTOR: The field is an np.ndarray representing a direction/vector. It is transformed using only the rotation component: rigid_transform.rotation.apply().

Methods

• transform(rigid_transform): Returns a new instance of the dataclass with all transformable fields updated.
• transform_frame(frame_transform, skip_check=False): Transforms the instance using a FrameTransform object, which also updates the instance's frame_info.

import dataclasses
import numpy as np
import scipy.spatial.transform as spt
from physical_ai_av.utils.tf import Transformable, TransformableType

@dataclasses.dataclass
class MyRobotState(Transformable):
    frame_info: FrameInfo
    position: np.ndarray = dataclasses.field(
        metadata={Transformable.TRANSFORMABLE_TYPE_KEY: TransformableType.POINT}
    )
    orientation: spt.RigidTransform = dataclasses.field(
        metadata={Transformable.TRANSFORMABLE_TYPE_KEY: TransformableType.POSE}
    )

# Usage
# new_state = my_robot_state.transform(some_rigid_transform)

The create_interpolant function is a factory for creating specific interpolators for vectors, rotations, or rigid transforms.

Important Note: Extrapolation is disallowed for numeric interpolants (linear and cubic spline).

Supported Inputs and Methods:

• Vectors (numpy arrays): Use InterpolationMethod.LINEAR or InterpolationMethod.CUBIC_SPLINE.
• Rotations (scipy.spatial.transform.Rotation): Use InterpolationMethod.SLERP or InterpolationMethod.ROTATION_SPLINE.
• RigidTransforms: Use a RigidTransformInterpolationMethod object.

import numpy as np
from scipy.spatial.transform import Rotation
from physical_ai_av.utils.interpolation import create_interpolant, InterpolationMethod

timestamps = np.array([0, 10, 20], dtype=np.int64)
values = np.array([0.0, 1.0, 2.0])

# Create a linear interpolant for a vector
interpolant = create_interpolant(InterpolationMethod.LINEAR, timestamps, values)

# Use the interpolant
result = interpolant(5)  # Returns 0.5