Neuracore Robot Learning Library

When deploying a model to a new robot with a different hardware configuration (e.g., different number of joints or cameras), you must use an EmbodimentDescription to semantically align the new hardware to the model's learned structure.

If the new robot has fewer components than the training data, you must leave the missing indices empty (effectively zero-padding) so that the components that do exist match their trained indices.

Example: Aligning an SO-100 (5 joints) to a model trained on Franka (7 joints) and UR5 (6 joints)

Important: RGB inputs must follow the same positional convention. If the model was trained with [wrist_camera, overhead_camera], an overhead camera must be placed at index 1, even if no wrist camera is present.

so100_embodiment_description: EmbodimentDescription = {
    JOINT_POSITIONS: {
        0: "joint_1",
        1: "joint_2",
        2: "joint_3",
        3: "joint_4",
        6: "wrist",
    },
    RGB_IMAGES: {
        1: "so100_overhead_camera",
    },
}

The daemon supports two video codecs for RGB cameras. This setting is global and applies to every camera. The daemon re-reads the codec at the start of each recording, so changes take effect for the next recording without a daemon restart.

• h264_medium (Default): Lossy-only. Uploads a single full-resolution video (lossy.mp4). This is irreversible; if you record in this mode, you cannot later generate a lossless archive for that recording.
• h264_lossless: Uploads both a lossless archive (lossless.mp4) for training and a small lossy preview (lossy.mp4). This increases disk use, upload size, and CPU usage.

Note: Depth cameras always maintain lossless storage regardless of this setting.

import neuracore as nc

# Set to lossy-only (default)
nc.set_video_encoding_options(codec=nc.Codec.H264_MEDIUM)

# Set to lossless (uploads both lossy and lossless files)
nc.set_video_encoding_options(codec=nc.Codec.H264_LOSSLESS)

The Rust data daemon is organized as a Cargo workspace located in the rust/ directory. It consists of three primary crates:

• data-daemon: The main daemon binary. It handles the CLI, lifecycle management, SQLite state, IPC listening, per-trace pipelines, and encoding.
• data_daemon_shared: A shared library containing IPC envelope types, service-name constants, the daemon configuration model, and filesystem-path resolution logic. This is linked by both the daemon and the producer.
• data_daemon_bridge: A PyO3 cdylib that acts as the producer-side IPC client. It is exposed to Python as neuracore.data_daemon._data_bridge.

| Crate | Path | What it is |
|---|---|---|
| `data-daemon` | [rust/data_daemon/](../rust/data_daemon/) | The daemon binary — CLI, lifecycle, SQLite state, IPC listener, per-trace pipeline, encoding. |
| `data_daemon_shared` | [rust/data_daemon_shared/](../rust/data_daemon_shared/) | Shared library — IPC envelope types and service-name constants, plus the daemon configuration model and filesystem-path resolution the two processes must compute identically. Linked by both the daemon and the producer crate. |
| `data_daemon_bridge` | [rust/data_daemon_bridge/](../rust/data_daemon_bridge/) | PyO3 `cdylib` — producer-side IPC client exposed to Python as `neuracore.data_daemon._data_bridge`. |

If your dataset contains end-effector poses but you need joint positions, set joint_position_input_type: END_EFFECTOR in the JOINT_POSITIONS format. This requires a valid URDF/MJCF and an end-effector frame name defined in the mapping.

Requirements:

1. A valid robot model (URDF/MJCF).
2. An end-effector frame name (provided via mapping.name).
3. Optional ik_init_config (a list of floats) to provide an initial joint configuration for the IK solver.

# End-effector pose to convert from
END_EFFECTOR_POSES:
    source: observation.state
    mapping:
      - name: ee_name
        index_range:
          start: 0
          end: 7

# Joint position calculated using Inverse Kinematics
JOINT_POSITIONS:
  source: observation.state
  format:
    joint_position_input_type: END_EFFECTOR
    ik_init_config: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
  mapping:
    - name: ee_frame_name
      source_name: ee_name

In Neuracore, model shape is spec-driven, not robot-driven. Instead of hard-coding tensor dimensions based on a specific robot, you define a declarative specification that maps data points (like joints or camera feeds) into a shared index space.

When launching a training job, you must provide an Input Cross-Embodiment Description and an Output Cross-Embodiment Description. These specifications determine the fixed input and output dimensionality for the model. This approach allows a single model to train across heterogeneous robots by ensuring that every dimension in the tensor has a consistent semantic meaning (e.g., index 2 always represents 'elbow bend' regardless of the robot's specific joint names).

# The model shape is determined by these two specs:
input_cross_embodiment_description: CrossEmbodimentDescription = { ... }
output_cross_embodiment_description: CrossEmbodimentDescription = { ... }

You can specify joint target positions as delta values relative to the current robot state. The importer will add these deltas to the current state to form the final target.

Modes:

• Joint Space: If action_space: JOINT, deltas are applied to current joint positions.
• End-Effector Space: If action_space: END_EFFECTOR, deltas are applied to the current end-effector pose. This requires importing end-effector poses.

Configuration:

• action_type: RELATIVE must be set.
• action_space must be set to either JOINT or END_EFFECTOR.

# Example 1: Relative action in pose space
JOINT_TARGET_POSITIONS:
  source: action
  format:
    action_type: RELATIVE
    action_space: END_EFFECTOR
    pose_type: POSITION_ORIENTATION
    orientation:
      type: EULER
      euler_order: XYZ
      angle_units: RADIANS
  mapping:
    - name: ee_frame_name
      index_range:
        start: 7
        end: 13

# Example 2: Relative action in joint space
JOINT_TARGET_POSITIONS:
  source: action
  format:
    action_type: RELATIVE
    action_space: JOINT
  mapping:
    - name: joint_1
    - name: joint_2
    - name: joint_3
    - name: joint_4
    - name: joint_5
    - name: joint_6
    - name: joint_7

Neuracore uses a linear memory model to predict the maximum batch size. The core idea is that peak GPU memory follows the formula:

peak(b) ≈ intercept + slope · b

Where:

• intercept: Fixed memory costs (model weights, optimizer state, CUDA context).
• slope: Extra memory required per additional sample (activations).

The Algorithm

1. Probe min_batch_size: If this OOMs, the model is too large for the GPU.
2. Probe high anchor: Usually min_batch_size * 8. If this OOMs, the algorithm shrinks toward the minimum.
3. Place a third anchor: A larger batch size calculated to find a reliable slope.
4. Fit the slope: The algorithm calculates the slope specifically between the 2nd and 3rd probes. It avoids using the slope from the first two tiny probes because they are dominated by the intercept (fixed costs) and provide an unreliable, underestimated slope.
5. Predict and Confirm: It predicts the b_max that hits the max_gpu_utilization budget and runs a probe to confirm. If it OOMs, it shrinks by _DOWNSCALE_FACTOR (0.9) until it fits.
6. Apply Safety Margin: The final chosen batch is floor(largest_fitting_batch * safety_factor).