DOCSEARCHbyAlgolia
Submit Docs

Kimodo Motion Generation Model

repository·main·Indexed 25 days ago

https://github.com/nv-tlabs/kimodo

Kimodo is a kinematic motion diffusion model for generating high-quality 3D human and robot motions from text prompts and kinematic constraints. It supports skeletons including SOMA, Unitree G1, and SMPL-X. The project includes a C++ motion trajectory correction library (motion_correction), a low-level Python API for model inference, a command-line interface (kimodo_gen), and an interactive web-based motion authoring demo.

Tokens
27.6K
Snippets
52
Records
212
Agent score
82%

What's inside kimodo

  2. Overview of Kimodo Python package structure

    main

    The kimodo package is organized into several functional modules:

    • model/: Contains the diffusion model architecture, including the kimodo_model wrapper, denoising architectures, and model loading utilities.
    • motion_rep/: Handles motion representations, conditioning (text/constraints), and feature extraction.
    • skeleton/: Manages skeleton definitions, kinematics, and BVH I/O.
    • viz/: Provides visualization tools, including 3D scene setup, playback, and specific rig support (e.g., G1 robot, SMPL-X).
    • demo/: Contains the logic for the interactive web demo (Gradio/Viser).
    • exports/: Provides utilities for motion I/O, format conversion (BVH, MuJoCo, SMPL-X), and loading/saving motion dictionaries.
    • metrics/: Includes evaluation metrics like foot skating and constraint satisfaction.
    • scripts/: Contains the CLI implementations and helper scripts.
    • constraints.py: Defines and handles motion constraints.
    • postprocess.py: Contains post-processing logic such as MotionCorrection.

  3. Overview of Kimodo

    main

    Kimodo is a kinematic motion diffusion model designed for scaling controllable human motion generation. It is trained on a large-scale (700 hours) commercially-friendly optical motion capture dataset.

    Key Capabilities:

    • Controlled Generation: Generates high-quality 3D human and robot motions using text prompts combined with constraints like full-body pose keyframes, end-effector positions/rotations, 2D paths, and 2D waypoints.
    • Human(oid) Support: Provides model variations for both digital humans and humanoid robots.
    • Interactive Authoring: Supports timeline editing and real-time 3D visualization via an interactive demo.

  4. Use the kimodo.exports package for motion data export

    main

    The kimodo.exports package provides submodules for exporting and converting motion data across various formats and simulators. Use the specific submodules based on your target format or environment:

    • kimodo.exports.bvh: For BVH (Biovision Hierarchy) motion format.
    • kimodo.exports.motion_convert_lib: Library for motion conversion tasks.
    • kimodo.exports.motion_formats: Definitions and handling for various motion formats.
    • kimodo.exports.motion_io: Input/Output operations for motion data.
    • kimodo.exports.mujoco: Integration and export for MuJoCo physics simulations.
    • kimodo.exports.smplx: Support for SMPL-X human body models.

  6. Key features of the Kimodo Interactive Demo

    main

    The interactive demo provides a timeline-based interface for motion composition with real-time 3D visualization. Key capabilities include:

    • Multiple Characters: Support for SOMA, G1, and SMPL-X models.
    • Text Prompts: Natural language descriptions of motions entered on a timeline.
    • Timeline Editor: Management of keyframes and constrained intervals across multiple tracks.
    • Constraint Types:
      • Full-Body: Complete joint position constraints at specific frames.
      • 2D Root: Waypoints or paths on the ground plane.
      • End-Effectors: Control of hand and foot positions/rotations.
    • Constraint Editing: Re-posing constraints or adjusting waypoints in editing mode.
    • 3D Visualization: Real-time rendering with skeleton and skinned mesh options.
    • Playback Controls: Preview motions with adjustable playback speed.
    • Multiple Samples: Generation and comparison of motion variations.
    • Examples: Loading pre-existing motion examples.
    • Export: Saving constraints and generated motions.

  7. Explore the kimodo.viz package submodules

    main

    The kimodo.viz package provides visualization tools for the Kimodo project. It is organized into several specialized submodules for handling different aspects of motion visualization, including UI components, coordinate systems, robot rigs, and various skinning models.

    Key submodules include:

    • constraint_ui: UI components for managing constraints.
    • coords: Coordinate system utilities.
    • g1_rig: Rigging for the G1 robot.
    • gui: General graphical user interface components.
    • playback: Tools for playing back motion data.
    • scene: Scene management and setup.
    • smplx_skin, soma_skin, soma_layer_skin: Various skinning implementations (SMPLX and SOMA).
    • viser_utils: Utilities for the viser visualization framework.

  8. Supported Skeletons in Kimodo

    main

    Kimodo supports different skeleton (character) models, with a separate model trained for each. The available options are:

    • SOMA (Default): Based on the SOMA body model. Used for general motion generation.
    • Unitree G1: Targeted at MuJoCo-compatible exports and robotics workflows.
    • SMPL-X: Aligned with the SMPL-X model for AMASS-style exports. Note that this is not recommended due to potential severe retargeting artifacts.

    Skeleton definitions are located in kimodo/skeleton/definitions.py.

  11. Use Kimodo core modules

    main

    The kimodo package provides several functional modules for motion processing and geometry:

    • kimodo.assets: Asset management.
    • kimodo.constraints: Definition and application of motion constraints.
    • kimodo.geometry: Geometric primitives and operations.
    • kimodo.meta: Metadata handling.
    • kimodo.postprocess: Post-processing routines for motion data.
    • kimodo.sanitize: Tools for cleaning or validating motion data.
    • kimodo.tools: General utility functions.

  12. Understand the Kimodo Motion Representation

    main

    Kimodo represents motion by combining a smoothed root trajectory with global joint positions, rotations, and auxiliary features. This representation is designed to make path-following constraints more natural by removing high-frequency pelvis jitter while preserving overall motion direction.

    The implementation can be found in kimodo/motion_rep/reps/kimodo_motionrep.py, which provides utilities to convert between this feature representation and other formats.