DOCSEARCHbyAlgolia
Submit Docs

Transformers.js Documentation

repository·master·Indexed 23 days ago

https://github.com/butzyung/systemanimatoronline

A JavaScript library for running state-of-the-art machine learning models in the browser or Node.js using ONNX Runtime. It supports NLP, Computer Vision, Audio, and Multimodal tasks via a pipeline API, providing a functional equivalent to the Python Transformers library. Includes support for WebGPU acceleration, quantization (fp32, fp16, q8, q4), and chat template rendering via @huggingface/jinja.

Tokens
18.2K
Snippets
25
Records
113
Agent score
83%

What's inside Transformers.js

  1. Overview of KARA STEP MMD Motion Files

    master

    The MMD/motion/demo/STEP/ directory contains sample MMD (MikuMikuDance) motion files for the KARA STEP (Short_3ver.) sequence. These files are provided as demonstrations of motion and lip-sync data for MMD-compatible models.

    Included Motion Files:

    • 01_step-motion1.vmd: Initial motion sequence.
    • 02_step-motion2.vmd: Second motion sequence.
    • 03_step-motion3.vmd: Third motion sequence.
    • 04_step-lip1_1.vmd through 06_step-lip1_3.vmd: Lip-sync motion segments (Part 1).
    • 07_step-lip2.vmd: Lip-sync motion segment (Part 2).

  2. Overview of System Animator Online technical preview

    master

    System Animator Online is a technical preview of a 3D game system based on derivative works of characters from the "PassionTeens" online comics. It is a web-based system designed for personal and non-commercial use.

    Key Features:

    • 3D Model Interaction: Users can freely rotate 3D models using a mouse.
    • Viewport Control: A button in the upper right corner allows users to resize/zoom the viewport.
    • Automatic Dancing: Users can upload MP3, AAC, or WAV music files using buttons located in the lower or upper left corners. The 3D characters will automatically dance in synchronization with the music beats.

  3. Overview of XR Animator

    master

    XR Animator is a web-based application for full-body, real-time motion tracking using a single webcam. It leverages MediaPipe and TensorFlow.js to detect 3D poses from live video to drive 3D avatars (MMD or VRM models). It is suitable for XR purposes and VTubing.

    Key Capabilities:

    • Motion Tracking: Supports face, full body, or combinations of face/body/hands.
    • Avatar Support: Works with any MMD or VRM model.
    • Motion Files: Supports loading VMD, BVH, and FBX formats; can export BVH/FBX to VMD.
    • Recording: Allows recording mocap motion and exporting to VMD format.
    • Environment Customization: Supports 2D images/videos, 3D panorama, 3D objects (.x/.glb), or green screen backgrounds.
    • Platform Support: Works on major web browsers (desktop and smartphone).

  4. Overview of the Hokusai (FGO) Model

    master

    This is a 3D model of Katsushika Hokusai from Fate/Grand Order, created by the modeller Yatsurugi (八剣).

    Key Details:

    • Model Name: 八剣式 葛飾北斎 Ver.1.01
    • Modeller: 八剣 (Yatsurugi)
    • Release Date: 2018/01/20
    • Intended Software: Specifically designed for use with Miku Miku Dance (MMD). Behavior in other software may vary.
    • Design Note: The model includes interpretations of parts not shown in official media (such as the back) based on the creator's discretion.

  5. Key Features of XR Animator

    master

    XR Animator provides full-body, real-time AI motion tracking using a single webcam. Key capabilities include:

    • Motion Tracking: Supports face, full body, or combinations of face/body/hands. Includes "Perfect Sync" (ARKit-compatible 52 blendshapes) for realistic facial expressions.
    • Avatar Support: Use any MMD or VRM model as your 3D avatar.
    • Motion Data:
      • Record mocap and export to VMD, BVH, or glTF formats.
      • Load VMD, FBX, or BVH 3D motions.
      • Convert FBX/BVH motions to VMD.
    • Environment Customization: Customize backgrounds with 2D images/videos, 3D panoramas, or 3D objects (.x/.glb).
    • Advanced Modes (Electron/Desktop only):
      • VMC-protocol: Animate models in external VMC-enabled applications.
      • Transparent Background: Use a frameless window for OBS streaming.
      • 3D Wallpaper: Run as a Windows background gadget.
    • Mobile AR: Supports Augmented Reality on Android Chrome via WebXR.

  6. How multiplayer works in System Animator

    master

    Multiplayer functionality is implemented using a Peer-to-Peer (P2P) network via WebRTC and PeerJS. All in-game communications occur directly between peers, ensuring that actual game data is not passed through the server (which is only used for signaling via the PeerServer Cloud service).

    Currently, the demo supports up to 3 players, including the host. Note that the multiplayer feature is in beta and primarily supports movement/walking around.

  7. Multiplayer usage constraints and rules

    master

    When using the multiplayer features, keep the following constraints in mind:

    • Switching Roles: Once you start hosting a game, you cannot join games hosted by others without restarting the application. Similarly, once you have successfully joined a game, you cannot host your own game or join a different game without restarting.
    • Starting without a game: If you press START without joining an existing game, you can only host your own game.
    • Browser Support: The game is tested on Google Chrome and Firefox. Other browsers are not officially supported.

  8. Understand the XR Animator License

    master

    XR Animator is licensed under CC BY-NC-SA 4.0.

    • Adapting Source Code: If you adapt the source code to build your own software or service, the CC BY-NC-SA 4.0 license applies.
    • Third-party Assets: The license does not cover third-party assets included in the project, which may have their own incompatible licenses.
    • Generated Content: The license does not apply to content generated using XR Animator's functionality (e.g., video content created from the motion capture feature using your own assets). The project claims no rights over such content.

  10. Use the SpringShower motion data

    master

    The SpringShower motion data is a specific motion set designed for MMD (MikuMikuDance) compatible models.

    Important Usage Notes:

    • Model Compatibility: This motion is intended for use with MMD models. Using it with non-MMD models may result in incorrect or broken animations.
    • License/Terms: The motion data was released under specific terms (noting a 2010 copyright context). Users should ensure they are following the intended usage guidelines for MMD motion assets.
    • Installation/Deployment: The motion data is provided as a compressed file (likely .zip). It should be extracted and placed within the appropriate motion directory of your MMD-compatible animation system to be recognized.