DOCSEARCHbyAlgolia
Submit Docs

amica Documentation

repository·master·Indexed 23 days ago

https://github.com/semperai/amica

A customizable personal AI platform for interacting with 3D VRM characters via natural voice chat and vision. Features include an emotion engine, support for multiple chat backends (Llama.cpp, ChatGPT, Ollama, OpenRouter, KoboldCpp), and integration with three.js, Transformers.js, and Whisper. Includes a comprehensive External API for managing chat, memory, and multimedia processing, as well as a Tauri-based desktop application.

Tokens
21.2K
Snippets
47
Records
159
Agent score
81%

What's inside amica

  1. Overview of supported technologies in Amica

    master

    Amica integrates various technologies to provide 3D character interaction, voice, and vision capabilities:

    • 3D Rendering: three.js and @pixiv/three-vrm for displaying VRM characters.
    • Browser AI: Transformers.js for running models in the browser.
    • Speech & Audio: Whisper for recognition, Silero VAD for voice activity detection, and various TTS options (Eleven Labs, OpenAI, Speech T5, etc.).
    • Chat Backends: Supports Llama.cpp, ChatGPT API, Ollama, OpenRouter, KoboldCpp, and more.
    • Vision: Bakllava.

  2. Overview of Amica components

    master

    Amica's architecture is divided into six primary functional systems:

    SystemResponsibility
    Chat SystemThe core orchestrator. Processes messages, generates responses, and manages other systems.
    Voice SystemText-to-speech (TTS) engine. Handles speech generation using specific voices and emotional intonation.
    Avatar SystemThe visual representation. Manages avatar display, expression changes, and lip-syncing.
    Transcription SystemSpeech-to-text (STT) engine. Includes Voice Activity Detection (VAD) to detect when a user starts and stops speaking.
    Expression SystemControls the avatar's facial expressions by manipulating blendshapes based on detected emotions.
    Visual SystemComputer vision module. Uses the device camera to detect faces, emotions, objects, and text.

  3. Explore Amica use cases

    master

    Amica is a platform designed for creating lifelike avatars for interactive experiences. Developers can leverage the platform across various domains including:

    • Virtual Assistants & Customer Support: Conversational agents for task assistance and personalized chat interfaces.
    • Education & Training: Interactive e-learning environments, language learning practice, and realistic simulations (e.g., emergency response or job training).
    • Healthcare: Telehealth assistants for medication reminders, information, and emotional support.
    • Entertainment & Gaming: Immersive gaming experiences featuring dynamic facial expressions and gestures.
    • Retail: Virtual shopping guides, product recommendations, and virtual try-on experiences.
    • Accessibility: Tools for interpreting sign language or providing visual cues for hearing/speech impairments.
    • Social & Communication: Avatars for virtual meetings, social media, and event hosting.
    • Specialized Education: Bringing historical figures to life or providing guided tours in virtual museums.
    • Well-being: Personal productivity companions and emotional support avatars.

  4. Key features and advantages of Amica

    master

    Amica is designed to streamline the building, deploying, developing, and testing of AI avatars. Key developer-centric features include:

    • Code Syncing: Enables live reloading during the development process.
    • Live Debug Log Streaming: Provides real-time visibility into logs for easier debugging.
    • Web Interface: An intuitive interface for managing and interacting with avatars.
    • High-Speed Speech Generation: Engineered for near-instantaneous speech synthesis to ensure fluid and natural real-time interactions.
    • Automation: Replaces manual pipelines or shell scripts with an integrated workflow.

  5. Amica Vision Capability capabilities

    master

    The Vision Capability allows the avatar to interpret visual data using:

    • Image Recognition: Integration of computer vision algorithms to respond to visual stimuli.
    • Object Recognition: Identification of objects within the environment to enable immersive interactions.

  6. Manage Memory with `Brain Message` and `Memory Request`

    master

    Amica allows you to manage 'subconscious' memory through two specific input types:

    1. Memory Request: Fetches existing memory data.
    2. Brain Message: Adds new memory data to the subconscious storage.

    Add Memory (Brain Message)

    {
  "inputType": "Brain Message",
  "payload": {
    "prompt": "Stored memory prompt example 2",
    "timestamp": "2024-12-30T12:00:00Z"
  }
}

    Fetch Memory (Memory Request)

    {
  "inputType": "Memory Request"
}

  8. How the Amica Emotion System works

    master
    The Amica emotion system enables realistic facial expressions on a VRM model by parsing the text stream from a Large Language Model (LLM). The system identifies specific emotion tags within the LLM's response and maps those tags to the corresponding blendshapes (morph targets) on the VRM model's face. To trigger these expressions, the LLM's system prompt must be designed to instruct the model to include these tags in its output.

  9. Understanding Amica's role relative to LLM inference software

    master
    Amica is not a replacement for LLM (Large Language Model) inference software. Instead, it is designed to complement existing LLM tools by providing a human-usable interface. It acts as a layer that allows users to interact with 3D AI avatars in a user-friendly manner, bridging the gap between raw AI model capabilities and intuitive human interaction.

  10. Amica Transcription Module capabilities

    master

    The Transcription Module handles speech-to-text conversion. It supports:

    • Accurate Speech-to-Text Conversion: High-accuracy conversion of spoken words to text.
    • Multilingual Support: Recognition and transcription across multiple languages.
    • Real-time Transcription: Capability for real-time processing to support dynamic, live interactions.

  12. Amica Text-to-Speech (TTS) Module capabilities

    master

    The Text-to-Speech (TTS) Module manages how the avatar speaks. Key features include:

    • Natural Language Processing: Generation of human-like, expressive speech patterns.
    • Customizable Voice Profiles: Ability to select and personalize various voices and speech characteristics.
    • Emotional Intonation: Support for conveying emotions through variations in tone, pitch, and pacing.