repository·main·Indexed 27 days ago
https://github.com/nvidia/personaplexA real-time, full-duplex speech-to-speech conversational model built on the Moshi architecture. PersonaPlex enables precise persona control through text-based role prompts and audio-based voice conditioning. The documentation covers server deployment via Docker Compose, offline evaluation using the moshi.offline module, and the implementation of the moshi-client frontend, including WebSocket protocol types and model parameter configuration.
PersonaPlex requires the Opus audio codec development library to be installed on your system.
Ubuntu/Debian:
sudo apt install libopus-devFedora/RHEL:
sudo dnf install opus-devel# Ubuntu/Debian
sudo apt install libopus-dev
# Fedora/RHEL
sudo dnf install opus-develDownload the repository and install the package using pip.
Note for Blackwell GPUs: If you are using Blackwell-based GPUs, you must install specific PyTorch versions using the following command:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130pip install moshi/.The moshi-client is the frontend for the PersonaPlex demo. To run it locally, follow these steps:
nvm use to switch to the project's recommended version.cert.pem and key.pem) and copy them to the root of the moshi-client package.env.local file in the package root and define the VITE_QUEUE_API_PATH variable (defaults to /api).npm install before the first run or after updating dependencies.npm run dev to start the development server.npm run build to create a production build.Launch the server for live interaction. The server uses temporary SSL certificates for HTTPS. Once running, you can access the Web UI at localhost:8998 (if running locally) or via the URL printed in the terminal.
CPU Offload: If your GPU has insufficient memory, use the --cpu-offload flag. This requires the accelerate package (pip install accelerate).
SSL_DIR=$(mktemp -d); python -m moshi.server --ssl "$SSL_DIR"
# With CPU offload
SSL_DIR=$(mktemp -d); python -m moshi.server --ssl "$SSL_DIR" --cpu-offloadBefore using PersonaPlex, you must accept the model license on Huggingface (link) and then export your Huggingface token as an environment variable.
export HF_TOKEN=<YOUR_HUGGINGFACE_TOKEN>export HF_TOKEN=<YOUR_HUGGINGFACE_TOKEN>PersonaPlex uses text-based role prompts to control persona. The model is trained on several categories:
Use this prompt for a wise, friendly teacher persona (ideal for QA/User Interruption evaluation):
You are a wise and friendly teacher. Answer questions or provide advice in a clear and engaging way.
Define specific identities and information. Example:
You work for CitySan Services which is a waste management and your name is Ayelen Lucero. Information: Verify customer name Omar Torres. Current schedule: every other week. Upcoming pickup: April 12th. Compost bin service available for $8/month add-on.
Use prompts to trigger open-ended discussions. For standard evaluation (Pause Handling, Backchannel, etc.), use:
You enjoy having a good conversation.
To test emergent abilities, you can provide highly specific, out-of-distribution prompts (e.g., an astronaut discussing a reactor meltdown).
To use the client in standalone mode and bypass the queue, append the worker_addr query parameter to the application URL. Replace {WORKER_ADDR} with the address of your specific worker instance.
Example URL format:
https://localhost:5173/?worker_addr=0.0.0.0:8088
https://localhost:5173/?worker_addr=0.0.0.0:8088The offline inference script requires voice prompt files. You can provide them in two ways:
--voice-prompt-dir argument, the script will download voices.tgz from the specified --hf-repo and extract it to a voices/ directory.--voice-prompt-dir. You must then specify the specific file basename using the --voice-prompt argument.Example using a local directory:
--voice-prompt-dir /path/to/my/voices --voice-prompt MY_VOICE.ptYou can deploy the PersonaPlex service using Docker Compose. The configuration requires an NVIDIA GPU and maps the service to port 8998. It also uses a .env file for environment variables and mounts a local ./.cache directory to /root/.cache inside the container to persist cache files.
Key Configuration Details:
8998:8998nvidia driver with gpu capabilities. One GPU is reserved by default..env file. The NO_TORCH_COMPILE=1 flag is set by default to disable torch compilation../.cache is mapped to /root/.cache.services:
personaplex:
build:
context: .
dockerfile: Dockerfile
ports:
- "8998:8998"
environment:
- NO_TORCH_COMPILE=1
env_file:
- .env
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [ gpu ]
volumes:
- ./.cache:/root/.cacheUse the moshi.offline module to stream an input .wav file and produce an output .wav file. The output duration matches the input duration.
Assistant Example:
Uses a voice prompt (e.g., NATF2.pt) to evaluate the assistant role.
Service Example: Uses both a voice prompt and a text prompt to evaluate specific service roles.
Note: Add --cpu-offload if GPU memory is insufficient (requires accelerate).
# Assistant example
HF_TOKEN=<TOKEN> \
python -m moshi.offline \
--voice-prompt "NATF2.pt" \
--input-wav "assets/test/input_assistant.wav" \
--seed 42424242 \
--output-wav "output.wav" \
--output-text "output.json"
# Service example
HF_TOKEN=<TOKEN> \
python -m moshi.offline \
--voice-prompt "NATM1.pt" \
--text-prompt "$(cat assets/test/prompt_service.txt)" \
--input-wav "assets/test/input_service.wav" \
--seed 42424242 \
--output-wav "output.wav" \
--output-text "output.json"PersonaPlex supports pre-packaged embeddings for Natural (NAT) and Variety (VAR) voices. Use these labels for voice conditioning.
| Category | Female | Male |
|---|---|---|
| Natural | NATF0, NATF1, NATF2, NATF3 | NATM0, NATM1, NATM2, NATM3 |
| Variety | VARF0, VARF1, VARF2, VARF3, VARF4 | VARM0, VARM1, VARM2, VARM3, VARM4 |
getAPIClient to create a client instance for interacting with the PersonaPlex service. You must provide the base url of the running server. The returned client provides methods to manage users, check session status, and submit feedback.