Because the Moshi model is stateful (it maintains conversation context), QuiLLMan ensures user isolation by spinning up a unique GPU per concurrent user session. This is achieved using Modal's @app.cls configuration.
To handle the continuous stream of audio data, the application uses FastAPI's bidirectional websockets. A FastAPI app is attached to a Modal class method using @modal.asgi_app(), allowing a prewarmed Moshi model to be coupled directly to a websocket session. This enables asynchronous loops to simultaneously receive audio bytes from the client and send inference output back to the user.
@app.cls(
image=image,
gpu="A10G",
scaledown_window=300,
...
)
class Moshi:
# ...
@modal.asgi_app()
def web(self):
from fastapi import FastAPI, Response, WebSocket, WebSocketDisconnect
web_app = FastAPI()
@web_app.websocket("/ws")
async def websocket(ws: WebSocket):
with torch.no_grad():
await ws.accept()
# handle user session
# spawn loops for async IO
async def recv_loop():
while True:
data = await ws.receive_bytes()
# send data into inference stream...
async def send_loop():
while True:
await asyncio.sleep(0.001)
msg = self.opus_stream_outbound.read_bytes()
# send inference output to user ...
