VoiceRAG Reference Implementation

VoiceRAG is an application pattern for implementing Retrieval Augmented Generation (RAG) with a voice user interface. It uses the GPT-4o Realtime API for audio to process voice input and generate audio responses, while leveraging Azure AI Search to retrieve relevant documents from a knowledge base to ground the model's responses.

Key features include:

• Voice interface: Captures browser microphone input for processing by Azure OpenAI.
• RAG: Uses Azure AI Search to provide context to the GPT-4o Realtime API.
• Audio output: Plays generated responses via browser audio capabilities.
• Citations: Displays the search results used to generate the response.

The VoiceRAG pattern follows a specific flow between the frontend and backend:

1. The frontend RTClient receives audio input from the user's microphone.
2. The audio is sent to a Python backend.
3. The backend uses an RTMiddleTier object to interface with the Azure OpenAI real-time API.
4. The RTMiddleTier includes a tool for searching Azure AI Search to perform RAG.

This architecture can be deployed to Azure Container Apps using the provided Infrastructure as Code (IaC) and Dockerfile, or run locally if Azure AI Search and Azure OpenAI services are configured.

Cost Management

Resources created by this project incur costs. To avoid unnecessary spending:

• Use azd down to tear down resources when not in use.
• Consider switching to free SKUs for services where possible (though this introduces limitations).
• Monitor usage for Azure OpenAI (per 1K tokens), Azure AI Search (per hour), and Azure Container Apps (Consumption plan).

Security

• The template uses Managed Identity to eliminate the need for managing credentials in deployed environments.
• For local development, you can use Entra ID by omitting API keys in the .env file.
• It is recommended to enable GitHub secret scanning in your repositories to prevent credential leaks.

You can use an existing Azure AI Search index directly.

• If the index was created using the Import and vectorize data option in the Azure Portal, no code changes are required.
• If the index was created manually, you must update the field names in app/backend/ragtools.py to match your specific text and vector field names.

To sync your azd environment changes to your local .env file (for example, to test a new voice choice with the local development server), run the appropriate script for your shell:

• Bash: ./scripts/write_env.sh
• PowerShell: pwsh ./scripts/write_env.ps1

After running the script, restart your development server to pick up the changes.

You can modify the Azure OpenAI deployment settings using azd env set. After setting these values, you must run azd up to apply the changes to the Azure resources.

Deployment Version

Set AZURE_OPENAI_REALTIME_DEPLOYMENT_VERSION to specify a different API version (e.g., 2024-12-17).

Deployment Capacity

Set AZURE_OPENAI_REALTIME_DEPLOYMENT_CAPACITY to specify the capacity. The default is 1.

# Set deployment version
azd env set AZURE_OPENAI_REALTIME_DEPLOYMENT_VERSION 2024-12-17

# Set deployment capacity
azd env set AZURE_OPENAI_REALTIME_DEPLOYMENT_CAPACITY 2

# Apply changes
azd up

To test VoiceRAG locally using an existing azure-search-openai-demo index, create a .env file in the app/backend directory. This file allows the local development server to connect to your existing Azure services without using azd environment variables.

# Create app/backend/.env with the following structure:
AZURE_TENANT_ID=<YOUR-TENANT-ID>
AZURE_OPENAI_ENDPOINT=https://<YOUR_OPENAI_ENDPOINT>.openai.azure.com
AZURE_OPENAI_REALTIME_DEPLOYMENT=gpt-4o-realtime-preview
AZURE_OPENAI_REALTIME_VOICE_CHOICE=<choose one: echo, alloy, shimmer>
AZURE_SEARCH_ENDPOINT=https://<YOUR_SEARCH_SERVICE>.search.windows.net
AZURE_SEARCH_INDEX=gptkbindex
AZURE_SEARCH_SEMANTIC_CONFIGURATION=default
AZURE_SEARCH_IDENTIFIER_FIELD=id
AZURE_SEARCH_CONTENT_FIELD=content
AZURE_SEARCH_TITLE_FIELD=sourcepage
AZURE_SEARCH_EMBEDDING_FIELD=embedding

To run the VoiceRAG application, you must provision the following Azure services:

1. Azure OpenAI: Requires two model deployments:
   • One deployment of the gpt-4o-realtime-preview model.
   • One deployment for embeddings (e.g., text-embedding-3-large, text-embedding-3-small, or text-embedding-ada-002).
2. Azure AI Search: Any tier from Basic or above is required. It is recommended to have Semantic Search enabled.
3. Azure Blob Storage: A container containing the content that serves as your knowledge base (e.g., PDFs, Office documents).

You can change the voice used by the real-time deployment by setting the AZURE_OPENAI_REALTIME_VOICE_CHOICE environment variable.

Available options are:

• alloy (default)
• echo
• shimmer

To apply changes to the deployed app, run azd up after setting the variable.

If you want to preview the voice locally using the development server before deploying, update your local .env file by running the provided script, then restart the development server.

azd env set AZURE_OPENAI_REALTIME_VOICE_CHOICE <echo, alloy, or shimmer>