DOCSEARCHbyAlgolia
Submit Docs

LiveKit Agent Starter Python

repository·main·Indexed 42 days ago

https://github.com/livekit-examples/agent-starter-python

A starter project for building voice AI applications using Python and LiveKit Agents. It provides a pre-configured pipeline featuring LLMs, TTS, turn detection, and expressive mode, with support for deployment to LiveKit Cloud or self-hosting. Includes guides for using the LiveKit CLI, integrating Realtime Models like OpenAI Realtime API, and adding custom tools via @function_tool.

Tokens
2.1K
Snippets
9
Records
11
Agent score
4%

What's inside LiveKit Agent Starter Python

  1. Install the LiveKit CLI

    main

    The LiveKit CLI (lk) is used for authentication, initializing projects from templates, and running simulations. Install it based on your platform:

    • macOS: Use Homebrew.
    • Linux: Use the curl installer.
    • Windows: Use winget.

    Note: The lk docs subcommand requires version 2.15.0 or higher. Check your version with lk --version.

    # macOS
brew install livekit-cli

# Linux
curl -sSL https://get.livekit.io/cli | bash

# Windows
winget install LiveKit.LiveKitCLI

  2. Manual setup without the CLI

    main

    If you prefer not to use the CLI for initialization, follow these steps:

    1. Clone the repository.
    2. Install dependencies using uv.
    3. Create a .env.local file by copying .env.example.
    4. Fill in the following required environment variables:
      • LIVEKIT_URL
      • LIVEKIT_API_KEY
      • LIVEKIT_API_SECRET

    You can also use the CLI to automatically write these credentials to your .env.local file after running lk cloud auth.

    # Install dependencies
cd agent-starter-python
uv sync

# Automatically write LiveKit credentials to .env.local
lk cloud auth
lk app env --write --destination .env.local

  3. Run the voice AI agent

    main

    Depending on your use case, use one of the following commands to run the agent via uv:

    • Console Mode: Speak to your agent directly in your terminal.
    • Dev Mode: Run the agent for use with a frontend or telephony.
    • Production Mode: Run the agent in a production environment.
    # Speak to the agent in the terminal
uv run python src/agent.py console

# Run for frontend/telephony development
uv run python src/agent.py dev

# Run in production
uv run python src/agent.py start

  4. Run agent simulations and evaluations

    main

    Simulations run multi-turn conversations between a simulated user and your agent on LiveKit Cloud, judging the transcripts based on scenarios defined in scenarios.yaml.

    To run these scenarios locally, use the LiveKit CLI:

    For turn-level checks that do not require a live session, you can use the LiveKit Agents testing & evaluation framework with pytest. An example is provided in tests/test_agent.py.

    lk agent simulate --scenarios scenarios.yaml

  5. Initialize a new agent project from template

    main

    The recommended way to start a new project is using the LiveKit CLI. This clones the agent-starter-python template and configures your environment automatically.

    First, authenticate with LiveKit Cloud, then run the agent init command.

    lk cloud auth
lk agent init my-agent --template agent-starter-python

  6. How to add custom tools to your Assistant

    main

    You can extend your agent's capabilities by defining functions decorated with @function_tool. These tools allow the LLM to perform actions like looking up weather or interacting with external APIs.

    To use tools, you must:

    1. Import function_tool and RunContext from livekit.agents.
    2. Define an async method within your Assistant class (or a compatible class).
    3. Use a docstring to describe the tool, as the LLM uses this description to understand when and how to call it.
    4. Define arguments with type hints so the LLM knows what data to provide.
    from livekit.agents import function_tool, RunContext

# Inside your Assistant class:
@function_tool
async def lookup_weather(self, context: RunContext, location: str):
    """Use this tool to look up current weather information in the given location.

    If the location is not supported by the weather service, the tool will indicate this. You must tell the user the location's weather is unavailable.

    Args:
        location: The location to look up weather information for (e.g. city name)
    """
    logger.info(f"Looking up weather for {location}")
    return "sunny with a temperature of 70 degrees."

  7. How to use a Realtime Model instead of a voice pipeline

    main

    To use a realtime model (like OpenAI Realtime API) instead of a standard STT/TTS pipeline, you must replace the llm argument in the Assistant class with a RealtimeModel and remove the stt and tts arguments from the AgentSession.

    Steps to switch to OpenAI Realtime:

    1. Install the plugin: pip install livekit-agents[openai]
    2. Set OPENAI_API_KEY in your .env.local file.
    3. Import the plugin: from livekit.plugins import openai
    4. Update the Assistant initialization: llm=openai.realtime.RealtimeModel(voice="marin")
    # 1. Install livekit-agents[openai]
# 2. Set OPENAI_API_KEY in .env.local
# 3. Add `from livekit.plugins import openai` to the top of this file
# 4. Replace the llm argument with:
llm=openai.realtime.RealtimeModel(voice="marin")

  10. Configure AgentSession for voice interaction

    main

    The AgentSession manages the orchestration of the voice pipeline. Key configuration options include:

    • stt: Speech-to-text provider (the agent's 'ears').
    • tts: Text-to-speech provider (the agent's 'voice').
    • turn_handling: Controls how the agent detects when a user has finished speaking and how it handles interruptions.
      • turn_detection: Uses a TurnDetector to analyze acoustic and semantic cues.
      • interruption: Set mode="adaptive" to distinguish between real interruptions and backchannels (like "mhm").
      • preemptive_generation: Set enabled=True to allow the LLM to start generating a response while waiting for the end of a turn.
    • expressive: When set to True, the LLM can emit inline delivery tags (emotion, pacing) that supported TTS models (like Fish Audio) will render.
    session = AgentSession(
    stt=inference.STT(model="assemblyai/universal-3-5-pro", language="en"),
    tts=inference.TTS(model="fishaudio/s2.1-pro", voice="fa4c9eb3dccc4806b382b40d61c6b10a"),
    turn_handling=TurnHandlingOptions(
        turn_detection=inference.TurnDetector(),
        interruption={"mode": "adaptive"},
        preemptive_generation={"enabled": True},
    ),
    expressive=True,
)