DOCSEARCHbyAlgolia
Submit Docs

Volcengine RTC AIGC Demo

repository·main·Indexed 43 days ago

https://github.com/volcengine/rtc-aigc-demo

An interactive demo demonstrating end-to-end streaming voice capabilities by integrating Real-Time Communication (RTC), Automatic Speech Recognition (ASR), Large Language Models (LLM), and Speech Synthesis (TTS). The project includes a Koa-based backend server for session and scene management and a React frontend.

Tokens
11.5K
Snippets
42
Records
55
Agent score
55%

What's inside RTC AIGC Demo

  1. How to define and add custom AIGC scenes

    main

    The Node Server automatically discovers available scenes by reading .json files located in the server/scenes directory. To add a new scene:

    1. Create a new xxxx.json file in server/scenes/ (where xxxx is your scene name).
    2. Use default.json as a template to ensure your JSON structure is correct. Note that filenames are case-sensitive.
    3. Restart the server after modifying scenes. If using yarn --cwd server dev, the server will automatically restart on JS/JSON changes, but it will not restart when .env.local changes.

    Important Lifecycle Notes:

    • The server dynamically generates RTC RoomId, UserId, Token, and VoiceChat TaskId for each session. These are not written back to your JSON files.
    • AgentConfig.UserId must be provided within your scene JSON.
    • The server overrides VoiceChat fields such as AppId, RoomId, TaskId, and AgentConfig.TargetUserId, and enables EnableConversationStateCallback. You do not need to include these runtime fields in your JSON.
    // Reference server/scenes/default.json for the correct schema

  2. Configure AIGC Scenarios in JSON

    main

    Scenarios are defined in server/scenes/*.json. The filename (without .json) serves as the SceneID. You can create new scenarios by copying default.json.

    Each scenario file contains two main sections:

    1. SceneConfig: Metadata for the frontend, such as name and icon.
    2. VoiceChat: The core AIGC configuration. This includes:
      • Config: Parameters for the AIGC service.
      • AgentConfig: Configuration for the AI agent. VoiceChat.AgentConfig.UserId is mandatory as it serves as the agent's RTC user identity.

    Pro-tip: You can use the Volcengine Console to quickly generate parameters, then copy the code from the '接入 API' button into your JSON configuration.

    {
  "SceneConfig": {
    "name": "My AI Agent",
    "icon": "url_to_icon"
  },
  "VoiceChat": {
    "Config": { /* ... */ },
    "AgentConfig": {
      "UserId": "agent_user_id_here",
      "TargetUserId": "..."
    }
  }
}

  3. Install and run the Node Server

    main

    The Node Server must be executed from the project root directory. It reads configuration from the root .env.local file, not the server/.env.local file. Existing process environment variables take precedence over values in the file.

    To install dependencies and run the server in development mode (with file watching enabled):

    yarn --cwd server install --frozen-lockfile
yarn --cwd server dev

    To run the server in production mode (without file watching):

    yarn --cwd server start

    By default, the server listens on 127.0.0.1:3001. You can modify the host and port using the HOST and PORT environment variables in the root .env.local file.

  4. Run the demo in different modes

    main

    You can run the entire project at once or manage the components individually.

    Full Stack (Development): Runs both the server and the web frontend simultaneously.

    yarn dev

    Individual Components: Use these in separate terminal windows.

    # Start Server
yarn --cwd server dev

# Start Frontend
yarn --cwd web start

    Testing and Building:

    # Run tests
yarn --cwd server test
yarn --cwd web test --runInBand

# Build frontend for production (outputs to web/build/)
yarn --cwd web build

  5. Install and set up the AIGC Demo

    main

    To run the demo, you need Node.js 22 and Yarn 1.22.22. The project consists of a web/ frontend (CRA) and a server/ backend (Koa).

    Follow these steps in the project root:

    1. Copy environment templates: Create .env.local files for both the root and the web directory.
    2. Install dependencies: Install for the root, the web package, and the server package.
    3. Configure credentials: Fill in the required Volcengine and RTC credentials in the root .env.local.
    4. Configure scenes: Edit the JSON files in server/scenes/ to define your AIGC scenarios.
    # 1. Prepare configuration files
cp -n .env.example .env.local
cp -n web/.env.example web/.env.local

# 2. Install dependencies
yarn install --frozen-lockfile
yarn --cwd web install --frozen-lockfile
yarn --cwd server install --frozen-lockfile

# 3. Start the demo
yarn dev

  6. Understand the RequestResponse data structure

    main

    The RequestResponse interface defines the standard envelope for API responses in this project. Every response contains a Result field (the actual payload) and an optional ResponseMetadata field containing execution details or error information.

    export interface RequestResponse {
  ResponseMetadata: Partial<{
    Action: string;
    Version: string;
    Service: string;
    Region: string;
    RequestId: string;
    Error: {
      Code: string;
      Message: string;
    };
  }>;
  Result: any;
}

  7. Configure environment variables and credentials

    main

    Configuration is managed via the .env.local file located in the project root.

    Required Variables

    • RTC_APP_ID & RTC_APP_KEY: Server-side RTC credentials.
    • VOLCENGINE_ACCESS_KEY_ID & VOLCENGINE_SECRET_ACCESS_KEY: Must be configured as a pair for signing OpenAPI calls.

    Advanced Configuration

    • RTC_BUSINESS_ID: (Optional) Can be passed to both the RTC SDK and VoiceChat.
    • Third-party Secrets: Store these in .env.local. They can be referenced within your scene JSON files using the ${ENV_NAME} syntax.

    VoiceChat Parameters

    For detailed VoiceChat parameter specifications, refer to the Volcengine VoiceChat Documentation. You can quickly obtain parameters by running a demo in the Volcengine console and clicking the 接入 API (Access API) button to copy the code into your scene JSON configuration.

  8. Configure required environment variables

    main

    The server requires specific credentials in the root .env.local to sign OpenAPI requests and manage RTC. If any are missing, the server will fail to start.

    Required Variables:

    • RTC_APP_ID: Your RTC Application ID.
    • RTC_APP_KEY: Your RTC Application Key.
    • VOLCENGINE_ACCESS_KEY_ID: Volcengine Access Key.
    • VOLCENGINE_SECRET_ACCESS_KEY: Volcengine Secret Access Key.

    Optional Variables:

    • RTC_BUSINESS_ID: Passed to both the RTC SDK and VoiceChat.
    • PORT: Sets the server port (default is used if not specified).

    Note: For security, do not put secrets directly in server/scenes/*.json. Instead, use the ${ENV_NAME} syntax to reference variables defined in .env.local.

  9. Initialize and use the RTCClient

    main

    The RTCClient is a singleton class used to manage Real-Time Communication (RTC) sessions, media streams (audio/video/screen), and AIGC (AI Generated Content) agent interactions.

    To use it, you must follow this lifecycle:

    1. Set Credentials: Provide AppId, RoomId, UserId, and Token using setBasicInfo.
    2. Create Engine: Initialize the underlying RTC engine with createEngine(). This also attempts to register the RTCAIAnsExtension for AI noise reduction.
    3. Add Listeners: Register event handlers via addEventListeners to respond to user joins, leaves, or stream changes.
    4. Join Room: Call joinRoom() to enter the RTC session.
    5. Manage Media: Use methods like startVideoCapture, startAudioCapture, or publishStream to control media.
    6. Cleanup: Call leaveRoom() to exit and destroy the engine.
    import RTCClient from '@/lib/RtcClient';

// 1. Setup credentials
RTCClient.setBasicInfo({
  AppId: 'your_app_id',
  RoomId: 'your_room_id',
  UserId: 'your_user_id',
  Token: 'your_token'
});

// 2. Initialize engine
await RTCClient.createEngine();

// 3. Setup event listeners
RTCClient.addEventListeners({
  handleUserJoin: (e) => console.log('User joined:', e),
  handleError: (e) => console.error('RTC Error:', e.errorCode),
  // ... other handlers
});

// 4. Join the room
await RTCClient.joinRoom();

  10. Configure the AIGC Server environment

    main

    The AIGC server requires specific environment variables to authenticate with Volcengine RTC services and configure the runtime.

    Required Variables:

    • VOLCENGINE_ACCESS_KEY_ID: Your Volcengine access key ID.
    • VOLCENGINE_SECRET_ACCESS_KEY: Your Volcengine secret access key.
    • RTC_APP_ID: The ID of your RTC application.
    • RTC_APP_KEY: The key of your RTC application.

    Optional Variables:

    • RTC_BUSINESS_ID: The business ID for RTC.
    • HOST: The host address to bind the server to (defaults to 127.0.0.1).
    • PORT: The port number for the server (defaults to 3001).

    Note: VOLCENGINE_ACCESS_KEY_ID and VOLCENGINE_SECRET_ACCESS_KEY must both be provided for the server to start.

  11. Troubleshoot common AIGC Demo issues

    main
    IssueSolution
    AI/Digital Human stuck on "Preparing"Check page errors and server logs. Verify field casing, types, and values against VoiceChat parameter docs. For digital humans, check AppId, Token, and concurrency limits.
    token_error in browserEnsure RTC_APP_ID and RTC_APP_KEY in .env.local belong to the same application. Restart the server and refresh the page.
    getUserMedia TypeErrorEnsure the page is running in a Secure Context (either localhost or https).
    Invalid 'Authorization' headerVerify that the AK/SK pair in .env.local is correct and has sufficient permissions.
    StartVoiceChat says task already startedThe demo normally merges duplicate requests in a session. If you modified logic, you must stop the existing task using its AppId, RoomId, or TaskId before starting a new one.

  12. Troubleshooting and Limitations

    main

    When developing with the Node Server, keep the following in mind:

    • Error Handling: Business errors are returned via the API response body, not just HTTP status codes.
    • STS: The server only proxies StartVoiceChat and StopVoiceChat; it does not provide an STS (Security Token Service) HTTP interface.
    • CORS: For local development, CORS is configured to allow any origin. Do not use this configuration in a production environment.
    • UI Control: The server inspects your VoiceChat configuration for visual model settings. It returns information to the frontend to determine whether to display specific UI elements.