Alibaba Cloud Bailian Speech Demo

This repository provides sample code for developers to call Tongyi Speech Models (such as CosyVoice, Paraformer, SenseVoice, and Gummy) via Alibaba Cloud Bailian.

Key capabilities include:

• Speech Recognition: Speech-to-text.
• Speech Synthesis: Text-to-speech.
• Advanced AI Functions: Combining speech models with Large Language Models (LLMs) like Tongyi OMNI, Tongyi Qianwen, Baichuan, Moonshot (Mooncake), etc., to achieve video/voice chat, speech analysis, and speech translation.

The DashScope Multimodal Dialog API enables real-time multimodal conversations. It supports Speech-to-Speech interaction and can process multimodal data such as images and videos. The service is designed for low-latency communication using WebSocket and supports integration with Bailian official and third-party agents.

Key Capabilities:

• Voice Conversation: Real-time speech recognition and synthesis.
• Image Q&A: Intelligent question-answering based on image content.
• Interaction Modes: Supports Push2Talk, Tap2Talk, and Duplex modes.
• Streaming: Real-time audio stream transmission and processing via WebSocket.

The DashScope Multimodal Dialog API provides real-time Speech-to-Speech interaction capabilities. It supports multimodal data including images and videos, and integrates with plugins, official applications, and third-party applications within the Bailian ecosystem.

Key Features:

• Voice Conversation: Real-time speech recognition and synthesis.
• Image QA: Intelligent question-answering based on image content.
• Interaction Modes: Supports Push2Talk, Tap2Talk, and Duplex modes.
• Streaming: Real-time audio stream transmission and processing.
• Low Latency: Uses WebSocket connections for real-time communication.
• Agent Support: Integration with official and third-party Agents.

The Qwen-Omni series supports multi-modal inputs including video, audio, images, and text, and outputs audio and text. The Real-time API provides low-latency interaction capabilities, supporting streaming input for audio and video, and streaming output for text and audio.

Key features include:

• Voice Conversation: Multi-turn dialogue using real-time voice and video as input.
• Voice Interruption: Supports interrupting the model at any time via voice.
• WebSocket Connection: Utilizes WebSocket for low-latency real-time communication.

This sample demonstrates how to batch submit URLs of audio or video files stored in cloud storage (such as Alibaba Cloud OSS) to the Alibaba Cloud Bailian offline file transcription API.

By using the SenseVoice speech model, you can perform multi-language speech recognition while simultaneously retrieving rich information such as emotions and audio events.

Note: If you only require standard speech-to-text without rich information (emotions/events), it is recommended to use the more cost-effective Paraformer model instead.

This sample demonstrates a complete pipeline for processing video files:

1. Preprocessing: Converts a video file into an OPUS audio file using FFmpeg.
2. Storage: Uploads the audio to Alibaba Cloud OSS.
3. Transcription: Uses audio recognition services (Fun-ASR or Paraformer) to transcribe the audio.
4. LLM Processing: Uses the Qwen large language model to perform translation, content summarization, and question-answering based on the transcription.

This is ideal for applications requiring audio/video speech analysis and understanding.

This sample demonstrates how to use the real-time mode of the sentence-level speech translation interface to process multiple audio file streams. It returns recognition and translation results for each file incrementally as they are processed.

Key Characteristics

• Real-time Mode: Optimized for processing local files with immediate result returns or for building streaming services that collect frontend audio streams and return results instantly.
• Batch Processing: Capable of handling multiple file streams simultaneously.
• Output: Provides continuous, incremental results including the original transcript and the translated text, associated with a unique [process id] for each file.

Recommended Configuration

• Model: gummy-chat-v1
• API: Paraformer Real-time Speech Recognition API

This sample demonstrates how to perform batch speech recognition on multiple audio or video files concurrently using real-time mode. The implementation uses an object pool and a thread pool to process multiple files simultaneously. Each file is read and converted into real-time recognition results independently, with results returned via individual callback functions.

For production environments requiring optimal performance under heavy load, refer to the High Concurrency Example Documentation.

This sample demonstrates how to achieve low-latency, real-time simultaneous interpretation and bilingual subtitles by combining the Gummy real-time speech translation model and the Cosyvoice streaming speech synthesis model from the Bailian platform.

Application Scenarios

• Real-time Translation: Simultaneous interpretation and real-time bilingual subtitles (e.g., translating audio/video files and displaying/reading the translated results).

This sample demonstrates how to use the real-time speech recognition API to process multiple audio or video file streams in batch. The real-time mode is designed for scenarios where you need immediate result returns, such as processing local files or building streaming services that collect frontend audio streams and return recognition results instantly.

Key Considerations:

• High Concurrency: If building a speech service with Java, refer to the high-concurrency example documentation for optimal performance.
• Batch vs. Real-time: For large-scale production tasks involving cloud files where immediate results are not required, use the Batch Mode example instead of this real-time mode example.