T-one Streaming ASR Pipeline

T-one is a streaming CTC-based ASR pipeline optimized for Russian telephony. It operates in three main stages:

1. Acoustic Model (Conformer-based): Incoming audio is sliced into 300 ms segments. The model processes a segment along with a hidden state from the previous segment to maintain context. It outputs frame-level log-probabilities (logprobs) for the Russian alphabet (letters, space, and a blank token).
2. Log-probability Splitter: This component uses the logprobs to detect speech boundaries. A phrase starts at the first speech frame and ends after $N$ consecutive non-speech frames. It outputs detected phrases with start/end timestamps.
3. Decoder: Converts logprobs into text using one of two methods:
   • Greedy decoding: Fast, direct selection.
   • Beam Search decoding: Uses a KenLM language model for higher accuracy.

This modular design allows for low-latency, high-throughput processing.

To install T-one for development or local usage, ensure you have Python 3.9+ and Poetry 2.1+ installed.

Note for Windows users: The dependency kenlm does not officially support Windows. It is highly recommended to use Linux or WSL (Windows Subsystem for Linux).

Method 1: Using Makefile

If you have cloned the repository, you can use the provided Makefile:

make install
make up_dev

This will install dependencies and start the demo web service.

Method 2: Manual Poetry Installation

1. Clone the repository:

   git clone https://github.com/voicekit-team/T-one.git
   cd T-one

2. Create and activate a virtual environment:

   python -m venv .venv
   source .venv/bin/activate

3. Install the package with demo dependencies:

   poetry install -E demo

4. Run the demo web-service using uvicorn:

   uvicorn --host 0.0.0.0 --port 8080 tone.demo.website:app --reload

To deploy the T-one acoustic model in Triton for high performance, you must organize the model files into a specific directory structure and use the appropriate configuration file.

1. Directory Structure

Create the following hierarchy in your models directory:

models
└── streaming_acoustic
    ├── 1
    │   └── model.onnx OR model.plan
    └── config.pbtxt

2. Configuration Selection

Copy the correct config.pbtxt from the configs directory based on your runtime:

• For ONNX Runtime: Use configs/streaming_acoustic/config.pbtxt.
• For TensorRT: Use configs/streaming_acoustic_trt/config.pbtxt.

3. Model Preparation

• For ONNX Runtime: Download the model into the version folder: python -m tone download $(pwd)/models/streaming_acoustic/1 --only-acoustic
• For TensorRT: Convert the ONNX model to TensorRT format using the trtexec instructions provided in the export guide. TensorRT 10.x is supported.

4. Run the Server

Launch the Triton server using Docker, mapping the necessary ports and your model repository volume.

# Run Triton Inference Server
docker run --rm \
  -p 8000:8000 -p 8001:8001 -p 8002:8002 \
  -v $(pwd)/models:/models \
  nvcr.io/nvidia/tritonserver:25.06-py3 \
  tritonserver --model-repository=/models

TensorRT engines are hardware-dependent and must be compiled on the specific target configuration (e.g., specific GPU and CUDA version) to ensure optimal performance. Follow these steps to compile the model:

1. Download the ONNX model: Use the tone module to download only the acoustic component.
2. Compile with trtexec: Use the NVIDIA Triton container to convert the ONNX model into a .plan engine file. You should adjust $BATCH_SIZE to balance throughput and latency for your specific device.

Input Shape Details:

• minShapes: The minimum batch size is 1. The input tensor length is 2400 (calculated as sample_rate * chunk_size, e.g., 8000 * 0.3). The state size is 219729.
• optShapes and maxShapes: Use your chosen $BATCH_SIZE for these dimensions.

After compilation, trtexec will output a performance metrics report.

# 1. Download ONNX model
python -m tone download $(pwd)/models/model.onnx --only-acoustic

# 2. Compile to TensorRT
docker run --rm -it \
  -v $(pwd)/models:/models \
  nvcr.io/nvidia/tritonserver:25.06-py3 \
  /usr/src/tensorrt/bin/trtexec \
    --onnx=/models/model.onnx \
    --minShapes=signal:1x2400x1,state:1x219729 \
    --optShapes=signal:$BATCH_SIZEx2400x1,state:$BATCH_SIZEx219729 \
    --maxShapes=signal:$BATCH_SIZEx2400x1,state:$BATCH_SIZEx219729 \
    --builderOptimizationLevel=5 \
    --stronglyTyped \
    --useSpinWait \
    --noDataTransfers \
    --saveEngine=/models/model.plan

The perf_analyzer tool from the Triton SDK can be used to test both ONNX and TensorRT models.

Setup: Run the Triton SDK container. Note: Use version 24.02 to avoid an Out-of-Memory (OOM) issue present in version 25.06.

docker run --rm -it --net host \
  nvcr.io/nvidia/tritonserver:24.02-py3-sdk

Running the benchmark: Execute perf_analyzer with parameters optimized for streaming ASR.

Key Flags:

• --sequence-length: Input sequence length. Note that sequence-length = 1 corresponds to 300ms of audio.
• --request-rate-range: The range of request frequencies to test.
• --latency-threshold: Latency threshold in milliseconds.
• -u: URL of the server.
• -i: Protocol (e.g., grpc).
• -m: Model name.
• --streaming: Enables streaming mode.

perf_analyzer -u localhost:8001 \
  -i grpc \
  -a \
  -m streaming_acoustic \
  --streaming \
  --sequence-length=50 \
  --measurement-mode=count_windows \
  --measurement-request-count=5000 \
  --request-rate-range=2048:4096:256 \
  --stability-percentage=100 \
  --latency-threshold=100