repository·master·Indexed 18 days ago
https://github.com/reazon-research/reazonspeechA speech recognition library (version 1.1.1) providing tools for Japanese speech-to-text using NeMo ASR and AVSR encoder-decoder models. It includes a CLI for transcribing audio into VTT, SRT, ASS, JSON, and TSV formats, a standardized evaluation interface for calculating Character Error Rate (CER), and utilities for handling audio corpora and M2TS files.
ReazonSpeech is a project focused on building large-scale open Japanese speech corpora. It provides various speech recognition (ASR) and audio-visual speech recognition (AVSR) packages.
AVista is a sub-project focused on noise-robust multimodal speech recognition for human-robot interaction, standing for Audio-VIsual Speech Transcription & Alignment.
ReazonSpeech is distributed as several specialized packages depending on the model architecture and requirements:
reazonspeech.k2.asr: Fast and accurate next-gen Kaldi models (159M parameters). Requires sherpa-onnx. Includes a bilingual (ja-en) model for high-accuracy language detection.reazonspeech.nemo.asr: Fast, accurate speech recognition based on FastConformer-RNNT (619M parameters). Requires Nvidia Nemo.reazonspeech.espnet.asr: Speech recognition using a Conformer-Transducer model (120M parameters). Requires ESPnet.reazonspeech.avsr: Audio-Visual Speech models for the AVista project. Follows the Hugging Face Transformers interface and supports Auto classes.reazonspeech.evaluation: Tools for evaluating ReazonSpeech and other speech recognition models.reazonspeech.espnet.oneseg: Tools for analyzing Japanese "one-segment" TV streams, used for creating Japanese audio corpora.When preparing inputs for AVSR models using a processor (like AVHubertProcessor) or a feature extractor (like AVHubertFeatureExtractor), you can control whether mouth extraction is performed.
If the video input does not already have mouth extraction performed, pass extract_mouth=True to the call.
To install the ReazonSpeech ESPnet ASR interface, clone the main repository and install the specific package using pip:
$ git clone https://github.com/reazon-research/ReazonSpeech
$ pip install ReazonSpeech/pkg/espnet-asrThe Python interface allows you to load ReazonSpeech models from Hugging Face, load local audio files, and perform speech recognition using load_model, audio_from_path, and transcribe.
from reazonspeech.nemo.asr import load_model, transcribe, audio_from_path
# Load ReazonSpeech model from Hugging Face
model = load_model()
# Read a local audio file
audio = audio_from_path("speech.wav")
# Recognize speech
ret = transcribe(model, audio)To use the ReazonSpeech K2 models, you must first install sherpa_onnx following its official documentation. Once sherpa_onnx is installed, you can install the reazonspeech.k2.asr package by cloning the repository and installing the specific package directory.
$ git clone https://github.com/reazon-research/ReazonSpeech
$ pip install ReazonSpeech/pkg/k2-asrTo install the reazonspeech.espnet.oneseg module, clone the main repository and install the package from the local directory. This module requires ESPnet and ffmpeg to be installed on your system.
$ git clone https://github.com/reazon-research/reazonspeech
$ pip install reazonspeech/pkg/espnet-onesegYou can install the evaluation package directly from the GitHub repository using pip by specifying the subdirectory, or by cloning the repository and installing the package locally.
$ pip install git+https://github.com/reazon-research/ReazonSpeech.git#subdirectory=pkg/evaluation
# OR
$ git clone https://github.com/reazon-research/ReazonSpeech
$ pip install ReazonSpeech/pkg/evaluationYou can generate a corpus (audio and transcriptions) from a .m2ts stream by combining reazonspeech.espnet.oneseg functions with an ESPnet CTCSegmentation model.
Prerequisites:
ffmpeg and git-lfs via your package manager.reazonspeech-espnet-v2) and link its exp directory.Workflow:
CTCSegmentation with the model's config and weights.get_utterances() to extract audio segments and transcriptions.save_as_zip() to package the results.$ sudo apt install ffmpeg git-lfs
$ git clone https://huggingface.co/reazon-research/reazonspeech-espnet-v2
$ ln -s reazonspeech-espnet-v2/expfrom espnet2.bin.asr_align import CTCSegmentation
from reazonspeech.espnet.oneseg import get_utterances, save_as_zip
# Load audio and ASR model
ctc_segmentation = CTCSegmentation(
asr_train_config="exp/asr_train_asr_conformer_raw_jp_char/config.yaml",
asr_model_file="exp/asr_train_asr_conformer_raw_jp_char/valid.acc.ave_10best.pth",
kaldi_style_text=False,
fs=16000,
)
# Extract audio and transcriptions
utt = get_utterances("test.m2ts", ctc_segmentation)
save_as_zip(utt, path="corpus.zip")The repository contains specific examples for evaluating various models, including Whisper and different versions of ReazonSpeech. You can find these in the pkg/evaluation/examples/ directory:
pkg/evaluation/examples/whisperpkg/evaluation/examples/rs-espnetpkg/evaluation/examples/rs-nemopkg/evaluation/examples/rs-k2You can install the reazonspeech.avsr package directly from the GitHub repository using pip or by cloning the repository first.
$ pip install git+https://github.com/reazon-research/ReazonSpeech.git#subdirectory=pkg/avsr
# Or via git clone
$ git clone https://github.com/reazon-research/ReazonSpeech
$ pip install ReazonSpeech/pkg/avsrYou can perform speech recognition in Python using the load_model, audio_from_path, and transcribe functions. The load_model function automatically fetches the ReazonSpeech model from Hugging Face.
from reazonspeech.espnet.asr import load_model, transcribe, audio_from_path
# Load ReazonSpeech model from Hugging Face
model = load_model()
# Read a local audio file
audio = audio_from_path("speech.wav")
# Recognize speech
ret = transcribe(model, audio)