TensorFlowASR

TensorFlowASR supports several Automatic Speech Recognition (ASR) architectures, categorized into two main baseline types:

Transducer Models

End-to-end models using RNNT Loss for training. Supported architectures include:

• Conformer (See examples/models/transducer/conformer)
• Streaming Conformer (See examples/models/transducer/conformer)
• ContextNet (See examples/models/transducer/contextnet)
• RNN Transducer (See examples/models/transducer/rnnt)
• Streaming Transducer

CTC Models

End-to-end models using CTC Loss for training. Supported architectures include:

• DeepSpeech2 (See examples/models/ctc/deepspeech2)
• Jasper (See examples/models/ctc/jasper)

All models can be converted to TFLite to reduce memory and computation for deployment.

Configuration files use the .yml.j2 extension, meaning they are Jinja2 templates containing YAML content. This allows you to import and compose configurations from different modules, such as decoder settings and model architectures.

Example of a composed config file:

{% import "examples/datasets/librispeech/sentencepiece/sp.yml.j2" as decoder_config with context %}
{{decoder_config}}

{% import "examples/models/transducer/conformer/small.yml.j2" as config with context %}
{{config}}

The Sentencepiece Tokenizer splits an entire sentence directly into subwords and maps each subword to an index.

Key details:

• The blank token can be set to � at index 0.
• Best use case: Languages with a large vocabulary where words are combinations of other words; it is applicable to any language.

The Wordpiece Tokenizer splits text into words and then further decomposes those words into subwords before mapping them to indices.

Key details:

• The blank token can be set to � at index 0.
• Best use case: Languages with a large vocabulary where words are combinations of other words; it is applicable to any language.
• Variants:
  • Wordpiece split by whitespace.
  • Wordpiece where whitespace is treated as a separate token.

The Character Tokenizer splits text into individual characters and maps each to an index.

Key details:

• The index starts from 1.
• The index 0 is reserved for the blank token.
• Best use case: Languages with a small number of characters where characters are not combinations of other characters (e.g., English, Vietnamese).

Refer to the librespeech configuration templates for implementation details.

To support streaming inference, use the model's output states to feed the next chunk of audio.

For each chunk of audio signal, the model produces outputs that include next_tokens, next_encoder_states, and next_decoder_states. To continue the stream, overwrite the current previous_tokens, previous_encoder_states, and previous_decoder_states with these next_* values before processing the next audio chunk.

schemas.PredictOutputWithTranscript(
    transcript=self.tokenizer.detokenize(outputs.tokens),
    tokens=outputs.tokens,
    next_tokens=outputs.next_tokens,
    next_encoder_states=outputs.next_encoder_states,
    next_decoder_states=outputs.next_decoder_states,
)

If you are training on TPUs, you must create tfrecords datasets. You can use the tensorflow_asr utils create_tfrecords command to convert your data. You can specify which modes to generate using the --mode flag (e.g., ["train","eval"]).

tensorflow_asr utils create_tfrecords \
    --config-path=/path/to/config.yml.j2 \
    --mode=["train","eval","test"] \
    --datadir=/path/to/datadir

To run the TFLite demonstration in streaming mode (processing audio in real-time), you must install the python-sounddevice dependency. The specific implementation details and execution logic are contained within the source code of the streaming example file.

pip install sounddevice

Use the tensorflow_asr tflite CLI command to convert a trained Keras model (.h5) into a TensorFlow Lite (.tflite) file.

Key parameters:

• --config-path: Path to the model configuration file (.yml.j2).
• --h5: Path to the trained weights file.
• --bs: Batch size (typically 1 for mobile/edge inference).
• --beam-width: Set to 0 for greedy search, or >0 to enable beam search.
• --output: Destination path for the .tflite file.

tensorflow_asr tflite \
    --config-path=/path/to/config.yml.j2 \
    --h5=/path/to/weight.h5 \
    --bs=1 \
    --beam-width=0 \
    --output=/path/to/output.tflite

Configuration files use the config.yml.j2 format, which is a Jinja2 template containing YAML content. The testing configuration should be identical to the configuration used during training. It defines inputs, outputs, and vocabulary options. You can use Jinja2 imports to compose configurations from existing templates found in examples/*/*.yml.j2.

{% import "examples/datasets/librispeech/sentencepiece/sp.yml.j2" as decoder_config with context %}
{{decoder_config}}

{% import "examples/models/transducer/conformer/small.yml.j2" as config with context %}
{{config}}