Wyoming Protocol Documentation

The Wyoming Protocol operates using a client-server event model where events are exchanged over a connection.

Event Directionality:

• → indicates an event from the client to the server.
• ← indicates an event from the server to the client.

Standard Connection Lifecycle:

1. Service Description: The client must first send a describe event, and the server responds with an info event.
2. Program Selection: If a server exposes multiple programs of the same type (e.g., multiple asr programs), the client can optionally send a select-program event with the name of the desired program. If omitted, the first program of each type listed in the info event is used.
3. Request Execution: Once a program is selected, the client sends request events (e.g., transcribe, synthesize, detect, recognize) specific to that program.

Once an intent is recognized, it must be handled by the server.

Structured Intents

1. → intent (required)
2. ← handled (if successful)
3. ← not-handled (if unsuccessful)

Text-Only Intents

1. → transcript (required: include text to handle)
2. ← handled (if successful)
3. ← not-handled (if unsuccessful)

Streaming Text-Only Intents (Successful)

1. → transcript (required: include text to handle)
2. ← handled-start (required)
3. ← handled-chunk (required: chunk of response text)
4. ← handled (sent for backwards compatibility)
5. ← handled-stop (required)

To generate speech from text, use the following event sequences.

Standard TTS

1. → synthesize (required: include text)
2. ← audio-start
3. ← audio-chunk (one or more chunks)
4. ← audio-stop

Streaming TTS

1. → synthesize-start (required)
2. → synthesize-chunk (required: send text chunks as they are produced)
3. ← audio-start, ← audio-chunk (one or more), ← audio-stop (server sends audio chunks as they are produced)
4. → synthesize (sent for backwards compatibility)
5. → synthesize-stop (end of text stream)
6. ← audio-start, ← audio-chunk (one or more), ← audio-stop (final audio must be sent)
7. ← synthesize-stopped (server acknowledges final audio has been sent)

To perform speech recognition, use the following event sequences depending on whether you need standard or streaming transcription.

Standard Transcription

1. → transcribe (optional: include name of model or language)
2. → audio-start (required)
3. → audio-chunk (required: send chunks until silence is detected)
4. → audio-stop (required)
5. ← transcript (required: contains the text transcription)

Streaming Transcription

1. → transcribe (optional)
2. → audio-start (required)
3. → audio-chunk (required: send chunks until silence is detected)
4. ← transcript-start (required)
5. ← transcript-chunk (required: server sends transcript chunks as they are produced)
6. → audio-stop (required)
7. ← transcript (sent for backwards compatibility)
8. ← transcript-stop (required)

To detect speech activity:

1. → audio-chunk (required: send chunks until silence is detected)
2. ← voice-started (when speech starts)
3. ← voice-stopped (when speech stops)

By default, a server uses the first program of each type listed in its info response. To use a specific program, send a select-program event immediately after connecting and before sending any request events (like describe or transcribe).

• The name must match a program name provided in the info event.
• The selection applies for the lifetime of the connection.
• The domain (e.g., asr, tts) is implied by the subsequent request events.

{
  "type": "select-program",
  "name": "my-custom-asr-model"
}

To detect specific wake words, follow this flow:

1. → detect (optional: include names of wake words to detect)
2. → audio-start (required)
3. → audio-chunk (required: keep sending chunks until a detection is received)
4. ← detection (sent for each wake word detection)
5. → audio-stop (optional: manually end the audio stream)
6. ← not-detected (sent after audio-stop if no detections occurred)

To recognize user intents, use the following flows.

Single Intent

1. → recognize (required)
2. ← intent (if successful)
3. ← not-recognized (if unsuccessful)

Multiple Intents

1. → recognize (required)
2. ← intents-start (if successful)
3. ← intent (one or more intents)
4. ← intents-stop (if successful)
5. ← not-recognized (if unsuccessful)

To play audio back to a user:

1. → audio-start (required)
2. → audio-chunk (required: one or more chunks)
3. → audio-stop (required)
4. ← played (acknowledgment)

Wyoming uses a discovery pattern to allow clients to understand what services a server provides.

1. Describe: The client sends a Describe event (type describe) to request information.
2. Info: The server responds with an Info event (type info) containing a detailed breakdown of available services, models, and hardware capabilities.

The Info event includes lists of programs for various domains such as asr (speech-to-text), tts (text-to-speech), handle (intent handling), intent (intent recognition), wake (wake word detection), mic (audio input), and snd (audio output), as well as satellite metadata.

# Client requesting info
describe_event = Describe()
# The server would then respond with an Info event