DOCSEARCHbyAlgolia
Submit Docs

Google Assistant SDK for Python

repository·master·Indexed 21 days ago

https://github.com/googlesamples/assistant-sdk-python

Python bindings and reference implementations for the Google Assistant SDK. It includes the google-assistant-grpc package for communicating with the Google Assistant Service via gRPC and the google-assistant-sdk collection of tools and samples, such as push-to-talk, text input, and audio file input implementations.

Tokens
2.8K
Snippets
10
Records
12
Agent score
75%

What's inside assistant-sdk-python

  1. Overview of Google Assistant SDK for Python

    master

    This repository provides Python packages and samples for interacting with the Google Assistant SDK. It is primarily composed of two main components:

    1. google-assistant-grpc: Autogenerated Python bindings used to communicate directly with the Google Assistant Service via gRPC. Note that this is available for non-commercial use.
    2. google-assistant-sdk: A collection of tools and Python samples designed to help developers implement the SDK. This includes the googlesamples/assistant/grpc reference sample for the Google Assistant Service.

    Note: The google-assistant-library and its associated samples are deprecated and should not be used for new projects.

  2. Use the Assist streaming method

    master

    The core interaction with the Assistant happens via the Assist streaming method. This method is a bidirectional stream: it accepts a generator of AssistRequest messages and returns a generator of AssistResponse messages.

    To use it, you must first yield an AssistConfig message to set up audio parameters, followed by AssistRequest messages containing the raw audio data.

    import google.assistant.embedded.v1alpha1.embedded_assistant_pb2

def generate_assist_requests():
    # 1. Send configuration first
    yield google.assistant.embedded.v1alpha1.embedded_assistant_pb2.AssistConfig(
        audio_in_config=google.assistant.embedded.v1alpha1.embedded_assistant_pb2.AudioInConfig(
            encoding='LINEAR16',
            sample_rate_hertz=16000,
        ),
        audio_out_config=google.assistant.embedded.v1alpha1.embedded_assistant_pb2.AudioOutConfig(
            encoding='LINEAR16',
            sample_rate_hertz=16000,
        ),
        device_config=google.assistant.embedded.v1alpha1.embedded_assistant_pb2.DeviceConfig(
            device_id=device_id,
            device_model_id=device_model_id,
        )
    )
    # 2. Send audio data stream
    for data in acquire_audio_data():
        yield google.assistant.embedded.v1alpha1.embedded_assistant_pb2.AssistRequest(audio_in=data)

# 3. Call the stream
assist_responses_generator = assistant.Assist(generate_assist_requests())

  3. Authorize the Assistant SDK using OAuthlib

    master

    After configuring your Actions Console project and downloading your client secrets file, you must generate device credentials using the google-oauthlib-tool.

    1. Install the tool:
    pip install --upgrade google-auth-oauthlib[tool]
    1. Generate credentials: Use the --scope https://www.googleapis.com/auth/assistant-sdk-prototype to authorize the device.
    google-oauthlib-tool --client-secrets path/to/client_secret_<client-id>.json --scope https://www.googleapis.com/auth/assistant-sdk-prototype --save --headless
    pip install --upgrade google-auth-oauthlib[tool]
google-oauthlib-tool --client-secrets path/to/client_secret_<client-id>.json --scope https://www.googleapis.com/auth/assistant-sdk-prototype --save --headless

  4. Authorize the Google Assistant SDK

    master

    Authorization requires configuring an Actions Console project, registering a device model, and generating device credentials.

    1. Follow the Google Assistant SDK guides to configure your project and register a device model to obtain a client_secret_<client-id>.json file.
    2. Install the OAuthlib tool:
      pip install --upgrade google-auth-oauthlib[tool]
    3. Generate credentials using google-oauthlib-tool with the required scope:
      google-oauthlib-tool --client-secrets path/to/client_secret_<client-id>.json --scope https://www.googleapis.com/auth/assistant-sdk-prototype --save --headless
    4. Load the resulting credentials in your Python code using google.oauth2.credentials.
    import io
import json
import google.oauth2.credentials

with io.open('/path/to/credentials.json', 'r') as f:
    credentials = google.oauth2.credentials.Credentials(token=None, **json.load(f))

  5. Install and Setup the Google Assistant gRPC Python Samples

    master

    To use these samples, you need Python (>= 3.4 recommended), an Actions Console Project, and a Google account.

    1. Install Python 3

    On Ubuntu/Debian:

    sudo apt-get update
sudo apt-get install python3 python3-venv

    2. Create a Virtual Environment

    It is recommended to use a virtual environment:

    python3 -m venv env
source env/bin/activate
# Upgrade core tools
env/bin/python -m pip install --upgrade pip setuptools wheel

    3. Install Dependencies

    You must install system-level audio and development libraries before installing the Python requirements:

    sudo apt-get install portaudio19-dev libffi-dev libssl-dev
pip install --upgrade -r requirements.txt
    sudo apt-get update
sudo apt-get install python3 python3-venv
python3 -m venv env
source env/bin/activate
sudo apt-get install portaudio19-dev libffi-dev libssl-dev
pip install --upgrade -r requirements.txt

  6. Handle AssistResponse messages

    master

    When iterating over the AssistResponse generator, you should handle several specific fields and event types:

    • event_type == END_OF_UTTERANCE: Use this signal to stop your local audio recording.
    • speech_results: Contains the transcription of the user's query. Each result has a transcript field.
    • dialog_state_out.supplemental_display_text: Contains metadata like text to be displayed on a screen.
    • audio_out.audio_data: Contains the raw audio bytes of the Assistant's response for playback.
    for resp in assist_responses_generator:
    # Detect end of user speech
    if resp.event_type == END_OF_UTTERANCE:
       stop_acquiring_audio()

    # Get transcription
    if resp.speech_results:
       print(' '.join(r.transcript for r in resp.speech_results))

    # Get supplemental display text
    if resp.dialog_state_out.supplemental_display_text:
       print(resp.dialog_state_out.supplemental_display_text)

    # Play back Assistant audio
    if len(resp.audio_out.audio_data) > 0:
       playback_audio_data(resp.audio_out.audio_data)

  9. Troubleshoot Audio issues (Choppy or Truncated audio)

    master

    If you experience audio issues, use the audio_helpers module to test and adjust block and flush sizes.

    Verify ALSA Setup

    Before debugging the SDK, ensure your system audio is working:

    # Play a test sound
speaker-test -t wav

# Record and play back using ALSA
arecord --format=S16_LE --duration=5 --rate=16000 --file-type=raw out.raw
aplay --format=S16_LE --rate=16000 --file-type=raw out.raw

    Fix Choppy Audio (Adjust Block Size)

    If audio is choppy, adjust the --audio-block-size.

    • USB Speaker/Soundcard: Set block size to 0 to auto-adjust.
    • 3.5mm Line-out Jack: Set to a value larger than the ConverseResponse payload (e.g., 3200).
    # Test block size
python -m audio_helpers --audio-block-size=3200

# Run sample with the tested block size
python -m pushtotalk --audio-block-size=3200

    Fix Truncated Audio (Adjust Flush Size)

    If audio is cut off, increase the --audio-flush-size. It should be larger than the block size.

    python -m audio_helpers --audio-block-size=3200 --audio-flush-size=6400
    python -m audio_helpers --audio-block-size=3200 --audio-flush-size=6400

  10. Run the Push-to-Talk sample

    master

    The pushtotalk module implements a feature where the sample records a voice query after a key press and plays back the Google Assistant's answer.

    Basic usage:

    python -m pushtotalk --device-id 'my-device-identifier' --device-model-id 'my-model-identifier'

    Common flags:

    • -v: Verbose mode to see gRPC communication.
    • -i <file.wav>: Send a pre-recorded audio file as the request.
    • -o <file.wav>: Save the Assistant's audio response to a file.

    Example: Send audio file and save response:

    python -m pushtotalk --device-id 'my-device-identifier' --device-model-id 'my-model-identifier' -i in.wav -o out.wav
    python -m pushtotalk --device-id 'my-device-identifier' --device-model-id 'my-model-identifier'

  11. Run Text Input and Audio File Input samples

    master

    In addition to push-to-talk, you can interact with the Assistant using text or direct audio files.

    Text Input: Send text requests directly to the Assistant:

    python -m textinput --device-id 'my-device-identifier' --device-model-id 'my-model-identifier'

    Audio File Input: Send a request from a local audio file and write the response to another file:

    python -m audiofileinput --device-id 'my-device-identifier' --device-model-id 'my-model-identifier' -i in.wav -o out.wav
    python -m textinput --device-id 'my-device-identifier' --device-model-id 'my-model-identifier'

  12. Initialize the Assistant gRPC stub

    master

    To interact with the Assistant service, initialize the gRPC stubs using the embedded_assistant_pb2_grpc module and pass your authenticated gRPC channel to the EmbeddedAssistantStub.

    import google.assistant.embedded.v1alpha1.embedded_assistant_pb2_grpc
# Assuming 'channel' is an initialized gRPC channel
assistant = google.assistant.embedded.v1alpha1.embedded_assistant_pb2_grpc.EmbeddedAssistantStub(channel)