Gemini API Cookbook

The Gemini Multimodal Live API is served over secure WebSockets. While Python developers are encouraged to use the official Google GenAI SDK to access these capabilities, the cookbook provides low-level WebSocket examples to help you understand the underlying protocol.

Available resources include:

• Local Python Script: A script that uses websockets to support streaming audio in and audio + video out from your local machine.
• Bash Example: A shell script using websocat to interact with the Live API.
• Google Colab Notebooks: Interactive notebooks for exploring the Live API and implementing tool use via WebSockets.

The Gemini File API allows you to upload files (such as images, audio, or video) to be used in multimodal GenerateContent calls.

Note: The File API is currently in beta and availability is restricted to certain regions.

For interactive exploration, you can use the following Colab notebooks:

• File API Colab
• Audio Colab
• Video understanding Colab

For an interactive environment without local setup, use the following Google Colab notebooks:

• Live API starter (Get_started_LiveAPI.ipynb): Provides an overview of the Multimodal Live API using WebSockets.
• Live API tool use (Get_started_LiveAPI_tools.ipynb): Demonstrates how to implement tool use within the Live API via WebSockets.

While the Gemini API is a REST API that can be called via curl or Postman, Google provides official SDKs for several languages:

• Python: googleapis/python-genai
• Go: googleapis/go-genai
• Node.js: googleapis/js-genai
• Java: googleapis/java-genai
• C#: googleapis/dotnet-genai

Enhance model accuracy using these techniques:

• Grounding: Use external sources like Google Search, Google Maps, YouTube, or URL context to ground answers.
• File Search: Use the hosted RAG (Retrieval-Augmented Generation) capability of the File Search tool to upload and search your own data.

Two primary paths exist for video generation:

1. Omni Flash: Use the Interactions API for high-speed video generation and conversational editing.
2. Veo: Use the Veo models for standard video generation tasks.

To run the Python sample client for the File API, follow these steps to prepare your environment and run the code:

1. Prepare a virtual environment:

   python3 -m venv venv
   source venv/bin/activate

2. Configure your API key: Create a .env file and add your GOOGLE_API_KEY.

   touch .env
   echo "GOOGLE_API_KEY='YOUR_API_KEY'" >> .env

3. Install dependencies:

   pip3 install -U -r requirements.txt

4. Run the sample:

   python3 sample.py

To run the Node.js sample client for the File API, ensure npm is installed and follow these steps:

1. Configure your API key: Create a .env file and add your GOOGLE_API_KEY.

   touch .env
   echo "GOOGLE_API_KEY='YOUR_API_KEY'" >> .env

2. Install dependencies:

   npm install

3. Run the sample:

   npm start

The cookbook provides several Quick Start tutorials (often available as Google Colab notebooks) to help you learn individual Gemini API features. Key tutorials include:

• Live API: Comprehensive overview of Live API capabilities.
• Live Translation: Real-time audio stream translation.
• Omni Flash: Conversational video generation and editing using the Interactions API.
• Veo: Video generation capabilities.
• Imagen & Native Image Generation: Image generation capabilities.
• Grounding: Using Google Search for grounded responses.
• Code Execution: Generating and running Python code to solve complex tasks and output graphs.