Google Gen AI JavaScript SDK

When generating video, you can control the output using masking and reference images.

Video Generation Masking

Use VideoGenerationMask to define how a mask is applied:

• image: The mask image.
• maskMode: Determines the operation:
  • INSERT: Add content.
  • OUTPAINT: Expand content.
  • REMOVE: Delete content.
  • REMOVE_STATIC: Remove static elements.

Reference Images

Use VideoGenerationReferenceImage to guide the style or content:

• image: The reference image.
• referenceType:
  • ASSET: Use the image as a source asset.
  • STYLE: Use the image to influence the visual style.

Choose a model based on your specific task requirements. Avoid using deprecated models like gemini-1.5-flash or gemini-pro.

The generateContent API uses a hierarchy of Content and Part objects. While you can pass a simple string to the contents parameter, it is shorthand for a structured array of Content objects. Each Content object contains a role (e.g., "user") and an array of parts. Each Part can contain text, inlineData, or other modalities.

import { GoogleGenAI } from '@google/genai';
const ai = new GoogleGenAI({});

async function run() {
    // Explicit structure
    const response = await ai.models.generateContent({
        model: "gemini-3-flash-preview",
        contents: [
            {
                role: "user",
                parts: [{ text: "How does AI work?" }]
            },
        ],
    });
    console.log(response.text);
}
run();

The Interactions API provides a unified interface for managing state, tool orchestration, and long-running tasks. It is ideal for multi-turn conversations.

Stateful Conversations: You can maintain a conversation by passing the previous_interaction_id from a prior interaction to the create method of the next interaction.

// 1. First turn
const interaction1 = await ai.interactions.create({
    model: 'gemini-2.5-flash',
    input: 'Hi, my name is Amir.',
});
console.debug(interaction1);

// 2. Second turn (passing previous_interaction_id)
const interaction2 = await ai.interactions.create({
  model: 'gemini-2.5-flash',
  input: 'What is my name?',
  previous_interaction_id: interaction1.id,
});
console.debug(interaction2);

The SDK provides enums to control content filtering via safety settings. You can specify which HarmCategory to monitor and what HarmBlockThreshold to apply.

Harm Categories:

• HARM_CATEGORY_DANGEROUS_CONTENT
• HARM_CATEGORY_HARASSMENT
• HARM_CATEGORY_HATE_SPEECH
• HARM_CATEGORY_SEXUALLY_EXPLICIT
• (and others including CIVIC_INTEGRITY and JAILBREAK)

Block Thresholds:

• BLOCK_LOW_AND_ABOVE
• BLOCK_MEDIUM_AND_ABOVE
• BLOCK_ONLY_HIGH
• BLOCK_NONE
• OFF

When using grounding features (like Google Search or Maps), the response includes GroundingMetadata. This metadata contains groundingChunks, which provide the source information for the model's response.

A GroundingChunk can contain different types of information depending on the source:

• web: Contains domain, title, and uri for web search results.
• maps: Contains placeId, title, uri, and route information for Google Maps grounding.
• retrievedContext: Contains text, title, uri, or ragChunk for RAG-based retrieval.
• image: Contains imageUri or sourceUri for image-based grounding.

All features are accessed through an instance of GoogleGenAI. The SDK is organized into submodules that group related functionality:

• ai.models: Query models (e.g., generateContent, generateImages) and examine model metadata.
• ai.caches: Create and manage caches to reduce costs when using large prompt prefixes repeatedly.
• ai.chats: Manage local stateful chat objects for multi-turn interactions.
• ai.files: Upload and reference files in prompts to reduce bandwidth and handle large files.
• ai.live: Start real-time sessions supporting text, audio, and video input/output.

The SDK provides interfaces for integrating with Vertex AI Search and Retrieval-Augmented Generation (RAG) capabilities.

Vertex AI Search

Use VertexAISearch to specify search parameters:

• datastore: The ID of the data store.
• engine: The search engine ID.
• filter: A string filter for results.
• maxResults: Limit the number of returned results.

Vertex RAG Store

Use VertexRagStore to manage RAG resources:

• ragCorpora: An array of corpus IDs.
• ragResources: Specific resources within corpora via VertexRagStoreRagResource (which can specify ragCorpus and ragFileIds).
• ragRetrievalConfig: Configuration for the retrieval process.
• similarityTopK: Number of similar items to retrieve.
• vectorDistanceThreshold: Threshold for vector similarity.

The contents parameter in generateContent supports flexible structuring:

Content Types:

• Content: The SDK wraps a single Content instance in an array.
• Content[]: The array is used as provided.

Part Types (Aggregated to a 'user' role):

• Part | string: The SDK wraps the string or Part in a Content instance with the role user.
• Part[] | string[]: The SDK wraps the entire list into a single Content instance with the role user.

CRITICAL NOTE: This automatic wrapping does not apply to FunctionCall and FunctionResponse parts. When specifying these, you must explicitly provide the full Content[] structure to define which parts are spoken by the model versus the user, otherwise the SDK will throw an exception.

When using tools, the model may return a FunctionCall, which includes the name of the function and the args to be passed. After executing the function in your application, you must provide a FunctionResponse back to the model to continue the conversation.

FunctionCall properties:

• name: The name of the function to call.
• args: A record of arguments for the function.
• id: The unique identifier for the call.

FunctionResponse properties:

• name: The name of the function being responded to.
• response: A record containing the output of the function.
• id: The ID of the original call.