DOCSEARCHbyAlgolia
Submit Docs

Sagittarius Documentation

repository·main·Indexed 20 days ago

https://github.com/gregsadetsky/sagittarius

A voice and video exploration tool for interacting with GPT-4 (gpt-4-vision-preview) and Gemini models. The tool features multimodal capabilities including camera and microphone integration, speech recognition via webkitSpeechRecognition, and text-to-speech output using the Web Speech API.

Tokens
4.6K
Snippets
15
Records
16
Agent score
72%

What's inside Sagittarius

  1. Build and run Sagittarius locally

    main

    To run the Sagittarius GPT-4/Gemini Voice/Video Exploration Tool on your local machine, follow these steps:

    1. Clone the repository and navigate into the directory.
    2. Create a .env file by duplicating .env.example.
    3. Configure your API keys in the .env file:
      • Set VITE_OPENAI_KEY with your OpenAI API key (requires access to the gpt-4-vision-preview model).
      • Optionally set VITE_GEMINI_KEY with your Gemini API key.
    4. Install dependencies using npm install.
    5. Start the development server using npm run dev.

    The application will be available at http://localhost:5173.

    Note: For the best experience with in-browser speech recognition, use Google Chrome.

    # Setup steps
cp .env.example .env
# Edit .env to add VITE_OPENAI_KEY or VITE_GEMINI_KEY

npm install
npm run dev

  3. Make an OpenAI request with `makeOpenAIRequest()`

    main

    The makeOpenAIRequest function facilitates a multimodal interaction with OpenAI's gpt-4-vision-preview model. It accepts a text prompt and an image URL, sends them to OpenAI using a streaming chat completion, and manages the lifecycle of speech output.

    Key behaviors:

    • System Prompt: It uses a hardcoded OPEN_AI_SYSTEM_PROMPT designed for concise, visual-focused responses.
    • Visual Debugging: It automatically attempts to render the text and image to DOM elements with IDs #debugImages.
    • Speech Management: It calls stopDictation() to prevent feedback loops and uses the provided speech object to stream the model's response as audio. It handles word-boundary logic to ensure smoother speech synthesis.
    • UI Updates: It calls updatePromptOutput to display the streaming text in the UI.

    Parameters:

    • text (string): The user's text prompt.
    • imageUrl (string): The URL of the image to be processed.
    • apiKey (string, optional): The OpenAI API key. Defaults to the value of import.meta.env.VITE_OPENAI_KEY.
    • speech (Speech): An object implementing the Speech interface used to manage audio streaming and state.
    import { makeOpenAIRequest } from './openai';

// Assuming 'speech' is an instance of a class implementing the Speech interface
await makeOpenAIRequest(
  "What am I holding?",
  "https://example.com/image.jpg",
  "your-api-key",
  speech
);

  5. Start dictation with startDictation()

    main

    Initialize and start speech recognition using startDictation(). This function sets up a webkitSpeechRecognition instance configured for continuous listening and interim results. It requires a language code (e.g., 'en-US') and a callback function that is invoked whenever new final transcriptions are available.

    Note: The function manages its own internal state for the recognition instance and handles automatic restarts if a no-speech error occurs.

    import { startDictation } from './dictation';

startDictation('en-US', (message: string) => {
  console.log('Received transcription:', message);
});

  6. Update the prompt output UI

    main

    The updatePromptOutput function appends new text messages to the #promptOutput element in the DOM. It automatically scrolls the container to the bottom to ensure the latest text is visible.

    Parameters:

    • newMessage (string): The text to append.
    • dontAddNewLine (boolean, optional): If true, the function will not append a <br> tag after the message.
    updatePromptOutput("Hello world!");
// Or without a new line:
updatePromptOutput("Hello world!", true);

  7. Use the Speech class for text-to-speech output

    main

    The Speech class provides an interface for converting text to spoken audio using the Web Speech API. It supports both single messages and a streaming mode for continuous speech.

    Single Message

    Use the speak(message: string) method to play a single piece of text. This method returns a Promise<void> that resolves when the utterance has finished playing.

    const speech = new Speech();
await speech.speak("Hello, world!");

  8. Use makeGeminiRequest to process visual and text prompts

    main

    The makeGeminiRequest function orchestrates a multimodal request to the Gemini Pro Vision model. It takes a text prompt and a base64-encoded image URL, sends them to Google's Generative AI, and then performs side effects: it stops the current dictation, updates the UI with the response, and speaks the response aloud using the provided speech object.

    Parameters:

    • text: The user's text prompt.
    • imageUrl: A base64-encoded data URI in the format data:<mimeType>;base64,<data>.
    • apiKey: Your Google Generative AI API key.
    • speech: An object implementing the Speech interface, used to call .speak(content).

    Behavior:

    1. It replaces {{USER_PROMPT}} in a predefined system prompt with your text.
    2. It parses the imageUrl to extract the mimeType and the raw base64 data.
    3. It calls the gemini-pro-vision model.
    4. Upon success, it calls stopDictation(), updatePromptOutput(content), and speech.speak(content).
    5. It returns the generated text content.
    import { makeGeminiRequest } from './gemini';

// Assuming 'speech' is an implementation of the Speech interface
const content = await makeGeminiRequest(
  "What am I holding?",
  "data:image/jpeg;base64,/9j/4AAQSkZJRg...",
  "YOUR_GEMINI_API_KEY",
  speech
);

  10. Stop the camera with stopCamera()

    main

    Use stopCamera() to terminate the active camera and microphone streams. This function retrieves the MediaStream from the existing <video> element, iterates through all tracks (audio and video), and calls .stop() on each. Finally, it clears the srcObject of the video element to release the stream.

    import { stopCamera } from './camera';

stopCamera();

  11. Use streaming speech with addToStream()

    main

    The Speech class includes a streaming mechanism to queue multiple messages and play them sequentially without gaps. This is useful for continuous audio output.

    Workflow

    1. Call startStream() to reset the internal queue.
    2. Call addToStream(message: string) to add a new message to the queue. This will trigger the playback of the current queue if the system is not already speaking.
    3. Use speakStreamIsDone() to check if all queued messages have been spoken and the system is idle.

    Note: addToStream joins messages in the current queue with a space before speaking.

    const speech = new Speech();

speech.startStream();

speech.addToStream("First part of the message.");
speech.addToStream("Second part of the message.");

// Check if the stream has finished playing everything
if (speech.speakStreamIsDone()) {
  console.log("All messages spoken.");
}

  12. Start the camera with startCamera()

    main

    Use startCamera() to request access to the user's microphone and camera. Once permission is granted, the video stream is piped into a <video> element and continuously drawn onto a <canvas> element using requestAnimationFrame.

    Requirements:

    • A <video> element must exist in the DOM.
    • A <canvas> element must exist in the DOM.

    Error Handling: If the camera access fails (e.g., permission denied), the function will trigger a browser confirm() dialog asking the user if they want to reload the page.

    import { startCamera } from './camera';

// Ensure <video> and <canvas> are in your HTML before calling
startCamera();