DOCSEARCHbyAlgolia
Submit Docs

Mai - Meta Glasses API for Messenger

repository·main·Indexed 20 days ago

https://github.com/dcrebbin/meta-glasses-api

A browser extension that bridges Meta Rayban Smart Glasses with LLM providers (OpenAI, Claude, Perplexity, Google, DeepSeek, and xAI) via Messenger. It enables voice-controlled AI interactions, video call monitoring with vision requests, and text-to-speech responses using providers like OpenAI, ElevenLabs, and Minimax.

Tokens
5.9K
Snippets
23
Records
30
Agent score
72%

What's inside meta-glasses-api

  1. Overview of Mai - Meta Glasses API for Messenger

    main

    Mai is a browser extension designed to add custom AI bots to Messenger. It enables integration with Meta Rayban Smart Glasses or the standalone Messenger app, allowing users to trigger AI workflows via voice commands or direct messaging.

    Key capabilities include:

    • Voice Commands: Using "Hey Meta send a photo to [target]" or "Hey Meta send a message to [target]" to trigger AI actions.
    • Video Monitoring: Sending screenshots of video calls to providers like ChatGPT or Claude and logging the output.
    • AI Integration: Connecting Messenger interactions to providers like OpenAI, Perplexity, or Claude.

  2. How Chat Monitoring works

    main

    Once the extension is running and the group chat is set up:

    1. On your alternative account, open the newly created group chat at messenger.com or facebook.com/messages.
    2. Start monitoring the chat via the extension.
    3. Triggering: Every new message or image request sent to the group will be forwarded to your chosen provider (ChatGPT, Claude, etc.).
    4. Response: The AI's output will be sent back to the chat.
    5. Audio (Optional): If enabled, the extension will generate an audio clip of the output using OpenAI and send it back to you.

  3. Requirements for Mai

    main

    To use this project, you need the following:

    1. Hardware/App: Meta Rayban Smart Glasses or the standalone Messenger app.
    2. API Keys: An API key from a provider such as OpenAI, Perplexity, or Claude.
    3. Account: An alternative Facebook/Messenger account to act as the bot/group interface.

  4. How to trick Meta Glasses into using custom AI bots

    main

    To allow voice commands like "Hey Meta send a message to ChatGPT", you must create a group chat that the Meta Glasses recognize as a valid contact/group. Follow these steps:

    1. Create a Group: Create a Messenger group chat with at least 2 other Facebook accounts.
    2. Cleanup: Remove the accounts you are not going to use so only your target 'bot' account remains.
    3. Branding:
      • Change the name of the chat to your desired AI name (e.g., "ChatGPT").
      • Update the group chat photo to make it feel legitimate.
    4. Identity: Set a nickname for your alternative bot account.
    5. Resync:
      • Go to the Meta view app within the communications section.
      • In Messenger, disconnect and then reconnect your Messenger account. This resyncs chats and friends, making the new group chat visible to the Meta Glasses for voice commands.

  5. Install and Setup the Browser Extension

    main

    Follow these steps to build and run the extension locally using bun:

    1. Install dependencies:
      bun install
    2. Run the development command for your preferred browser:
      bun run dev:chrome
# or
bun run dev:brave
# or
bun run dev:firefox
    3. The extension should build, run, and open automatically.
    4. Configure your API keys in the extension's API settings tab.
    5. Sign into your alternative Facebook account, navigate to facebook.com/messages/t, and begin monitoring the conversation.
    bun install
bun run dev:chrome

  6. Configure Vite and Tailwind CSS integration

    main

    The vite property allows for custom Vite configuration via a function. This is used to integrate plugins and manage build behavior:

    • Plugins: Uses svgr() for handling SVG files as React components and tailwindcss() for styling.
    • Server: Enables Hot Module Replacement (hmr: true).
    • Build: Configures sourcemap generation based on the environment. Sourcemaps are enabled only when process.env.NODE_ENV is set to development.
    vite: () =>
  ({
    plugins: [svgr(), tailwindcss()],
    server: {
      hmr: true,
    },
    build: {
      sourcemap: process.env.NODE_ENV === "development",
    },
  } as WxtViteConfig),

  7. Configure Brave browser binary path

    main

    The webExt.binaries configuration specifies the file path to the Brave browser executable, which is used for automated testing or development workflows. The path is determined based on the operating system:

    • Windows: C:\Program Files\BraveSoftware\Brave-Browser\Application\brave.exe
    • macOS: /Applications/Brave Browser.app/Contents/MacOS/Brave Browser
    webExt: {
  binaries: {
    brave: 
      process.platform === "win32"
        ? "C:\\Program Files\\BraveSoftware\\Brave-Browser\\Application\\brave.exe"
        : "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser",
  },
},

  8. Configure WXT directory and build settings

    main

    The WXT configuration allows you to define the project structure and build output locations:

    • srcDir: The directory containing the source code (set to src).
    • entrypointsDir: The directory where extension entrypoints are located (set to app).
    • outDir: The directory where the final build will be generated (set to build).
    • modules: An array of WXT modules to include. This project uses @wxt-dev/module-react and @wxt-dev/auto-icons.
    srcDir: "src",
entrypointsDir: "app",
outDir: "build",
modules: ["@wxt-dev/module-react", "@wxt-dev/auto-icons"],

  9. Configure the WXT extension manifest

    main

    The manifest object in wxt.config.ts defines the core properties of the browser extension. It specifies the extension's name and description (using localization placeholders), the default locale, and the required permissions for the extension to function.

    Key configuration keys:

    • name: The name of the extension (supports __MSG_extensionName__ for localization).
    • description: The description of the extension (supports __MSG_extensionDescription__ for localization).
    • default_locale: The default language for the extension (e.g., en).
    • permissions: An array of required browser API permissions. This project requires storage, sidePanel, and scripting.
    • host_permissions: An array of URL patterns the extension is allowed to access. This project is configured with <all_urls> to allow broad access.
    manifest: {
  name: "__MSG_extensionName__",
  description: "__MSG_extensionDescription__",
  default_locale: "en",
  permissions: ["storage", "sidePanel", "scripting"],
  host_permissions: ["<all_urls>"],
},

  10. Troubleshoot missing API keys

    main

    If an AI or TTS request fails, check the error message. The library throws specific errors if API keys are missing for the selected provider:

    • No API key found for provider: [PROVIDER_NAME]. Please set it in the toolbar.

    Ensure that the keys for your chosen Provider (for text/vision) or TTSProvider (for speech) are correctly saved in the application's storage via the toolbar interface.

  11. Use the useStorage hook for reactive state

    main

    The useStorage hook provides a React-friendly way to interact with local storage. It automatically watches for changes in the storage item and updates the local state.

    It returns an object containing:

    • data: The current value of the storage item (or the fallback value if null).
    • set(value): An async function to update the storage value.
    • remove(): An async function to clear the storage value.

    This is the recommended way to consume storage data within React components to ensure the UI stays in sync with the underlying data.

    import { useStorage, StorageKey } from "./storage";

const SettingsComponent = () => {
  const { data, set, remove } = useStorage(StorageKey.SETTINGS);

  const updateQuality = (newQuality: number) => {
    if (data) {
      set({ ...data, imageQuality: newQuality });
    }
  };

  return (
    <div>
      <p>Current Quality: {data?.imageQuality}</p>
      <button onClick={() => updateQuality(0.8)}>Set High Quality</button>
      <button onClick={remove}>Reset Settings</button>
    </div>
  );
};

  12. Manage application settings with useSettingsStore

    main

    The useSettingsStore hook (built with Zustand) provides access to the application's configuration and a method to update them. Settings are persisted to storage automatically via a subscription.

    Settings Schema

    The settings object contains the following keys:

    • imageQuality: number
    • provider: Provider (an enum/type defining the service provider)
    • useTTS: boolean
    • model: An object mapping Provider keys to string model names (e.g., { [providerName]: modelName })
    • ttsModel: string
    • videoCaptureInterval: number
    • isMaiUIVisible: boolean
    • isConversationSidebarVisible: boolean
    import { useSettingsStore } from "./path-to-store";

// Accessing settings
const settings = useSettingsStore((state) => state.settings);

// Updating settings
useSettingsStore.getState().setSettings({
  imageQuality: 90,
  useTTS: true,
  model: {
    // Note: updating model merges with existing provider models
    someProvider: "new-model-name"
  }
});