DOCSEARCHbyAlgolia
Submit Docs

vimGPT Documentation

repository·main·Indexed 25 days ago

https://github.com/ishan0102/vimgpt

A tool that provides multimodal models, such as GPT-4V, an interface to interact with the web. It utilizes Vimium extension keyboard-based navigation logic and vision capabilities to browse and interact with web elements instead of raw DOM text. Supports text-based objectives and a voice mode via the --voice flag using WhisperMic.

Tokens
335
Snippets
2
Records
4
Agent score
33%

What's inside vimGPT

  1. Install and run vimGPT

    main

    To use vimGPT, you need to install the Python dependencies, set up the local Vimium extension for Playwright, and then run the main script.

    1. Install Python requirements: pip install -r requirements.txt
    2. Download Vimium locally (this manually loads the extension when running Playwright): ./setup.sh
    3. Run the script: python main.py
    npip install -r requirements.txt
./setup.sh
python main.py

  2. Use Voice Mode in vimGPT

    main

    You can engage with the browser using voice commands by enabling Voice Mode. Simply state your objective, and vimGPT will perform actions in real-time based on your speech.

    python main.py --voice

  3. Use the main() function for automation

    main
    The main(voice_mode) function serves as the primary execution loop for the browser automation. It initializes the Vimbot driver, navigates to a starting URL (Google by default), captures the objective (via voice or text input), and enters a loop where it captures screenshots and uses vision.get_actions to determine and perform browser actions until the objective is met.

  4. Run vimgpt via CLI

    main
    You can run the vimgpt automation tool from the command line. By default, it prompts for a text-based objective. You can enable voice input mode using the --voice flag, which uses WhisperMic to listen for your command.