DOCSEARCHbyAlgolia
Submit Docs

PyGPT Documentation

repository·master·Indexed 23 days ago

https://github.com/szczyglis-dev/py-gpt

A comprehensive desktop AI assistant for Linux, Windows, and Mac supporting LLMs from OpenAI, Anthropic, Google, xAI, and local models via Ollama and LlamaIndex. PyGPT features multimodal capabilities including image/video generation, vision analysis, speech-to-text, and file-based RAG. It includes a node-based Agents Builder, Python Code Interpreter, and an extensible architecture for developers to implement custom plugins, tools, LLM wrappers, and vector stores.

Tokens
64.4K
Snippets
61
Records
347
Agent score
82%

What's inside PyGPT

  1. Overview of available PyGPT plugins

    master

    PyGPT can be enhanced with a wide variety of plugins to add new features. These plugins allow the model to interact with external services, local hardware, and system resources.

    Key categories of plugins include:

    • Communication & Social: Slack, Telegram, X/Twitter, Facebook, Mailer.
    • Productivity & Cloud: Google (Gmail, Drive, Docs, etc.), GitHub, Bitbucket, Dropbox (via other means), Bitbucket.
    • System & Hardware: System (OS), Files I/O, Mouse and Keyboard, Serial port / USB, Crontab / Task scheduler, Server (SSH/FTP).
    • AI & Intelligence: Code Interpreter (local Python execution), Autonomous Agent (inline), Experts (inline), Vision (inline), Image Generation (inline), Web Search, Wikipedia, Wolfram Alpha.
    • Connectivity & Protocols: MCP (Model Context Protocol), API calls (custom external services), OpenStreetMap.
    • IoT & Automation: Tuya (IoT).
    • Audio/Speech: Audio Input (speech recognition), Audio Output (voice synthesis), Voice Control (inline).

  2. Overview of PyGPT features and capabilities

    master

    PyGPT is an all-in-one desktop AI assistant for Linux, Windows, and Mac. It provides direct interaction with OpenAI models (GPT-5, GPT-4, o1, o3, etc.) via the OpenAI API, but also supports alternative LLMs through LlamaIndex and Ollama (e.g., Llama 3, Mistral, DeepSeek, Google Gemini, Anthropic Claude, and xAI Grok).

    Key capabilities include:

    • Multiple Operation Modes: Chat, Chat with Files (via LlamaIndex), Realtime + audio, Research (Perplexity), Completion, Image/Video generation, Assistants, Experts, Computer use, Agents, and Autonomous Mode.
    • File & Data Interaction: Chat with various file types (txt, pdf, csv, html, md, docx, json, epub, xlsx, xml, etc.), webpages, and GitHub. Includes built-in vector database support and automated embedding.
    • Multimodal Support: Image generation (DALL-E, Gemini, etc.), video generation (Veo3, Sora2), and vision-based image analysis.
    • Audio & Speech: Speech synthesis (Azure, Google, Eleven Labs, OpenAI) and speech recognition (Whisper, Google, Bing).
    • Automation & Tools: Plugin support (Files I/O, Code Interpreter, Web Search, MCP, etc.), Python Code Interpreter, system command execution, and a node-based Agents Builder.
    • Memory: Handles short-term and long-term conversation context.

    Note: To use OpenAI models, you must provide your own OpenAI API key. Local models (via Ollama) do not require API keys.

  3. Overview of PyGPT capabilities and supported models

    master

    PyGPT is an all-in-one desktop AI assistant designed for Linux, Windows (10/11), and Mac. It provides a local interface for interacting with various Large Language Models (LLMs) using your own API keys or local model providers.

    Supported Model Providers

    • OpenAI: Direct interaction via OpenAI API (e.g., GPT-5, GPT-4, o1, o3, o4).
    • Alternative LLMs: Supports Google Gemini, Anthropic Claude, Perplexity / Sonar, and xAI Grok.
    • Local Models: Via Ollama (e.g., Llama 3, Mistral, DeepSeek V3/R1, gpt-oss, Bielik) or any model accessible through LlamaIndex.

    Key Functionalities

    • Operation Modes: Includes Chat, Chat with Files, Realtime + audio, Research (Perplexity), Completion, Image/Video generation, Assistants, Experts, Computer use, Agents, and Autonomous Mode.
    • Multimodal Support: Image generation (DALL-E, Imagen, etc.), video generation (Veo3, Sora2), and image analysis via vision models.
    • Data & Files: Integrated LlamaIndex support for chatting with files (txt, pdf, csv, html, md, docx, json, epub, xlsx, xml, webpages, etc.) and built-in vector database support for automated embedding.
    • Audio: Speech synthesis (Azure, Google, Eleven Labs, OpenAI) and speech recognition (Whisper, Google, Microsoft).
    • Automation & Tools: Python Code Interpreter, filesystem I/O, system command execution, web search (DuckDuckGo, Google, Bing), and a node-based Agents Builder.

  4. Overview of PyGPT Plugins

    master

    PyGPT can be enhanced with various plugins to add new capabilities. These plugins allow the model to interact with external services, manage local files, execute code, and control hardware.

    Key categories of plugins include:

    • Communication & Social: Bitbucket, Facebook, GitHub, Slack, Telegram, X/Twitter.
    • Productivity & Tools: API calls, Code Interpreter, Custom Commands, Files I/O, Google (Gmail, Drive, etc.), Mailer, Web Search, Wikipedia, Wolfram Alpha.
    • Media & Interaction: Audio Input (speech recognition), Audio Output (TTS), Image Generation, Vision, Voice Control.
    • System & Automation: Crontab/Task scheduler, Mouse and Keyboard, Serial port/USB, System (OS), Autonomous Agent (inline).
    • Advanced Integration: MCP (Model Context Protocol), LlamaIndex (Chat with Files), Server (SSH/FTP), Tuya (IoT).

  6. How Vision (inline) works

    master

    The Vision (inline) plugin enables vision capabilities across all chat modes (e.g., standard Chat mode). When enabled, the assistant can automatically switch to vision processing when an image attachment or vision capture is detected.

    Behavior

    • When you provide image content, a + Vision special checkbox appears at the bottom of the Chat window and is automatically enabled.
    • The vision model is used while the checkbox is enabled.
    • To exit vision mode after analysis, manually uncheck the checkbox.
    • The checkbox will re-activate automatically when the next image is provided.

    Options

    • Model (model): The vision model to use. Default: gpt-4-vision-preview.
    • Prompt (prompt): The prompt used for vision mode. This is appended to or replaces the current system prompt.
    • Replace prompt (replace_prompt): If True, the vision prompt replaces the whole system prompt instead of appending to it. Default: False.
    • Tool: capturing images from camera (cmd.camera_capture): Allows the capture command. Requires + Tools to be enabled. Default: False.
    • Tool: making screenshots (cmd.make_screenshot): Allows the screenshot command. Requires + Tools to be enabled. Default: False.

  7. Integrate with Slack via the Slack plugin

    master

    The Slack plugin uses the Slack Web API and OAuth2 to allow interaction with Slack workspaces.

    Capabilities:

    • Retrieve user lists.
    • List all conversations.
    • Access conversation history and replies.
    • Open a conversation.
    • Post messages in a chat.
    • Delete chat messages.
    • Upload files to Slack.

  8. Control mouse and keyboard with the Mouse And Keyboard plugin

    master

    WARNING: Use this plugin with caution. Allowing all options gives the model full control over your mouse and keyboard.

    This plugin enables the model to perform UI tasks like "open notepad, type something in it".

    Capabilities:

    • Get and control mouse cursor position.
    • Control mouse clicks and scrolling.
    • Control the keyboard (pressing keys, typing text).
    • Take screenshots.

  9. Configure Advanced Audio Input (Magic Words and Listening)

    master

    When advanced mode is enabled in the Audio Input plugin, you can fine-tune how the application listens to your microphone. This is useful for hands-free operation using 'Magic Words'.

    Magic Word Configuration:

    • magic_word: Enables activation only when a specific word is spoken. (Default: False)
    • magic_words: A list of words that trigger listening (e.g., OK, Okay, Hey GPT). (Default: OK, Okay, Hey GPT, OK GPT)
    • magic_word_reset: If True, the magic word must be spoken again for every new input. (Default: True)
    • magic_word_timeout: Seconds to wait for the magic word. (Default: 1)

    Listening Behavior:

    • continuous_listen: Experimental. If enabled, the app does not stop listening after one input. (Default: False)
    • stop_words: Words that terminate the listening process (e.g., stop, exit, quit).
    • prefix_words: Words that must start every phrase to be processed (e.g., OK, GPT).
    • min_energy: The threshold multiplier above noise level to start recording. (Default: 1.3)

  10. What are Presets in PyGPT?

    master

    Presets in PyGPT are templates used to store and quickly apply different configurations. They allow you to manage various interaction styles without manually reconfiguring settings every time.

    A preset encapsulates the following configuration parameters:

    • Mode: The interaction type (e.g., chat, completion, or image generation).
    • System Prompt: The initial instructions provided to the AI.
    • AI Name: The assigned name for the AI model.
    • Username: The username for the session.
    • Temperature: A setting that controls response creativity. A warmer (higher) temperature allows for more creative responses, while a cooler (lower) temperature encourages more predictable, stable replies.

    Presets are compatible with models accessed via the OpenAI API or LlamaIndex and can be used across various modes.

  11. Schedule tasks with the Crontab / Task scheduler plugin

    master
    The Crontab plugin allows you to schedule tasks or prompts to be sent at specific times using standard cron-based syntax. This is useful for automating recurring prompts or scheduled actions.

  12. Choose an attachment context mode

    master

    When uploading attachments, you can select one of three modes to determine how the model interacts with the file content:

    • Full context: The entire content of the file is attached to the user's prompt. This provides the best results but can result in very high token usage for large files.
    • RAG: The attachment is indexed in a vector database and queried in real-time using LlamaIndex. This is token-efficient but may not provide 1:1 access to the full content.
    • Summary: An additional query is generated in the background by a separate model to summarize the attachment content and return the required information to the main model.

    Note on Images: By default, images are analyzed in real-time using a vision model. To use them as additional context (via RAG/Summary), you must enable "Allow images as additional context" in Files and attachments settings.