DOCSEARCHbyAlgolia
Submit Docs

ComfyUI Griptape Nodes

repository·main·Indexed 42 days ago

https://github.com/griptape-ai/comfyui-griptape

A bridge between the Griptape Python Framework and ComfyUI, enabling the integration of LLMs, AI agents, and tools into ComfyUI workflows. Version 2.2.17 supports RAG, parallel tasks, and image generation via Black Forest Labs Flux models. It provides nodes for agent configuration, PDF processing, and audio transcription, with support for local models via Ollama and various paid APIs including OpenAI, Anthropic, and Google Gemini.

Tokens
1.9K
Snippets
8
Records
14
Agent score
4%

What's inside ComfyUI Griptape

  1. Overview of Griptape Node Capabilities

    main

    The ComfyUI Griptape nodes allow you to integrate the Griptape Python Framework into your ComfyUI workflows. Key capabilities include:

    Agent Creation

    Create Agents that can use various models:

    • Local: via Ollama or LM Studio (e.g., Llama 3, Mistral).
    • Paid APIs: OpenAI, Azure OpenAI, Amazon Bedrock, Cohere, Google Gemini, Anthropic Claude, and Hugging Face.

    Agent Control & Tools

    • Behavior: Control personality using Rules and Rulesets.
    • Tools: Give agents access to tools like Calculator, DateTime, WebScraper, FileManager, AudioTranscriptionClient, GriptapeCloudKnowledgeBaseClient, and GriptapeVectorStoreClient.

    Task Execution

    Run specific tasks such as:

    • PromptTask, TextSummaryTask, ToolTask, ToolkitTask.
    • Image/Audio Tasks: PromptImageGenerationTask, ImageQueryTask, VariationImageGenerationTask (Beta), AudioTranscriptionTask, and TextToSpeechTask.

    Image & Audio Generation

    • Image Models: OpenAI, Amazon Bedrock (Stable Diffusion/Titan), and Leonardo.AI.
    • Audio: Transcribe audio and Text-to-Speech (TTS).

  2. Configure Agent behavior with Drivers

    main

    Griptape nodes use a driver-based system to control different aspects of an Agent's behavior. You can use specific nodes to configure:

    • Prompt Driver
    • Image Generation Driver
    • Embedding Driver
    • Vector Store Driver
    • Text to Speech Driver
    • Audio Transcription Driver

  3. Set up local LLMs with Ollama

    main

    To run Griptape nodes using local models instead of paid APIs, use Ollama:

    1. Download and install Ollama.
    2. Pull a model (e.g., Llama 3) by running the following in your terminal:
      ollama run llama3
    3. Once Ollama is running, you can use it as a configuration driver within the Griptape nodes in ComfyUI.
    ollama run llama3

  4. Configure Griptape API Keys in ComfyUI

    main

    For access to paid LLM providers and advanced features, you must configure your API keys within the ComfyUI interface:

    1. Click the Settings button in the ComfyUI Sidebar.
    2. Select the Griptape option.
    3. Locate the specific provider (e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY) and enter your key.

    Note: If an API key is already set in your system environment variables, it will automatically appear in this menu.

    Supported Environment Variable Keys:

    • OPENAI_API_KEY
    • GOOGLE_API_KEY
    • AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY
    • LEONARDO_API_KEY
    • ANTHROPIC_API_KEY
    • VOYAGE_API_KEY
    • HUGGINGFACE_HUB_ACCESS_TOKEN (Note: Only models < 10GB are supported via serverless inference)
    • AZURE_OPENAI_ENDPOINT & AZURE_OPENAI_API_KEY
    • COHERE_API_KEY
    • ELEVEN_LABS_API_KEY
    • GT_CLOUD_API_KEY

  5. Install ComfyUI Griptape Nodes

    main

    To use Griptape nodes within ComfyUI, follow these steps:

    1. Install ComfyUI: Ensure you have ComfyUI installed on your system.
    2. Install Griptape-ComfyUI:
      • Option A (Recommended): ComfyUI Manager
        1. Open ComfyUI and click Manager.
        2. Search for "Griptape".
        3. Find the ComfyUI-Griptape repository and click INSTALL.
      • Option B: Git Clone Navigate to your ComfyUI/custom_nodes directory and run:
        git clone https://github.com/griptape-ai/ComfyUI-Griptape
    3. Install Dependencies: If libraries do not load automatically, manually install the required packages:
      • Using pip:
        pip install "griptape[all]" python-dotenv
      • Using poetry:
        poetry add "griptape[all]" python-dotenv
    4. Restart ComfyUI: After installation, restart ComfyUI. You should see a Griptape menu when right-clicking in the workspace.
    git clone https://github.com/griptape-ai/ComfyUI-Griptape

  6. Resolve Torch issues in StabilityMatrix

    main

    If you use StabilityMatrix to run ComfyUI and encounter errors after installing Griptape, you must manually update your torch installation through the StabilityMatrix interface:

    1. Go to the Packages tab.
    2. On the ComfyUI card, click the vertical ... menu and select Python Packages.
    3. Search for torch and click the - button to uninstall it.
    4. Click the + button to add a new package.
    5. Enter the following string to install the correct versions: torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121
    6. Click OK, wait for completion, and relaunch ComfyUI.
    torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121

  7. Fix Griptape update errors and ImportErrors

    main

    If you encounter an ImportError (e.g., cannot import name '...' from 'griptape.drivers'), it usually means the Griptape library did not update correctly, which can happen when using ComfyUI Manager.

    To resolve this, try the following in order:

    1. Update again via the ComfyUI Manager.
    2. Uninstall and Re-install the Griptape nodes via the ComfyUI Manager.
    3. Manually force an update via the terminal in your ComfyUI directory: python -m pip install griptape -U
    4. If issues persist, contact support on Discord.
    python -m pip install griptape -U

  8. Resolve Torch version conflicts

    main

    Installing Griptape may install a version of torch that conflicts with your existing ComfyUI installation (particularly on Nvidia hardware). If you encounter issues, you may need to uninstall and reinstall torch using the specific CUDA index URL recommended by ComfyUI documentation.

    pip uninstall torch
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121

  12. Implement RAG (Retrieval Augmented Generation)

    main

    The RAG workflow demonstrates how to give agents controlled access to data. This implementation includes:

    • Multiple data sources.
    • Reranking results for better accuracy.
    • Controlling agent responses based on retrieved context.
    [retrieval_augmented_generation.json](retrieval_augmented_generation.json)