DOCSEARCHbyAlgolia
Submit Docs

USB-Uncensored-LLM Documentation

repository·main·Indexed 23 days ago

https://github.com/techjarves/usb-uncensored-llm

A portable, air-gapped Local AI environment designed to run uncensored large language models (LLMs) from a USB drive or local disk. It features a zero-dependency, cross-platform setup supporting Windows, macOS, Linux, and Android (via Termux) using a unified 'Shared' data architecture for models and conversation history.

Tokens
2K
Snippets
2
Records
9
Agent score
73%

What's inside USB-Uncensored-LLM

  1. Understand the USB-Uncensored-LLM Folder Architecture

    main

    The project uses a Shared volume system to allow cross-platform interoperability. This allows you to download large AI models once and use them across Windows, macOS, Linux, and Android without duplication.

    [Portable USB Drive]
 ├── 📁 Android    # Native Android (Termux) installers & launchers
 ├── 📁 Linux      # Native Ubuntu/Debian offline installers & launchers
 ├── 📁 Mac        # Native macOS offline installers & launchers
 ├── 📁 Windows    # Native Windows offline automatic UI menus
 └── 📁 Shared     # Unified Data System
      ├── 📁 bin         (Holds isolated executables: ollama-windows.exe, ollama-darwin...)
      ├── 📁 chat_data   (Houses cross-platform persistent conversation history)
      ├── 📁 models      (HuggingFace GGUF Weights & local database mapping)
      └── 📁 python      (Isolated portable python environment)

  2. Quick Start Guide for USB-Uncensored-LLM

    main

    USB-Uncensored-LLM is a portable, air-gapped AI environment. Follow these three steps to set up the engine, download models, and launch the interface.

    1. Initialize the Engine

    Navigate to the folder corresponding to your Operating System and run the installation script to download the ~50MB execution engine to Shared/bin.

    • Windows: Double-click Windows/install.bat
    • macOS: Open Terminal, drag in Mac/install.command, and press Enter.
    • Linux: Run bash Linux/install.sh
    • Android: Open Termux and run bash Android/install.sh

    2. Download AI Models

    It is highly recommended to use a Windows PC (Windows/install.bat) for this step, as it provides an interactive terminal-based catalog of curated, uncensored GGUF models. Note: If you cannot use Windows, manually place .gguf weights from HuggingFace into the Shared/models folder.

    3. Launch

    Run the start script for your OS to spin up the engine and automatically open the Chat UI in your web browser.

    • Windows: Windows/start-fast-chat.bat
    • macOS: Mac/start.command
    • Linux: bash Linux/start.sh
    • Android: bash Android/start.sh
    # Windows Example
Windows/install.bat
Windows/start-fast-chat.bat

# Linux Example
bash Linux/install.sh
bash Linux/start.sh

# macOS Example
Mac/install.command
Mac/start.command

# Android Example
bash Android/install.sh
bash Android/start.sh

  3. Access the AI via LAN Mobile Access

    main

    You can use the heavy AI models running on your PC from a mobile device on the same WiFi network:

    1. Ensure your PC (running the start script) and your phone are on the same WiFi network.
    2. Look at the terminal window on your PC; it will display a Network Access IP address (e.g., http://192.168.1.15:3333).
    3. Enter that URL into your mobile browser (Safari, Chrome, etc.).

    Troubleshooting: If the page fails to load, ensure your Windows Firewall allows incoming connections on port 3333.*

  4. Install USB-Uncensored-LLM on Android via Termux

    main

    To run the AI engine directly on an Android device, follow these requirements and steps:

    Requirements

    • Termux installed from F-Droid (do not use the Play Store version).
    • Hardware: ARM64 processor and at least 6 GB RAM (8 GB+ recommended).
    • Connectivity: WiFi or mobile data for initial setup.

    Setup Steps

    1. Copy the project folder to your Android device.
    2. Open Termux and navigate to the project folder.
    3. Run the installer: bash Android/install.sh.
    4. Select your model (Gemma 2 2B is recommended for most devices).
    5. Keep Termux in the foreground while downloads complete.

    Launching

    Run the following command to start the engine and open the browser UI:

    bash Android/start.sh

    Performance Tips

    • Run termux-wake-lock before starting to prevent Android from killing the process.
    • Use the 2B model if your device has less than 12 GB of RAM.
    • Plug in your charger, as LLM inference is battery-intensive.
    bash Android/install.sh
bash Android/start.sh

  5. Launch the AI Engine on Android via start.sh

    main

    The Android/start.sh script is the primary entrypoint for running the portable AI environment on Android using Termux. It automates the following workflow:

    1. Environment Validation: Ensures the script is running within Termux and that the native engine (llama-server-android) has been previously compiled via Android/install.sh.
    2. Model Selection: Automatically selects the first .gguf file found in the Shared/models/ directory.
    3. Engine Initialization:
      • Acquires a Termux wakelock to prevent Android from killing the process during sleep.
      • Starts the llama-server-android binary with a context limit of 2048 to preserve mobile RAM.
      • Monitors the engine startup via http://127.0.0.1:8080/v1/models (waits up to 90 seconds).
    4. UI Selection: Prompts the user to choose between:
      • USB FastChat UI (Port 3333): A dark-mode interface with auto-save capabilities.
      • Llama.cpp Default UI (Port 8080): The raw developer interface.
    5. Proxy Server: Launches a Python-based proxy (chat_server.py) to facilitate the FastChat UI and manage chat history.

    Prerequisites:

    • Must be run inside Termux.
    • Native engine must be installed via bash Android/install.sh.
    • At least one .gguf model must exist in Shared/models/.
    • Python must be installed (apt install python -y).

  6. Launch the Portable AI engine on Linux

    main

    Use the start.sh script located in the Linux/ directory to launch the AI engine and the browser-based chat UI. This script is designed for portability, ensuring that the Ollama runtime, models, and chat history are all stored on the USB drive rather than the host system.

    Prerequisites:

    • The AI engine must be installed. If you receive an error that the engine is missing, run bash install.sh within the same Linux folder first.
    • A Python interpreter (python3 or python) must be installed on the host Linux system to run the chat server.

    Behavior:

    1. It checks for the existence of the ollama-linux binary and llama-server library in the USB's Shared/ directory.
    2. It starts the Ollama engine in serve mode if it is not already running on 127.0.0.1:11434.
    3. It waits up to 60 seconds for the engine to initialize.
    4. It launches the Python-based chat server (chat_server.py).
    5. Press Ctrl+C to shut down the chat server and the Ollama engine process.

  7. Troubleshoot USB-Uncensored-LLM issues

    main

    Common issues and solutions:

    • Windows script instantly closes: This is often caused by legacy Windows App Execution Aliases. To fix this, run the script via a Command Prompt (CMD) or right-click the .bat file and select 'Run as Administrator'.
    • 'Ollama Engine Not Found' error: You are trying to run the start script before the install script has finished. Run the OS-specific installer first.
    • Slow generation speeds: Your model is likely too large for your system's RAM. Re-run install.bat and select the Gemma 2 2B Abliterated model, which is optimized for speed and lower memory usage.

  8. Troubleshoot Android Engine Startup Failures

    main

    If the AI engine fails to start within 90 seconds, the script will exit. This is typically caused by insufficient device RAM or the selected .gguf model being too large for the mobile hardware.

    To diagnose the failure, check the engine logs located at: Shared/llama-server.log

  9. Environment variables used by the Linux AI engine

    main

    The start.sh script configures several environment variables to ensure the Ollama engine remains portable and contained within the USB drive's Shared directory. If you are manually invoking the engine, be aware of these keys:

    • OLLAMA_MODELS: Points to the directory where AI models are stored ($SHARED_DIR/models/ollama_data).
    • OLLAMA_HOME: The root directory for the Ollama runtime ($SHARED_DIR/.ollama-runtime).
    • OLLAMA_TMPDIR: The temporary directory for Ollama operations ($SHARED_DIR/.ollama-runtime/tmp).
    • OLLAMA_ORIGINS: Set to * to allow cross-origin requests.
    • OLLAMA_HOST: Set to 127.0.0.1:11434 for local communication.