DOCSEARCHbyAlgolia
Submit Docs

servicestack/llms

repository·main·Indexed 20 days ago

https://github.com/servicestack/llms

A lightweight, privacy-focused interface (CLI, API, and Web UI) for interacting with multiple LLMs. It features offline capability, local data storage, and an extension-based architecture allowing for custom builds. The system supports OpenAI-compatible providers, custom configuration via llms.json and providers.json, and specialized extensions such as an Agent Browser Extension for automated browser scripting using agent-browser.

Tokens
88.1K
Snippets
214
Records
359
Agent score
69%

What's inside servicestack-llms

  1. Overview of the Calculator tool

    main
    The Calculator is a mathematical expression evaluator designed for both web-based interaction and safe computation. It provides a rich UI for manual input and supports complex mathematical and statistical operations by leveraging Python's math library logic through a secure Abstract Syntax Tree (AST) evaluator rather than unsafe eval() calls.

  2. Overview of the Projects Extension

    main

    The Projects extension for llms.py provides a workspace management system. It allows you to create dedicated project folders where AI Agents are permitted to read from and write to.

    Key Features:

    • Workspace Sandboxing: Restricts AI Agent filesystem access exclusively to the active project folder.
    • Automatic Folder Creation: Project directories are automatically created on disk when you save a new project.
    • Auto Kebab-Case Folders: Folder names are automatically generated in kebab-case from the project name (though this can be manually overridden).
    • Relative Publish Paths: You can specify a publish output directory as a relative path (e.g., dist) which is combined with the project folder path.

  3. Overview of llms.py

    main
    llms.py is a lightweight CLI, API, and ChatGPT-like alternative to Open WebUI. It is designed for accessing multiple Large Language Models (LLMs) entirely offline, ensuring all data remains private by using browser storage for data persistence.

  4. Overview of the Gallery Extension

    main

    The Gallery extension provides a centralized interface for browsing, managing, and navigating assets generated by AI models or uploaded by the user. It automatically intercepts files saved to the LLMs cache and maintains their metadata in a local SQLite database for fast searching and filtering.

    Supported Assets:

    • Generated Images: (e.g., DALL-E, Stable Diffusion)
    • Generated Audio: (e.g., text-to-speech or audio models)
    • Uploaded Files: Any files uploaded via the UI.

  6. Use the KaTeX Extension for LaTeX math rendering

    main
    The KaTeX Extension enables high-performance rendering of LaTeX math expressions in AI responses. It automatically integrates with the marked markdown parser to detect and render math equations in both inline and block formats. It handles the necessary frontend assets, including loading katex.min.mjs via Import Maps and injecting katex.min.css for styling.

  7. Use multimodal function responses with Gemini 3

    main

    For Gemini 3 series models, you can include multimodal content (images or documents) directly within a functionResponse. This allows the model to process visual or textual data from an external tool to produce a more informed response in the next turn.

    Supported MIME types

    • Images: image/png, image/jpeg, image/webp
    • Documents: application/pdf, text/plain

    Implementation Pattern

    1. Include Multimodal Data: Nest one or more parts containing inlineData within the functionResponse part.
    2. Assign a Display Name: Each multimodal part must contain a unique displayName.
    3. Reference via JSON Reference: Within the structured response field of the functionResponse, you can reference the multimodal part using the JSON reference format: {"$ref": "<displayName>"}. The model will substitute this reference with the actual content.

    Note: Each displayName can only be referenced once within the structured response field.

    {
  "functionResponse": {
    "name": "get_image",
    "response": {
      "image_ref": { "$ref": "instrument.jpg" }
    },
    "parts": [
      {
        "inlineData": {
          "displayName": "instrument.jpg",
          "mimeType": "image/jpeg",
          "data": "<base64_data>"
        }
      }
    ]
  }
}

  8. Understand Credentials password storage and security

    main

    Passwords are not stored in plain text. They are hashed using SHA-256 with a unique 16-byte random salt. The stored format in users.json is salt:hex_digest.

    Session Management:

    • Storage: In-memory with persistence to ~/.llms/credentials/sessions/.
    • Expiry: 30 days.
    • Persistence: Sessions survive server restarts.
    • Token: Stored in an HTTP-only cookie named llms-token.
    • Invalidation: Locking or deleting a user immediately invalidates all their active sessions.

  9. Select active skills for a conversation

    main

    The skills panel allows you to control which skills Claude can access using three modes:

    • All Skills (Green): All available skills are accessible. Claude decides which are relevant. Best for general conversations.
    • No Skills (Purple): No skills are loaded. Claude uses base capabilities only. Best for simple tasks where you want to avoid skill interference.
    • Custom Selection (Blue): Manually toggle specific skills on or off. Best for focusing on specific domains or troubleshooting.

  10. Define Server Tools using JSON Schema

    main

    Server Tools are defined in a server-tools.json file using standard JSON Schema format. The schema serves two purposes:

    1. UI Generation: The frontend parses the schema to automatically build configuration interfaces like inputs, dropdowns, switches, and sliders.
    2. Tool Definition: It generates the JSON definition used to communicate tool capabilities to the LLM during chat or agentic runs.

    To define a tool, create an object in a JSON array within server-tools.json. Use the ui property within schema definitions to specify how elements should be rendered (e.g., "widget": "select" or "widget": "hidden").

    [
    {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "$id": "tool://openrouter/web_search",
        "title": "OpenRouter Web Search Tool",
        "description": "Configuration schema for the OpenRouter web_search server tool",
        "type": "object",
        "properties": {
            "type": {
                "type": "string",
                "const": "openrouter:web_search",
                "title": "Tool Type",
                "ui": {
                    "widget": "hidden"
                }
            },
            "parameters": {
                "type": "object",
                "title": "Search Parameters",
                "properties": {
                    "engine": {
                        "type": "string",
                        "title": "Search Engine",
                        "default": "auto",
                        "enum": ["auto", "native", "exa", "firecrawl", "parallel", "perplexity"],
                        "ui": {
                            "widget": "select"
                        }
                    },
                    "max_results": {
                        "type": "integer",
                        "title": "Max Results",
                        "default": 5,
                        "minimum": 1,
                        "maximum": 25,
                        "ui": {
                            "widget": "number"
                        }
                    }
                }
            }
        }
    }
]

  11. Manage shared styles with a Typst library

    main

    To maintain a consistent brand or "house style" across multiple templates, use a shared library file (typically lib.typ).

    Every template should start with an import statement to access these shared styles, helpers, and configurations:

    #import "lib.typ": *

#let data = load-data("invoice.json")
#show: theme

    By editing lib.typ once, you can update fonts, colors, logos, and layout helpers (like money() or title-block()) across all templates that import it.

    #import "lib.typ": *

#let data = load-data("invoice.json")
#show: theme

  12. How llms.py extensibility works

    main

    The llms.py architecture is based on a lean core and an extension-based model. Most major features are encapsulated in separate extension folders.

    Key Concepts:

    • Lean Core: The central logic is kept minimal to ensure fast startup and a small attack surface.
    • Extensions: Features (like providers, app, or tools) are layered onto the core.
    • Custom Builds: Developers can curate a specific set of extensions to create specialized distributions, such as a CLI-only tool, an API server, or a full UI application.
    • Built-in Extensions: You can manually include any built-in extension by copying it from the llms/extensions directory in the source repository.