Spezi LLM

Spezi LLM is a Swift Package designed to integrate Large Language Model (LLM) functionality into applications. It provides modular support for three primary execution environments:

• Local LLM execution: Running models directly on the device.
• OpenAI-based LLMs: Utilizing remote OpenAI APIs.
• Fog LLMs: Utilizing LLMs running on Fog node resources within a local network.

The package includes pre-built UI components like ChatView for OpenAI and Local LLM workflows, as well as views for managing language model downloads.

The GeneratedOpenAIClient is a Swift Package Manager (SPM) target that uses swift-openapi-generator to create Swift client code from the official OpenAI OpenAPI specification.

Key configuration files:

• openapi-generator-config.yaml: Defines the generator's configuration.
• openapi.yaml: Contains the OpenAPI specification.

Preprocessing is required to resolve specific issues in the OpenAI spec, such as incorrect required property markings, oneOf syntax incompatibilities with the generator, and excessive deprecation warnings.

Spezi LLM provides several specialized targets for integrating different LLM execution environments into Spezi-based applications:

• SpeziLLM: The base infrastructure for LLM execution in the Spezi ecosystem.
• SpeziLLMLocal: Enables on-device execution of open-source LLMs (e.g., Llama2, Phi, Gemma, DeepSeek-R1) using mlx-swift.
• SpeziLLMLocalDownload: Manages downloading and storing local models from Hugging Face, including onboarding UI components.
• SpeziLLMOpenAI: Integrates with OpenAI's GPT models via their API, including support for function calling.
• SpeziLLMFog: Dispatches LLM inference jobs to Fog node resources discovered within the local network via mDNS/Bonjour.

SpeziLLMGemini allows you to interact with Google Gemini models within your Swift application.

Important Implementation Note: Gemini support in SpeziLLM is currently considered experimental. It is implemented using the existing SpeziLLMOpenAI infrastructure by leveraging Google's OpenAI compatibility layer. Because of this, not all features may be supported; refer to Google's official documentation for specific feature availability.

The SpeziLLMAnthropic module provides the following core components for interacting with Anthropic's API:

• AnthropicPlatformDefinition: Defines the platform requirements.
• LLMAnthropicPlatform: The platform implementation for Anthropic.
• LLMAnthropicSession: Manages a specific interaction session with an Anthropic model.
• LLMAnthropicSchema: Defines the data schemas used by the Anthropic implementation.

The SpeziLLMLocalDownload target is split into two primary layers:

1. The UI Layer (LLMLocalDownloadView): A high-level view that handles user interaction, progress visualization, and existence checks. It is intended for onboarding flows.
2. The Logic Layer (LLMLocalDownloadManager): The engine that performs the actual network requests and file system operations. It manages the DownloadState which the UI layer (or your own custom logic) can observe to track progress.

When using LLMFogParameters.init(modelType:overwritingAuthToken:systemPrompt:), you can provide a closure that returns an authorization token. This token is passed in the Bearer HTTP field of every request.

Best Practice: Because LLMFogSession instances can be long-lived, the token might expire. The closure should implement caching to prevent unnecessary network roundtrips to your token provider while ensuring the token remains valid.

The SpeziLLMLocal target uses three core components to manage on-device inference via mlx-swift:

1. LLMLocalSchema: Defines the configuration of the LLM, including the model file, context window, maximum output size, and batch size. It is configured using LLMLocalSchema/init(model:parameters:samplingParameters:injectIntoContext:).
2. LLMLocalSession: Represents the active execution of a schema. It manages the interaction history via the context property and performs inference.
3. LLMLocalPlatform: The execution platform that handles the underlying inference logic.

Crucial Pattern: These components should never be used in isolation. They must always be used in conjunction with the Spezi LLMRunner to transform a LLMLocalSchema into a LLMLocalSession.

SpeziLLM uses three primary protocols to abstract Large Language Models, allowing for consistent usage regardless of whether the model is running locally or remotely:

• LLMSchema: The configuration object. It contains all the information necessary to create an executable LLMSession.
• LLMSession: The executable instance of the LLM. It holds the conversational context (history) and the current state. It requires implementing generate() and cancel().
• LLMPlatform: The bridge that turns an LLMSchema into an executable LLMSession.

Note: These abstractions should not be used in isolation; they are designed to be managed by an LLMRunner.