DOCSEARCHbyAlgolia
Submit Docs

LocalLLMClient

repository·main·Indexed 42 days ago

https://github.com/tattn/localllmclient

A Swift-based client library for interacting with local Large Language Models on iOS, macOS, and Linux. It provides a unified interface for GGUF, MLX, and Apple FoundationModels, featuring a streaming API, tool calling, and multimodal image processing capabilities.

Tokens
3.6K
Snippets
10
Records
14
Agent score
54%

What's inside LocalLLMClient

  1. Overview of LocalLLMClient features

    main

    LocalLLMClient is a Swift package designed for interacting with local Large Language Models (LLMs) on Apple platforms (iOS, macOS) and Linux.

    Key capabilities include:

    • Model Support: Works with GGUF, MLX models, and the FoundationModels framework.
    • Streaming: Provides a streaming API for real-time text generation.
    • Experimental Features: Includes support for Multimodal (image processing) and Tool calling.
    IMPORTANT

    This project is experimental and the API is subject to change.

  2. Implement Tool Calling

    main

    LocalLLMClient supports tool calling by registering tools with an LLMSession.

    Requirements:

    • The model must explicitly support tool calling.
    • Backend compatibility varies.

    Implementation Steps:

    1. Define a tool using the @Tool attribute.
    2. Define arguments using a nested struct with the @ToolArguments attribute.
    3. Use @ToolArgument for individual fields and @ToolArgumentEnum for enums.
    4. Implement the call(arguments:) method which returns a ToolOutput.
    5. Pass the tool instance to the tools array in the LLMSession initializer.
    import LocalLLMClient
import LocalLLMClientLlama

@Tool("get_weather")
struct GetWeatherTool {
    let description = "Get the current weather in a given location"
    
    @ToolArguments
    struct Arguments {
        @ToolArgument("The city and state, e.g. San Francisco, CA")
        var location: String
        
        @ToolArgument("Temperature unit")
        var unit: Unit?
        
        @ToolArgumentEnum
        enum Unit: String {
            case celsius
            case fahrenheit
        }
    }
    
    func call(arguments: Arguments) async throws -> ToolOutput {
        let temp = arguments.unit == .celsius ? "22°C" : "72°F"
        return ToolOutput([
            "location": arguments.location,
            "temperature": temp,
            "condition": "sunny"
        ])
    }
}

let weatherTool = GetWeatherTool()
let session = LLMSession(
    model: .llama(
        id: "Qwen/Qwen2.5-1.5B-Instruct-GGUF",
        model: "qwen2.5-1.5b-instruct-q4_k_m.gguf"
    ),
    tools: [weatherTool]
)

let response = try await session.respond(to: "What's the weather like in Tokyo?")
print(response)

  3. Use the Apple MLX backend

    main

    The MLX backend is optimized for Apple Silicon. Use LLMSession.DownloadModel.mlx to define the model and parameters. Similar to the llama.cpp backend, you can track download progress and then initialize an LLMSession with the downloaded model.

    import LocalLLMClient
import LocalLLMClientMLX

let model = LLMSession.DownloadModel.mlx(
    id: "mlx-community/Qwen3-1.7B-4bit",
    parameter: .init(
        temperature: 0.7,
        topP: 0.9
    )
)

try await model.downloadModel { progress in 
    print("Download progress: \(progress)")
}

let session = LLMSession(model: model)
session.messages = [.system("You are a helpful assistant.")]
let response = try await session.respond(to: "Tell me a story about a cat.")
print(response)

  4. Use Multimodal capabilities for Image Processing

    main

    Multimodal models allow you to process images alongside text prompts.

    llama.cpp backend: Requires providing a mmproj model file when initializing the session. MLX backend: Supports multimodal models directly via the model ID.

    Use the attachments parameter in respond(to:) or streamResponse(to:) with .image(.init(resource: .yourImage)) to include an image.

    // llama.cpp example
import LocalLLMClient
import LocalLLMClientLlama

let session = LLMSession(model: .llama(
    id: "ggml-org/gemma-3-4b-it-GGUF",
    model: "gemma-3-4b-it-Q8_0.gguf",
    mmproj: "mmproj-model-f16.gguf"
))

let response = try await session.respond(
    to: "What's in this image?", 
    attachments: [.image(.init(resource: .yourImage))]
)

  5. Install LocalLLMClient via Swift Package Manager

    main

    To use LocalLLMClient in your Swift project, add it as a dependency in your Package.swift file using the following URL and branch:

    dependencies: [
    .package(url: "https://github.com/tattn/LocalLLMClient.git", branch: "main")
]

  6. Use Apple FoundationModels

    main

    For integration with Apple Intelligence, use the foundationModels backend. This is available on iOS 26.0+ / macOS 26.0+ and requires Apple Intelligence. You can use .default to use the system's default model.

    import LocalLLMClient
import LocalLLMClientFoundationModels

let session = LLMSession(model: .foundationModels(
    model: .default,
    parameter: .init(
        temperature: 0.7,
    )
))

let response = try await session.respond(to: "Tell me a short story about a clever fox.")
print(response)

  7. Requirements for LocalLLMClient

    main

    To use LocalLLMClient (specifically for the example app), ensure your environment meets the following specifications:

    • Operating Systems: iOS 18.0+ or macOS 15.0+
    • Development Tools: Xcode 16.3+
    • Hardware [Recommended]: M1 Mac or newer, or recent iPhone Pro models.

  8. Optimize memory usage for large models

    main
    When running larger models, your app may encounter memory constraints. To run these models more reliably on Apple platforms, you should add the com.apple.developer.kernel.increased-memory-limit entitlement to your app.

  9. Quick Start with LocalLLMClient

    main

    To use LocalLLMClient on Apple platforms, you need to download a model (e.g., from Hugging Face), initialize a client using a specific backend (like MLX), and then call generation methods.

    1. Download a model: Use FileDownloader with a .huggingFace source.
    2. Initialize the client: Use the backend-specific factory method (e.g., LocalLLMClient.mlx(url:)) passing the downloaded model URL.
    3. Generate text: Use generateText(from:) for a single response or textStream(from:) for streaming responses.
    import LocalLLMClient
import LocalLLMClientMLX
import LocalLLMClientUtility

// 1. Download model from Hugging Face
let downloader = FileDownloader(
    source: .huggingFace(id: "mlx-community/Qwen3-1.7B-4bit", globs: .mlx)
)
let modelURL = try await downloader.download { print("Progress: \($0)") }

// 2. Initialize a client with the downloaded model
let client = try await LocalLLMClient.mlx(url: modelURL)

// 3. Generate text
let prompt = "Tell me a story about a cat"
let text = try await client.generateText(from: prompt)
print(text)

  10. Run the LocalLLMClient Example App

    main

    To explore the capabilities of LocalLLMClient through the provided example application, follow these steps to clone the repository with submodules and run it in Xcode.

    Note: The example app requires a physical device (iPhone or Mac) and will not work on a simulator.

    # Clone the repository with all submodules
git clone --recursive https://github.com/tattn/LocalLLMClient

# If you already cloned without --recursive, run this instead:
git submodule update --init --recursive

  11. Use the llama.cpp backend

    main

    The llama.cpp backend allows you to download and run GGUF models. You can use LLMSession.DownloadModel.llama to specify model parameters like temperature, topK, topP, and responseFormat. You can track download progress using the downloadModel method. Once downloaded, pass the model object to LLMSession to start interacting.

    import LocalLLMClient
import LocalLLMClientLlama

// Create a model
let model = LLMSession.DownloadModel.llama(
    id: "lmstudio-community/gemma-3-4B-it-qat-GGUF",
    model: "gemma-3-4B-it-QAT-Q4_0.gguf",
    parameter: .init(
        temperature: 0.7,
        topK: 40,
        topP: 0.9,
        options: .init(responseFormat: .json)
    )
)

try await model.downloadModel { progress in 
    print("Download progress: \(progress)")
}

let session = LLMSession(model: model)
session.messages = [.system("You are a helpful assistant.")]
let response = try await session.respond(to: "Your prompt here")
print(response)

  12. Stream text responses with textStream(from:)

    main

    For real-time UI updates, use the textStream(from:) method. This returns an AsyncSequence of strings that you can iterate over using for try await to print or display text as it is being generated.

    // Streaming text
let prompt = "Tell me a story about a cat"
for try await text in try await client.textStream(from: prompt) {
    print(text, terminator: "")
}