LLM.swift

For models that support Chain of Thought (CoT), LLM.swift allows you to separate the model's internal reasoning from its final response.

Using ThinkingMode

When calling respond, you can specify a thinking mode:

• .enabled: Appends the thinking start token to force the model to think. The library parses and separates the content.
• .suppressed: Uses the "nothink" technique (appending empty thinking tokens) to force the model to skip thinking.
• .none: (Default) No special handling for thinking tokens.

Retrieving Content

• bot.thinking: Contains the accumulated reasoning/thought process.
• bot.output: Contains the final response (excluding the thinking process).

Example:

// Enable thinking
await bot.respond(to: input, thinking: .enabled)

// Access results
print("Thought: \(bot.thinking)")
print("Response: \(bot.output)")

await bot.respond(to: input, thinking: .enabled)
print(bot.thinking)

When using the LLM class, you can hook into the response lifecycle by defining four specific properties. These are used during the execution of the respond function:

• preprocess: A closure used to format user input and conversation history into a string that conforms to a specific chat template (e.g., ChatML). It receives the input, history, and current thinking mode.
• postprocess: A closure executed after the model finishes generating an output. The default is { print($0) }. This is useful for triggering logic like function calling based on the final text.
• update: A closure called every time a new outputDelta (a chunk of text) is received during streaming. outputDelta is nil when generation stops.
• updateThinking: A closure called when new thinking tokens (Chain of Thought) are received.

If you require fine-grained control over the streaming process, you can override the open func respond(to input: String) async method in a subclass.

By default, LLM.swift uses the chat template embedded within the .gguf file (via llama.cpp's Jinja engine). This allows models to work out-of-the-box without manual template configuration.

Key behaviors:

• Automatic Templates: If you don't pass a template: parameter, the library uses the model's embedded template.
• Manual Overrides: You can pass an explicit Template (e.g., .gemma, .chatML) to override the embedded one if the metadata is missing or broken.
• Incremental Context: Conversation context is maintained incrementally between turns; only new tokens are evaluated, and history is not re-fed to the model every turn.
• Thinking/Reasoning: For models that support it, reasoning/thinking separation is handled automatically via bot.thinking without extra configuration.

// Using embedded template automatically
let bot = try await LLM(from: HuggingFaceModel("unsloth/Qwen3-0.6B-GGUF", .Q4_K_M))!
bot.systemPrompt = "You are a sentient AI with emotions."
await bot.respond(to: "What's the meaning of life?")

To ensure user input conforms to a model's expected chat format (like ChatML), use the Template struct. Instead of manually writing a preprocess function, you can set the template property on your LLM instance. This automatically configures both preprocess and stopSequence.

Common ways to set a template:

1. Using built-in static methods: self.template = Template.chatML("System Prompt").
2. Manual configuration: Define custom start/end tokens for system, user, and bot roles.

Example of manual Template configuration:

self.template = Template(
    system: ("<|im_start|>system\n", "<|im_end|>\n"),
    user: ("<|im_start|>user\n", "<|im_end|>\n"),
    bot: ("<|im_start|>assistant\n", "<|im_end|>\n"),
    stopSequence: "<|im_end|>",
    systemPrompt: "You are a sentient AI with emotions."
)

self.template = Template.chatML("You are a sentient AI with emotions.")

You can extend the model's capabilities by providing it with Tools. A tool allows the model to call your Swift code to retrieve information or perform actions.

Implementation Steps:

1. Create a struct that conforms to the Tool protocol.
2. Define a nested struct for the arguments, annotated with @Generatable (this generates the JSON schema the model uses).
3. Implement the call(_:) method to execute your logic.
4. Assign the tool(s) to the bot.tools array.

Important Notes:

• Template Requirement: Tool use requires the model's native tool-call format. You must use the embedded template (do not pass a manual template: during initialization) for this to work.
• Monitoring: Tool calls are recorded in the @Published var toolCalls: [ToolCall] property.
• Error Recovery: If a tool fails or arguments fail to decode, the error is fed back to the model to allow it to attempt recovery.

struct GetWeather: Tool {
    let description = "Get the current weather for a city"

    @Generatable
    struct Arguments {
        let city: String
    }

    func call(_ arguments: Arguments) async throws -> String {
        "It is sunny and 22 degrees celsius in \(arguments.city)."
    }
}

let bot = try await LLM(from: HuggingFaceModel("unsloth/Qwen3-0.6B-GGUF", .Q4_K_M))!
bot.systemPrompt = "You are a helpful assistant."
bot.tools = [GetWeather()]
await bot.respond(to: "What is the weather in Seoul right now?")