LLamaSharp Documentation

LLamaSharp executors define how a model behaves when called. There are four primary types of executors available:

• InteractiveExecutor: Best for continuous question-and-answer sessions where the model acts as an assistant.
• InstructExecutor: Designed for following specific instructions (e.g., "continue writing").
• StatelessExecutor: Ideal for one-time tasks. It has no memory of previous inferences, ensuring a clean context for every call.
• BatchedExecutor: High-throughput executor that accepts multiple inputs from different sessions to generate multiple outputs simultaneously.

LLamaContext is the central component linking native APIs to higher-level APIs in LLamaSharp. It serves two primary purposes:

1. Inference Settings: It holds the configuration for model inference.
2. KV-Cache Management: It holds the key-value (KV) cache, which is critical for accelerating model inference.

Key Architectural Behaviors:

• Decoupling: LLamaContext is not coupled with LLamaWeights. This allows you to create multiple contexts (different sessions/states) from a single set of model weights.
• Executor Relationship: Each ILLamaExecutor holds a LLamaContext instance, though an executor can be switched to use different contexts.
• Session Management: If your application manages multiple user sessions, you must manually manage the lifecycle and state of your LLamaContext instances.

Both executors aim to "complete the prompt," but they differ in how they handle conversation roles and prompt formatting:

• Interactive Mode: The user acts as a 'User' and the LLM acts as an 'Assistant'. This is suitable for models like chat-with-bob. Use a prompt format that establishes a dialogue (e.g., User: ... Bob: ...).
• Instruct Mode: The user provides instructions for the LLM to follow. This is suitable for models like alpaca. Use a prompt format that explicitly describes a task (e.g., Below is an instruction...).

Always modify your prompt template when switching between these two modes to ensure optimal performance.

When using InteractiveExecutor with multimodal weights, you can include images in prompts by wrapping their file paths in braces (e.g., {c:/path/to/image.jpg}).

Processing Logic

• Loading Media: Referenced files are loaded via SafeMtmdWeights.LoadMedia, which produces SafeMtmdEmbed instances.
• Prompt Replacement: The executor identifies brace-wrapped paths, clears the KV cache using MemorySequenceRemove, and replaces the paths with the designated media marker.
• Embedding Submission: The embeds for the current turn are collected into ex.Embeds. The executor then submits both the text prompt and the pending media embeds to the helper for generation.

You can use LLamaSharpTextCompletion to add local LLaMA queries as a text completion service within a Semantic Kernel KernelBuilder. This implementation requires an ILLamaExecutor, such as a StatelessExecutor.

using var model = LLamaWeights.LoadFromFile(parameters);
// LLamaSharpTextCompletion can accept ILLamaExecutor. 
var ex = new StatelessExecutor(model, parameters);
var builder = new KernelBuilder();
builder.WithAIService<ITextCompletion>("local-llama", new LLamaSharpTextCompletion(ex), true);

To avoid errors or truncated responses, ensure your total token count stays within the model's context limit. The following inequality must hold:

len(prompt) + len(response) < len(context)

Where len(response) is the number of tokens you expect the LLM to generate.

LLamaSharp supports multi-modal capabilities, allowing models to process both text and image inputs. To use multi-modal models like LLaVA, you must provide two separate model files:

1. The main model file.
2. The mm-proj (multi-modal projector) model file.

You can use LLamaSharpEmbeddingGeneration to provide local embedding capabilities to Semantic Kernel. This is typically used when building a kernel with memory storage (e.g., VolatileMemoryStore). This implementation requires a LLamaEmbedder instance.

using var model = LLamaWeights.LoadFromFile(parameters);
var embedding = new LLamaEmbedder(model, parameters);
var kernelWithCustomDb = Kernel.Builder
    .WithLoggerFactory(ConsoleLogger.LoggerFactory)
    .WithAIService<ITextEmbeddingGeneration>("local-llama-embed", new LLamaSharpEmbeddingGeneration(embedding), true)
    .WithMemoryStorage(new VolatileMemoryStore())
    .Build();

To use LLamaSharp for chat-based interactions in Semantic Kernel, use the LLamaSharpChatCompletion class. This implementation requires an InteractiveExecutor to best fit the chat command pattern.

using var model = LLamaWeights.LoadFromFile(parameters);
using var context = model.CreateContext(parameters);
// LLamaSharpChatCompletion requires InteractiveExecutor, as it's the best fit for the given command.
var ex = new InteractiveExecutor(context);
var chatGPT = new LLamaSharpChatCompletion(ex);