lmcpp Rust Integration

You can use the lmcpp-server-cli to quickly start a server with a Web UI. You can use the default model, specify a model via a Hugging Face URL, or point to a local GGUF file.

Using the default model

cargo run --bin lmcpp-server-cli -- --webui

Using a specific model from a URL

cargo run --bin lmcpp-server-cli -- --webui -u https://huggingface.co/bartowski/google_gemma-3-1b-it-qat-GGUF/blob/main/google_gemma-3-1b-it-qat-Q4_K_M.gguf

Using a local model file

cargo run --bin lmcpp-server-cli -- --webui -l /path/to/local/model.gguf

# With default model
cargo run --bin lmcpp-server-cli -- --webui
# Or with a specific model from URL:
cargo run --bin lmcpp-server-cli -- --webui -u https://huggingface.co/bartowski/google_gemma-3-1b-it-qat-GGUF/blob/main/google_gemma-3-1b-it-qat-Q4_K_M.gguf
# Or with a specific local model:
cargo run --bin lmcpp-server-cli -- --webui -l /path/to/local/model.gguf

To add a new GGUF preset model to the system, follow these steps to organize the data and trigger the code generation process:

1. Handle Organization: If the model belongs to a new organization, create a directory in llm_client/llm_models_macros/src/gguf_models/data/<organization> and include an organization.json file.
2. Create Model Directory: Create a directory for the specific model inside the organization directory. It is recommended to use the model_id derived from the HuggingFace repository.
3. Required Data: Add a model_macro_data.json file to the new model's directory.
4. Optional Files:
   • Add config.json to the model's directory.
   • Add tokenizer.json to the model's directory (required if tokenizer extraction from GGUF is not supported for that model).
5. Generate Code: Run the generation script to update the llm_models crate.

# Run the generator to update the llm_models crate
rustc llm_client/llm_models_macros/src/bin/generate_models.rs

The lmcpp package provides two new-type wrappers, ValidDir and ValidFile, that use the type system to guarantee filesystem invariants at construction time. This allows you to pass these types to standard library APIs without performing additional std::fs::metadata checks.

ValidDir Invariants

• Absolute & Canonical: The path is always absolute and has been canonicalized (resolving .., symlinks, and platform-specific case-folding).
• Directory Guarantee: The path is guaranteed to exist and be a directory (metadata().is_dir() is true).
• Auto-creation: Calling ValidDir::new will automatically perform mkdir -p semantics, creating any missing parent directories.

ValidFile Invariants

• Absolute: The path is converted to an absolute path.
• Regular File Guarantee: The path is guaranteed to be a regular file. It will reject paths that are directories or symlinks that point to directories.
• Symlink Behavior: Unlike ValidDir, ValidFile does not automatically resolve symlinks; the symlink itself is preserved if provided, as long as its ultimate target is a regular file.

use lmcpp::server::types::file::{ValidDir, ValidFile};

// ValidDir will create the path if it doesn't exist
let dir = ValidDir::new("/path/to/new_dir")?; 

// ValidFile ensures the path is a regular file
let file = ValidFile::new("/path/to/file.txt")?;

The UdsClient is designed for efficient and resilient IPC communication using a subset of HTTP.

Connection Lifecycle

• Lazy Connection: The connection to the socket is not established until the first request is made.
• Caching: Successful connections are cached in a RwLock to be reused for subsequent requests.
• Automatic Recycling: If a request encounters an I/O error, a timeout, or a serialization error, the client automatically clears the cached stream. The next request will then attempt to establish a fresh connection.
• Connection Closure: If the server responds with a Connection: close header, the client will drop the cached stream after the request completes.

Safety and Limits

• Response Body Cap: To prevent memory exhaustion, the client limits response bodies to 16 MiB.
• Timeouts: By default, the client enforces a 180-second read/write timeout to prevent stalling.

The LocalPrompt struct is used to format messages for local LLMs that require specific chat templates. It manages special tokens like bos_token (Beginning of Sentence), eos_token (End of Sentence), and unk_token (Unknown), and handles the conversion of messages into a formatted string and a sequence of tokens.

Key capabilities include:

• Chat Template Application: Uses a provided template string to format a list of messages.
• Generation Prefixes: Allows setting a prefix (e.g., Assistant: ) to guide model output.
• Token Management: Automatically calculates the total token count and provides the prompt as a vector of token IDs using the configured PromptTokenizer.
• Thread Safety: Uses internal mutability (Mutex) so the prompt state can be updated safely.

// Conceptual usage of LocalPrompt
let local_prompt = LocalPrompt::new(
    tokenizer,
    chat_template_string,
    Some(bos_token),
    eos_token,
    Some(unk_token),
    Some(base_prefix),
);

// Build the prompt from messages
local_prompt.build_prompt(&messages);

// Retrieve results
let prompt_text = local_prompt.get_built_prompt().unwrap();
let tokens = local_prompt.get_built_prompt_as_tokens().unwrap();
let count = local_prompt.get_total_prompt_tokens().unwrap();

The ServerProcessGuard is an RAII handle used to launch and supervise a single instance of an inference server binary. It ensures that the server and its entire process tree are cleaned up when the guard is dropped or when stop() is explicitly called.

Key features:

• Automatic Cleanup: Uses RAII to kill the process tree and remove the PID file upon Drop.
• Concurrency Protection: Creates a unique pid-file to prevent multiple concurrent launches of the same server.
• Robust Containment: Uses platform-specific mechanisms to ensure the entire process tree is reaped:
  • Linux/BSD: Process-group + prctl(PDEATHSIG).
  • macOS: Anonymous pipe (lifeline).
  • Windows: NT Job object (kill-on-close).

To prevent errors, new() will fail if a PID file already exists (indicating a stale or active process).

// Example conceptual usage
let guard = ServerProcessGuard::new(
    &bin_path,
    &bin_dir,
    &pidfile_path,
    &server_args,
)?; 

// The server is now running. 
// When `guard` goes out of scope, the server is stopped and the pid-file is removed.