PowerInfer Documentation

The LLaMA.cpp HTTP Server is a fast, lightweight, pure C/C++ HTTP server built using httplib, nlohmann::json, and llama.cpp. It provides a set of LLM REST APIs and a simple web front end for interacting with llama.cpp models.

Key Capabilities:

• Inference: Supports F16 and quantized models on both GPU and CPU.
• API Compatibility: Provides OpenAI API compatible routes for chat completions and embeddings.
• Advanced Features: Supports parallel decoding (multi-user), continuous batching, speculative decoding, and schema-constrained JSON response formats.
• Specialized Endpoints: Includes a Reranking endpoint and monitoring endpoints.
• Model Capabilities: Supports Multimodal inputs, Function calling/tool use, and prefilling of assistant messages (similar to the Claude API).

CLI11 provides a wide range of command-line parsing capabilities including:

• Positional arguments: Arguments identified by position rather than flags.
• Flags: Boolean switches.
• Subcommands: Nested App instances that support callback lambda functions.
• Configuration files: Support for reading/producing .ini files.
• Environment variables: Using environment variables as input for options.
• Validators: Custom and built-in validators (e.g., CLI::ExistingFile).
• Standard shell idioms: Support for -- to stop parsing options.

gMock is a C++ framework for writing and using mock classes to improve system design and testing. It provides a declarative syntax for defining mocks and supports partial (hybrid) mocks, which combine real and mock objects.

Key features include:

• Support for functions of arbitrary types and overloaded functions.
• A rich set of matchers for validating function arguments.
• Intuitive syntax for controlling mock behavior.
• Automatic verification of expectations.
• Support for arbitrary (partial) ordering constraints on function calls.
• Extensibility via custom matchers and actions.
• Exception-free design.

CLI11 is a command-line parser for C++. The core functionality is built around several key classes:

• CLI::App: Represents the main application or a subcommand.
• CLI::Option: Represents individual options stored within an App.
• CLI::Validator: Used to perform checks on option values, which can influence the type name.
• CLI::Formatter: A subclassable component used for customizing help message printing.
• CLI::ExitCode: A scoped enum providing standardized exit codes.
• CLI::Timer: A timer class located in CLI/Timer.hpp (note: this is not included in the main CLI11.hpp header).
• CLI::AutoTimer: A timer that automatically prints its duration upon destruction.

Multimodal support in llama.cpp allows language models to process non-text inputs like images and audio. This is achieved by using a separate component called a multimodal projector (mmproj) to encode inputs into embeddings, which are then fed into the core language model.

To run a multimodal model, you typically need two distinct GGUF files:

1. The standard language model file.
2. A corresponding mmproj file specific to the model architecture.

Note: Multimodal support is under heavy development and is subject to breaking changes.

GoogleTest provides a comprehensive suite of testing capabilities:

• Test Discovery: Automatically discovers and runs tests without manual registration.
• Assertions: A rich set of built-in assertions (equality, inequality, exceptions, etc.) and support for user-defined assertions.
• Death Tests: Verifies that code exits in a specific way (useful for error-handling tests).
• Failure Modes: Supports both fatal and non-fatal failures to control test execution flow.
• Parameterized Tests:
  • Value-parameterized tests: Run the same test multiple times with different input values.
  • Type-parameterized tests: Run tests with different data types.
• Execution Options: Supports running individual tests, specifying execution order, and running tests in parallel.

gMock is a C++ library for creating mock objects to test interactions between modules. It is important to distinguish between Fakes and Mocks:

• Fake objects: Have working implementations but use shortcuts (e.g., an in-memory file system) to make operations less expensive. They are not suitable for production.
• Mocks: Are objects pre-programmed with expectations. They form a specification of the calls they are expected to receive (which methods, in what order, how many times, with what arguments, and what they return).

Use gMock to:

• Experiment with interface designs early.
• Remove slow or unreliable outbound dependencies (databases, networks).
• Test how code handles specific failures (e.g., checksum errors).
• Observe interactions between modules rather than just side effects.