DOCSEARCHbyAlgolia
Submit Docs

java-llama.cpp

repository·master·Indexed 19 days ago

https://github.com/kherud/java-llama.cpp

Java bindings for llama.cpp providing high-performance LLM inference, including support for Gemma 3, for Java and Android applications. The library supports text generation, streaming, and embeddings via the LlamaModel class. It provides prebuilt CPU inference for Linux, MacOS, and Windows, with options for manual compilation for GPU acceleration (e.g., CUDA). Key components include ModelParameters for model loading and InferenceParameters for task-specific configuration.

Tokens
2.5K
Snippets
8
Records
13
Agent score
64%

What's inside java-llama.cpp

  1. Configure Model and Inference parameters

    master

    Configuration is split into two builder-based classes:

    1. ModelParameters: Configured once when loading the model. Includes settings like .setModel(String) and .addLoraAdapter(String).
    2. InferenceParameters: Configured for every inference task. Includes settings like .setTemperature(float), .setStopStrings(String), .setGrammar(String), and .setMiroStat(MiroStat).

    For Infilling tasks, use InferenceParameters#setInputPrefix(String) and InferenceParameters#setInputSuffix(String).

    ModelParameters modelParams = new ModelParameters()
        .setModel("/path/to/model.gguf")
        .addLoraAdapter("/path/to/lora/adapter");

InferenceParameters inferParams = new InferenceParameters("")
        .setGrammar("root ::= (expr "=\" term "\\n")+")
        .setTemperature(0.8);

try (LlamaModel model = new LlamaModel(modelParams)) {
    model.generate(inferParams);
}

  2. Understand the Local Model Directory

    master
    The models/ directory in this repository is used to store model files that are automatically downloaded for the purpose of running java-llama.cpp unit tests. This directory is not intended for user-provided models for production inference, but rather serves as a cache for test dependencies.

  4. Import java-llama.cpp into an Android project

    master

    To use this library in Android, follow these steps:

    1. Add the repository as a git submodule in your app directory: git submodule add https://github.com/kherud/java-llama.cpp

    2. Configure your build.gradle to include the library as a source, handle the C++ build via CMake, and run mvn compile if necessary.

    3. Add a ProGuard rule to prevent the library from being obfuscated: keep class de.kherud.llama.** { *; }

    android {
    val jllamaLib = file("java-llama.cpp")

    if (!file("$jllamaLib/target").exists()) {
        exec {
            commandLine = listOf("mvn", "compile")
            workingDir = file("java-llama.cpp/")
        }
    }

    externalNativeBuild {
        cmake {
            path = file("$jllamaLib/CMakeLists.txt")
            version = "3.22.1"
        }
    }

    sourceSets {
        named("main") {
            java.srcDir("$jllamaLib/src/main/java")
        }
    }
}

  5. Compile the library for custom platforms or GPU acceleration

    master

    If you are on an unsupported platform or require GPU acceleration (e.g., CUDA), you must compile the library manually.

    1. Ensure you have cmake and mvn installed.
    2. Run the following commands in the java-llama.cpp repository directory:
    mvn compile
cmake -B build # Add backend arguments here, e.g., -DGGML_CUDA=ON
cmake --build build --config Release

    Tip: Use -DLLAMA_CURL=ON during the cmake step to enable downloading models via Java code using ModelParameters#setModelUrl(String).

    mvn compile
cmake -B build
cmake --build build --config Release

  7. Configure shared library loading locations

    master

    The application searches for the jllama shared library (.dll, .so, or .dylib) in this order:

    1. de.kherud.llama.lib.path: Set this via the VM option -Dde.kherud.llama.lib.path=/path/to/directory to specify a custom location.
    2. java.library.path: The standard system library paths.
    3. From the JAR: If no other location is found, the application attempts to use a prebuilt library included in the JAR (only for supported platforms).

  8. Initialize the native libraries with initialize()

    master

    Before using the java-llama.cpp library, you must call LlamaLoader.initialize() to load the necessary llama and jllama shared libraries (.dll, .dylib, or .so).

    This method performs several automated steps:

    1. Cleans up old native libraries from the temporary directory.
    2. On macOS, it extracts and prepares ggml-metal.metal for Metal acceleration.
    3. Attempts to load the libraries in the following order:
      • From the directory specified by the system property de.kherud.llama.lib.path.
      • Directly from the Android APK (if running on Android).
      • From the paths listed in java.library.path.
      • By extracting the OS-specific library directly from the project's JAR file into a temporary directory.

    If no compatible native library is found, it throws an UnsatisfiedLinkError.

    de.kherud.llama.LlamaLoader.initialize();

  9. Configure logging with LlamaModel.setLogger

    master

    Logs are written to stdout by default. You can intercept or disable them using the static method LlamaModel.setLogger(LogFormat, BiConsumer<LogLevel, String>).

    • LogFormat.TEXT: Includes text logs and GGML backend output.
    • LogFormat.JSON: Includes only request logs (GGML messages still go to stdout).
    • Custom Callback: Pass a BiConsumer to redirect logs to your own logging framework.
    • Change Format Only: Pass null as the callback to keep stdout output but change the format.
    • Disable Logging: Pass an empty callback.
    // Redirect to custom logger
LlamaModel.setLogger(LogFormat.TEXT, (level, message) -> System.out.println(level.name() + ": " + message));

// Change format to TEXT but keep stdout
LlamaModel.setLogger(LogFormat.TEXT, null);

// Disable logging
LlamaModel.setLogger(null, (level, message) -> {});

  10. Perform text inference and embeddings

    master

    Use LlamaModel to perform inference. Note that LlamaModel is stateless; you must append model outputs to your prompt to maintain context. LlamaModel implements AutoCloseable, so use try-with-resources to prevent memory leaks from the underlying C++ allocation.

    • Streaming: Use model.generate(InferenceParameters) to iterate over LlamaOutput objects.
    • Completion: Use model.complete(InferenceParameters) to get the full response string.
    • Embeddings: Use model.embed(String) to get the hidden representation of the text.
    ModelParameters modelParams = new ModelParameters().setModel("/path/to/model.gguf");
InferenceParameters inferParams = new InferenceParameters("Tell me a joke.");

try (LlamaModel model = new LlamaModel(modelParams)) {
    // Stream response
    for (LlamaOutput output : model.generate(inferParams)) {
        System.out.print(output);
    }

    // Get full response
    String response = model.complete(inferParams);

    // Get embeddings
    float[] embedding = model.embed("Embed this");
}

  11. Configure native library loading via system properties

    master

    You can control how and where the native libraries are loaded using the following system properties:

    PropertyDescription
    de.kherud.llama.lib.pathSpecifies a custom directory containing the required *.dll, *.dylib, or *.so files. If set, the loader will check this path before other locations.
    de.kherud.llama.tmpdirSpecifies the directory where native libraries will be extracted from the JAR. Defaults to java.io.tmpdir if not provided.
    java.library.pathStandard Java property used by the loader to search for native libraries in specified system paths.

  12. Manually load a native library from a specific path

    master

    If you have the native library files located at a specific location and want to bypass the automatic extraction/search logic, you can use LlamaLoader.loadNativeLibrary(Path path).

    This method attempts to load the library using System.load(absolutePath). It returns true if the library was successfully loaded, or false if the file does not exist or an UnsatisfiedLinkError occurred.

    import java.nio.file.Paths;
import de.kherud.llama.LlamaLoader;

boolean success = LlamaLoader.loadNativeLibrary(Paths.get("/path/to/libllama.so"));
if (success) {
    // Proceed with inference
}