GenieX Documentation

GenieX is an on-device Gen AI inference runtime for Qualcomm platforms (Snapdragon X, Snapdragon 8 Elite, and Dragonwing IoT). It provides a single SDK that supports five primary entry points for running LLMs and VLMs:

• CLI: Run and serve models directly from the terminal.
• Python: Embed inference into applications using the Python SDK.
• Java/Kotlin: Android SDK for mobile application integration.
• Docker: Containerized images for reproducible deployments.
• OpenAI-compatible server: A local server that acts as a drop-in replacement for existing OpenAI clients.

The SDK dispatches to one of two underlying runtimes depending on your model and performance needs:

1. llama.cpp runtime: Uses GGML kernels for CPU, GPU, or Hexagon HTP. This is best for broad model coverage, allowing you to run almost any GGUF model from Hugging Face.
2. Qualcomm® AI Engine Direct (QAIRT): An NPU-only runtime. This is best for peak performance using pre-compiled models from the Qualcomm AI Hub.

GenieX is an on-device generative AI inference framework designed for Qualcomm platforms. It provides a unified SDK to run frontier Large Language Models (LLMs) and Vision Language Models (VLMs) locally on Hexagon NPUs, Adreno GPUs, or CPUs. It is the community version of Qualcomm GENIE.

GenieX supports the following target platforms:

• Windows ARM64 (Snapdragon X)
• Android (Snapdragon 8 Elite/Gen series)
• Linux ARM64 (Snapdragon IoT)

GenieX offers five primary entry points for developers:

1. CLI: Run models directly from the terminal or provide model services.
2. Python: Embed inference into applications via the Python SDK.
3. Java/Kotlin: Android SDK for mobile application development.
4. Docker: Containerized images for reproducible deployments.
5. OpenAI-compatible server: A local server that can be used as a drop-in replacement for existing OpenAI clients.

The standard workflow for running a model is: Initialize SDK → Pull Weights → Load → Generate.

Below is a minimal end-to-end example using unsloth/Qwen3-0.6B-GGUF.

// 1. Initialize SDK (call once in Activity.onCreate)
GenieXSdk.getInstance().init(context)

// 2. Pull Model (run in Dispatchers.IO)
ModelManagerWrapper.pullFlow(
    ModelPullInput(
        model_name = "unsloth/Qwen3-0.6B-GGUF",
        precision  = "Q4_0",
        hub        = HubSource.HUGGINGFACE,
    )
).collect { event ->
    when (event) {
        is ModelManagerWrapper.PullEvent.Progress  -> /* update UI */
        ModelManagerWrapper.PullEvent.Completed    -> /* done */
        is ModelManagerWrapper.PullEvent.Error     -> /* show error */
    }
}

// 3. Load Model
val paths = ModelManagerWrapper.getPaths("unsloth/Qwen3-0.6B-GGUF")
    ?: error("Model not downloaded")

val llm = LlmWrapper.builder()
    .llmCreateInput(
        LlmCreateInput(
            model_name = paths.model_name,
            model_path = paths.model_path,
            config     = ModelConfig(nCtx = 4096),
            runtime_id  = "llama_cpp",
            compute_unit  = null,   // null → NPU on Snapdragon (recommended)
        )
    )
    .build()
    .getOrThrow()

// 4. Generate
val chat = arrayListOf(ChatMessage("user", "What is AI?"))
val templated = llm.applyChatTemplate(chat.toTypedArray(), null, false).getOrThrow()

llm.generateStreamFlow(
    templated.formattedText,
    GenerationConfig(maxTokens = 2048),
).collect { result ->
    when (result) {
        is LlmStreamResult.Token     -> print(result.text)
        is LlmStreamResult.Completed -> println("\nDone")
        is LlmStreamResult.Error     -> println("Error: ${result.throwable}")
    }
}

You can run any GGUF model from Hugging Face using the llama_cpp runtime. Model weights are automatically downloaded on the first use. Use AutoModelForCausalLM.from_pretrained() to load the model and specify the device_map to choose the compute unit.

Supported device_map values:

• "auto": Automatically selects NPU for both llama_cpp and qairt.
• "cpu", "gpu", "npu", "hybrid".
• "<runtime>:<compute-unit>" (e.g., llama_cpp:gpu).

from geniex import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B-GGUF",     # HF repo id of a GGUF model, or a local .gguf path
    device_map="auto",          # "auto" | "cpu" | "gpu" | "npu" | "hybrid"
)

messages = [{"role": "user", "content": "What is 2+2?"}]
prompt = model.tokenizer.apply_chat_template(
    messages, add_generation_prompt=True,
)

# Single generation
output = model.generate(prompt, max_new_tokens=256)
print(output.text)
print(f"[{output.profile.generated_tokens} tok, "
      f"{output.profile.decode_speed:.1f} tok/s, stop={output.profile.stop_reason}]")

# Streaming generation
streamer = model.generate(prompt, max_new_tokens=256, stream=True)
for chunk in streamer:
    print(chunk, end="", flush=True)

model.close()

To run an LLM on the NPU using Qualcomm AI Engine Direct, use LlmWrapper.builder() with LlmCreateInput.

Configuration Requirements:

• runtime_id must be set to "qairt".
• compute_unit must be null (this selects the NPU, which is the only option for Qualcomm AI Engine Direct).
• Important: Do not attempt to set nGpuLayers or nCtx (context length), as these are fixed at compile time in the AI Hub bundle. Setting them will result in a PARAM_NOT_SUPPORTED error. Use max_tokens and enable_thinking within ModelConfig instead.

To generate text, first call applyChatTemplate to format the chat history, then pass the resulting formattedText to generateStreamFlow.

val paths = ModelManagerWrapper.getPaths("ai-hub-models/Qwen3-4B-Instruct-2507")
    ?: error("Model not downloaded")

LlmWrapper.builder()
    .llmCreateInput(
        LlmCreateInput(
            model_name = paths.model_name,
            model_path = paths.model_path,
            config     = ModelConfig(max_tokens = 2048, enable_thinking = false),
            runtime_id  = "qairt",
            compute_unit  = null,           // null → NPU (only option for Qualcomm AI Engine Direct)
        )
    )
    .build()
    .onSuccess { llmWrapper = it }
    .onFailure { println("Error: ${it.message}") }

val chat = arrayListOf(ChatMessage("user", "What is AI?"))

llmWrapper.applyChatTemplate(chat.toTypedArray(), null, false).onSuccess { t ->
    llmWrapper.generateStreamFlow(t.formattedText, GenerationConfig()).collect { result ->
        when (result) {
            is LlmStreamResult.Token     -> print(result.text)
            is LlmStreamResult.Completed -> println("\nDone")
            is LlmStreamResult.Error     -> println("Error: ${result.throwable}")
        }
    }
}