DOCSEARCHbyAlgolia
Submit Docs

WasmEdge WASI-NN Examples

repository·master·Indexed 42 days ago

https://github.com/second-state/wasmedge-wasinn-examples

High-level Rust bindings and examples for building machine learning applications using the WASI-NN (WebAssembly System Interface for Neural Networks) proposal. Demonstrates running ML models such as MobileNet and YOLOV8 on the WasmEdge runtime using OpenVINO, PyTorch, and TensorFlow backends.

Tokens
41.5K
Snippets
164
Records
199
Agent score
53%

What's inside WasmEdge WASI-NN Examples

  3. How to chain multiple models using --nn-preload

    master
    The wasmedge-ggml-multimodel example demonstrates how to link two or more models together using the WasmEdge WASI-NN plugin. By using the --nn-preload flag with unique identifiers (e.g., llama2:... and llava:...), you can preload multiple large models into the runtime. This allows a single WASM application to orchestrate a pipeline where one model's output serves as input for another (e.g., a vision model describing an image, followed by a language model processing that description).

  4. Understand the Gemma-4 Prompt Format

    master

    The Gemma-4 example uses a specific prompt structure to handle system instructions, thinking modes, and multimodal (image) inputs.

    Standard Format:

    system
<{system prompt}
<image>
{user prompt}

    Key Rules:

    1. Thinking Mode: When enable_thinking=true, the system turn is prefixed with `system

    <. When enable_thinking=false, the <prefix is omitted. 2. **Multimodal Input:** If usingmmprojandimageenvironment variables, you **must** include the literal<image>marker in your prompt. The number of<image>` markers must match the number of input bitmaps provided to the WasmEdge GGML backend.

  5. Download models for the Multiple Models Example

    master

    To run the multi-model chaining example, you need to download the Llama2 and Llava models in GGUF format. Use the following commands to fetch the required files:

    curl -LO https://huggingface.co/second-state/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q5_K_M.gguf
curl -LO https://huggingface.co/cmp-nct/llava-1.6-gguf/resolve/main/vicuna-7b-q5_k.gguf
curl -LO https://huggingface.co/cmp-nct/llava-1.6-gguf/resolve/main/mmproj-vicuna7b-f16.gguf

  6. Install OpenVINO 2023 for WasmEdge WASI-NN

    master

    To use the OpenVINO backend with WasmEdge WASI-NN, you must install OpenVINO version 2023.0.0 on your system. Follow these steps to add the Intel repository and install the package via apt:

    wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
sudo apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
echo "deb https://apt.repos.intel.com/openvino/2023 ubuntu20 main" | sudo tee /etc/apt/sources.list.d/intel-openvino-2023.list
sudo apt update
sudo apt-get -y install openvino
ldconfig

  7. Execute Llama 3 with WasmEdge WASI-NN

    master

    To run Llama 3 using the wasmedge-ggml-llama.wasm module, you must provide a GGUF model file and configure specific environment variables for the prompt template and hardware acceleration.

    Prerequisites

    1. Download the model file (e.g., Meta-Llama-3-8B-Instruct.Q5_K_M.gguf) and place it in your current directory.
    2. Ensure you have the wasmedge runtime installed.

    Configuration Options

    • llama3=true: Set this environment variable to enable the Llama 3 prompt template.
    • n_gpu_layers: Set this environment variable to a numeric value to enable GPU support (e.g., 100).
    • ctx_size: Use the --env ctx_size=N flag to increase the context window. The default is 1024.
    $ wasmedge --dir .:. \
  --env llama3=true \
  --env n_gpu_layers=100 \
  --env ctx_size=8192 \
  --nn-preload default:GGML:AUTO:Meta-Llama-3-8B-Instruct.Q5_K_M.gguf \
  wasmedge-ggml-llama.wasm default

  8. Build WasmEdge with WASI-NN Piper plugin

    master

    To use the Piper plugin for Text-to-Speech, you must build WasmEdge from source with the DWASMEDGE_PLUGIN_WASI_NN_BACKEND=Piper flag. The plugin relies on the ONNX Runtime C++ API.

    Before building WasmEdge, ensure you have installed the ONNX Runtime (e.g., version 1.14.1 on Ubuntu) by moving the include and lib directories to /usr/local/include and /usr/local/lib respectively and running ldconfig.

    # Install ONNX Runtime 1.14.1 on Ubuntu
curl -LO https://github.com/microsoft/onnxruntime/releases/download/v1.14.1/onnxruntime-linux-x64-1.14.1.tgz
tar zxf onnxruntime-linux-x64-1.14.1.tgz
mv onnxruntime-linux-x64-1.14.1/include/* /usr/local/include/
mv onnxruntime-linux-x64-1.14.1/lib/* /usr/local/lib/
rm -rf onnxruntime-linux-x64-1.14.1.tgz onnxruntime-linux-x64-1.14.1
ldconfig

# Build WasmEdge with Piper backend
cmake -GNinja -Bbuild -DCMAKE_BUILD_TYPE=Release -DWASMEDGE_USE_LLVM=OFF -DWASMEDGE_PLUGIN_WASI_NN_BACKEND=Piper
cmake --build build

  9. Install WasmEdge with WASI-NN MLX plugin

    master

    To use the MLX backend for WASI-NN, you must build WasmEdge from source with the DWASMEDGE_PLUGIN_WASI_NN_BACKEND flag set to "mlx". The build process will automatically download MLX. If you require a custom MLX installation, you can set the CMAKE_PREFIX_PATH environment variable during the cmake configuration step.

    After installation, the wasmedge runtime is typically located at /usr/local/bin and the MLX backend plugin is located at /usr/local/lib/wasmedge/libwasmedgePluginWasiNN.so.

    cd <path/to/your/wasmedge/source/folder>

cmake -GNinja -Bbuild -DCMAKE_BUILD_TYPE=Release -DWASMEDGE_PLUGIN_WASI_NN_BACKEND="mlx"
cmake --build build

# Install the runtime and the WASI-NN plugin
cmake --install build

  12. Build the Mobilenet WASI-NN example

    master

    Compile the Rust application to WebAssembly using the wasm32-wasip1 target. This produces two distinct WASM files: one for standard usage and one for using the named model feature.

    To improve execution speed, you can use wasmedgec to compile the WASM files into Ahead-of-Time (AOT) modules.

    # Compile to WASM
cargo build --target=wasm32-wasip1 --release

# Enable AOT mode for faster processing
wasmedgec target/wasm32-wasip1/release/wasmedge-wasinn-example-mobilenet-image.wasm wasmedge-wasinn-example-mobilenet-image-aot.wasm
wasmedgec target/wasm32-wasip1/release/wasmedge-wasinn-example-mobilenet-image-named-model.wasm wasmedge-wasinn-example-mobilenet-image-named-model-aot.wasm