picoLLM Inference Engine

picoLLM Inference Engine is a cross-platform SDK designed for running compressed large language models (LLMs) locally on CPU or GPU. It focuses on accuracy through picoLLM Compression and ensures privacy by running inference 100% offline.

Key Features:

• Accurate: Improved quantization compared to standard GPTQ.
• Private: Local inference ensures data stays on the device.
• Cross-Platform: Supports macOS, Windows, Linux, and Raspberry Pi.
• Cost-Effective: Free for open-weight models.

Note on Connectivity: While inference is offline, an internet connection is required to validate your AccessKey with Picovoice license servers during setup.

picoLLM Inference Engine is a highly accurate, cross-platform SDK designed for running compressed large language models (LLMs) locally. It is optimized for privacy (100% local inference) and performance across various hardware, including CPUs and GPUs.

Supported Platforms:

• Desktop: Linux (x86_64), macOS (arm64, x86_64), and Windows (x86_64, arm64).
• Embedded: Raspberry Pi (3, 4, 5).
• Mobile: Android and iOS.
• Web: Chrome, Safari, Edge, and Firefox.

Key Features:

• High Accuracy: Uses picoLLM Compression, which improves upon GPTQ quantization by optimizing bit allocation.
• Privacy: Inference runs entirely on-device.
• Open-Weight Support: Free to use with open-weight models.

Every user must provide a valid AccessKey obtained from the Picovoice Console.

Important Notes:

• The AccessKey is required for authentication and authorization.
• You must have internet connectivity to validate your AccessKey with Picovoice license servers, even though the actual LLM inference runs 100% offline and locally.
• Keep your AccessKey secret.

An AccessKey is required for authentication and authorization when deploying Picovoice SDKs.

Key details:

• You must obtain a unique AccessKey from the Picovoice Console.
• Connectivity: Even though LLM inference runs 100% offline and locally, you need internet connectivity to validate your AccessKey with Picovoice license servers.
• Security: Keep your AccessKey secret.

An AccessKey is required to deploy Picovoice SDKs, including picoLLM. It serves as your authentication and authorization token.

Important Notes:

• Security: Keep your AccessKey secret.
• Connectivity: While LLM inference runs 100% offline, you must have internet connectivity to validate your AccessKey with Picovoice license servers.
• Usage Limits: The AccessKey verifies that your usage stays within your account limits. You can monitor usage in the Picovoice Console.

An AccessKey is a unique authentication and authorization token required to deploy Picovoice SDKs.

• You must obtain an AccessKey from the Picovoice Console.
• Connectivity: You need internet connectivity to validate your AccessKey with Picovoice license servers during initialization.
• Privacy: Even though the key requires internet validation, the actual LLM inference runs 100% locally and offline.

There are three primary ways to deploy model files in your iOS application:

1. Include in App Bundle: Add the model file as a resource in your application's bundle. Note that Apple enforces maximum app size limits.
2. Host Externally: Host the model file on a server or cloud storage and download it within the app at runtime.
3. Copy to Device: For testing or manual installation, use AirDrop or USB to copy the model to the device's storage, then access it programmatically.

To include the picoLLM Inference Engine in your Android project, ensure mavenCentral() is included in your top-level build.gradle file. Then, add the picollm-android dependency to your app's build.gradle file using the following configuration:

dependencies {
    // ...
    implementation 'ai.picovoice:picollm-android:${LATEST_VERSION}'
}

Install the @picovoice/picollm-node package using either npm or yarn.

Compatibility Requirements:

• Node.js 18 or higher.
• Supported Platforms: Linux (x86_64), macOS (arm64, x86_64), Windows (x86_64, arm64), and Raspberry Pi (3, 4, 5).

yarn add @picovoice/picollm-node

npm install --save @picovoice/picollm-node

1. Open the Completion demo in Android Studio as an Android project.
2. Copy your AccessKey into MainActivity.java.
3. Run the application.

For chat-specific functionality, use the Chat demo.

picoLLM Web Binding requires SharedArrayBuffer, IndexedDB, and SIMD. To enable SharedArrayBuffers, your server must include the following HTTP response headers:

• Cross-Origin-Opener-Policy: same-origin
• Cross-Origin-Embedder-Policy: require-corp

Note: picoLLM can only be run within a worker thread.