llmware Framework

llmware is a unified framework designed for building local, private, and secure knowledge-based LLM applications. It is optimized for AI PCs, laptops, edge devices, and self-hosted deployments across Windows, Mac, and Linux. The framework provides high-level interfaces to leverage various inferencing technologies like GGUF, OpenVINO, ONNXRuntime, ONNXRuntime-QNN (Qualcomm), WindowsLocalFoundry, and PyTorch.

The framework consists of two primary components:

1. Model Catalog: Contains over 300 models prepackaged in quantized and optimized formats. This includes 50+ specialized models (SLIM, Bling, Dragon, and Industry-Bert) for enterprise process automation, as well as support for major cloud models from OpenAI, Anthropic, and Google.
2. RAG Pipeline: Provides integrated components for the full lifecycle of connecting knowledge sources to generative AI, including document parsing, ingestion, and scalable knowledge base creation.

When working with llmware tutorials and notebooks, it is helpful to understand the different notebook environments available:

Jupyter Notebooks

An open-source web application for creating documents containing live code, equations, visualizations, and narrative text. It is typically run locally or on a server.

Google Colab

A cloud-hosted Jupyter notebook environment. It requires no local setup and provides free access to computing resources like GPUs and TPUs, making it ideal for machine learning tasks that require high compute power.

marimo notebooks

A modern open-source alternative to Jupyter. Key features include:

• Reactive Execution: Unlike Jupyter, marimo understands the relationship between cells (similar to a spreadsheet) and automatically updates outputs when a dependency changes.
• Interactive UI: Built-in interactive UI elements for data exploration.
• Deployment: Notebooks can be deployed as interactive web apps or executed as standard Python scripts.

Comparison Summary

The Library class is used to ingest, organize, and index collections of knowledge. A library acts as a container for both text collection (database) resources and file resources.

Workflow:

1. Create: Initialize a new library using create_new_library(library_name).
2. Ingest: Use add_files(path) to ingest a folder of mixed file types (pdf, pptx, docx, xlsx, txt, csv, md, json/jsonl, wav, png, jpg, html). Files are automatically parsed, chunked, and indexed.
3. Embed: Install embeddings by calling install_new_embedding(embedding_model_name, vector_db, batch_size). You can install multiple embeddings (e.g., different models or different vector databases) on the same library.
4. Inspect: Use get_library_card(library_name) to retrieve metadata or get_all_library_cards() to see all libraries.

from llmware.library import Library

# Create a library
lib = Library().create_new_library("my_library")

# Ingest files
lib.add_files("/folder/path/to/my/files")

# Install an embedding (e.g., using Milvus)
lib.install_new_embedding(embedding_model_name="mini-lm-sbert", vector_db="milvus", batch_size=500)

# Install a second embedding (e.g., using ChromaDB)
lib.install_new_embedding(embedding_model_name="industry-bert-sec", vector_db="chromadb", batch_size=100)

# Get metadata
lib_card = Library().get_library_card("my_library")

A Library is a container for ingesting, organizing, and indexing knowledge. It manages both text collection (database) resources and file resources (stored in llmware_data/accounts/{library_name}).

Workflow:

1. Create: Use create_new_library(library_name) to initialize a new container.
2. Ingest: Use add_files(path) to ingest a folder of mixed file types (pdf, pptx, docx, xlsx, txt, csv, md, json/jsonl, wav, png, jpg, html). Files are automatically routed to the correct parser, chunked, and indexed.
3. Embed: Install embeddings using install_new_embedding(). You can mix and match different embedding models and vector databases (e.g., milvus, chromadb) within the same library.
4. Inspect: Use get_library_card(library_name) to retrieve metadata (documents, chunks, images, tables, embeddings) or get_all_library_cards() to see all libraries.

from llmware.library import Library

# Create a library
lib = Library().create_new_library("my_library")

# Ingest files
lib.add_files("/folder/path/to/my/files")

# Install an embedding (e.g., using Milvus)
lib.install_new_embedding(embedding_model_name="mini-lm-sbert", vector_db="milvus", batch_size=500)

# Install a second embedding (e.g., using ChromaDB)
lib.install_new_embedding(embedding_model_name="industry-bert-sec", vector_db="chromadb", batch_size=100)

# Get metadata
lib_card = Library().get_library_card("my_library")