hugot

Hugot uses a Session to manage and retrieve pipelines. The core workflow involves creating a session, initializing pipelines via standalone functions, and retrieving them by alias.

Core Components

• Session struct: Created via NewSession, this object acts as a registry that holds all created pipelines, mapping them to name aliases.
• NewPipeline function: A standalone generic function used to create a new pipeline and register it within a Session. It takes a session object and a pipeline configuration struct as input.
• GetPipeline function: Used to retrieve a specific pipeline from a Session using its assigned alias.

Pipeline Abstractions

Every pipeline is built upon two primary components defined in pipelines/pipeline.go:

1. BasePipeline struct: Provides common attributes (like ModelPath) and shared methods (like loadModel()). Specific pipeline implementations should embed this struct.
2. The pipeline interface: Defines the required behavior for all pipelines. Any new pipeline must implement this interface.

To implement a new pipeline type, you should create a new struct, embed BasePipeline, implement the pipeline interface methods, and update the Session and NewPipeline/GetPipeline logic in hugot.go to support the new type.

// Conceptual workflow:
// 1. session := NewSession()
// 2. pipeline := NewPipeline[MyNewPipeline](session, config)
// 3. retrieved := GetPipeline[MyNewPipeline](session, "alias")

Onnx Runtime (ORT)

• Build Tag: -tags ORT or -tags ALL.
• Dependency: Requires libonnxruntime.so.
• Setup: By default, Hugot looks for the library at /usr/lib/libonnxruntime.so. You can specify a custom path using WithOnnxLibraryPath() in NewORTSession().

OpenXLA (XLA)

• Build Tag: -tags XLA or -tags ALL.
• Setup: Install the XLA backend via the pjrt_installer:

  GOPROXY=direct go run github.com/gomlx/go-xla/cmd/pjrt_installer@latest -plugin=linux -version=v${GOPJRT_VERSION} -path=/usr/local/lib/go-xla

Tokenizers (Required for XLA and ORT)

• Dependency: Requires tokenizers.a (Rust-based).
• Setup: Place tokenizers.a at /usr/lib/tokenizers.a or specify the directory via the CGO_LDFLAGS environment variable.

Tip: Use the official Hugot Docker image to have all dependencies pre-installed.

The recommended way to develop with Hugot is using a Docker container pre-configured with tokenizer and onnxruntime libraries. This ensures all necessary dependencies and test models are available.

Using Make

From the root of the repository, use the following commands:

1. Start the environment: Downloads test models, builds the container, and launches it (mounting your source code to /home/testuser/repositories/hugot).

   make start-dev-container

2. Stop the environment: Tears down the container.

   make stop-dev-container

Using VS Code Remote

You can attach to the container using the VS Code Remote extension. Use the following configuration for your devcontainer settings:

{
    "remoteUser": "testuser",
    "workspaceFolder": "/home/testuser/repositories/hugot",
    "extensions": [
		"bierner.markdown-preview-github-styles",
		"golang.go",
		"ms-azuretools.vscode-docker"
	],
    "remoteEnv": {"GOPATH": "/home/testuser/go"}
}

Bare Metal Requirements

If you choose not to use Docker, you must manually install the following libraries to your system:

• tokenizers.a located at /usr/lib/tokenizers.a
• onnxruntime.so located at /usr/lib/onnxruntime.so

Hugot supports training and fine-tuning for the FeatureExtractionPipeline. This requires building Hugot with XLA enabled, as it uses goMLX to convert ONNX models to XLA for training before serializing them back to ONNX.

This is commonly used to fine-tune vector embeddings for semantic textual similarity (e.g., for RAG).

Training Data Format

Your training dataset must be in JSONL format with the following structure:

{"sentence1": "text", "sentence2": "text", "score": 0.0}

Where score is a float between 0 and 1 representing semantic similarity.

Loss Functions

By default, a cosine similarity loss is used. You can specify alternative loss functions from goMLX by configuring the XLATrainingOptions field within the TrainingConfig struct.

{"sentence1": "The quick brown fox jumps over the lazy dog", "sentence2": "A quick brown fox jumps over a lazy dog", "score": 1}
{"sentence1": "The quick brown fox jumps over the lazy dog", "sentence2": "A quick brown cow jumps over a lazy caterpillar", "score": 0.5}

A Session is the primary entry point for managing the lifecycle of ONNX transformer pipelines and models. It handles model loading, prevents redundant loading of the same model, and ensures that resources are cleaned up when the session is destroyed.

Key responsibilities of a Session:

• Model Management: Automatically loads and caches models based on their path and ONNX filename. If multiple pipelines use the same model, the session ensures the model is only loaded once.
• Pipeline Lifecycle: Allows creating (NewPipeline), retrieving (GetPipeline), and closing (ClosePipeline) specific pipelines.
• Resource Cleanup: The Destroy() method tears down all pipelines, models, and the underlying ONNX runtime environment.

It is highly recommended to call s.Destroy() using defer to prevent memory leaks.

// Example of session lifecycle management
session, err := hugot.NewSession(ctx, "XLA", hugot.WithOption(...))
if err != nil {
	log.Fatal(err)
}
defer session.Destroy()

// Create a pipeline
pipeline, err := hugot.NewPipeline(session, hugot.TextGenerationConfig{
	Name: "my-gen-pipeline",
	ModelPath: "/path/to/model",
	OnnxFilename: "model.onnx",
})