Unity Barracuda Documentation

Unity Barracuda is a lightweight, cross-platform neural networks inference library designed for Unity. It allows you to run neural networks on both the GPU and CPU.

Important Migration Note: Barracuda has been replaced by the Sentis package (currently in closed beta). If you are starting a new project, consider using Sentis. Barracuda remains production-ready for use with ML-Agents and specific supported architectures, but usage in other scenarios is considered to be in the preview development stage.

Barracuda is a lightweight, cross-platform neural network inference library for Unity. It allows you to run neural networks on both the GPU and CPU.

Note: The Barracuda package is deprecated. For new projects, use the Sentis package instead.

When creating a Worker, you can choose between different GPU-based WorkerType implementations depending on your needs for speed, flexibility, or debugging:

• ComputeRef: A slow reference implementation. Use this for debugging; it is more memory-intensive.
• Compute: A fast GPU implementation. It offers flexibility to changes in input dimensions but incurs some CPU overhead.
• ComputePrecompiled: The fastest GPU implementation. It strips out CPU overhead by optimizing execution for a specific input dimension. Note that changing input dimensions will incur a compilation overhead.

The IWorker interface is the core component of the Barracuda engine. A Worker is responsible for breaking down the Model into executable tasks and scheduling them on a specific backend (CPU or GPU). You create a worker using the WorkerFactory.CreateWorker method, specifying the desired WorkerFactory.Type and the loaded Model.

var worker = WorkerFactory.CreateWorker(<WorkerFactory.Type>, m_RuntimeModel);

When preparing models for use in Barracuda, follow these compatibility guidelines:

1. Opset Version: Use ONNX opset=9 whenever possible, as it has the widest operator coverage in Barracuda.
2. TensorFlow/Keras Version: Use TF-1 instead of TF-2 for exports.
3. Data Layout: Since ONNX uses NCHW and TensorFlow/Keras use NHWC, always use the inputs_as_nchw flag (or equivalent) during conversion to ensure the model is correctly formatted for Barracuda.

Barracuda uses a channels-last (NHWC) memory layout. While native ONNX models typically use channels-first, Barracuda automatically converts them to channels-last during model conversion.

For performance, it is recommended to use tensors with 4 dimensions or fewer. These use the batch, height, width, channels dimensions.

Full support is available for up to 8 dimensions in the following order: sequence, direction, batch, extraDimension, depth, height, width, channels.

When creating a Worker, you can choose between different CPU-based WorkerType implementations:

• CSharpRef: A very slow reference implementation. Use this primarily for debugging.
• CSharp: A fast, synchronous C# implementation.
• CSharpBurst: A very fast, asynchronous CPU implementation using Burst.

Barracuda uses the ONNX (Open Neural Network Exchange) format to import models from frameworks like PyTorch, TensorFlow, and Keras.

To execute a model, you follow three main steps:

1. Load the model using ModelLoader.Load.
2. Create a worker (execution engine) using WorkerFactory.CreateWorker specifying the device (e.g., WorkerFactory.Device.GPU).
3. Execute the engine with a Tensor input and retrieve the output using PeekOutput().

var model = ModelLoader.Load(filename);
var engine = WorkerFactory.CreateWorker(model, WorkerFactory.Device.GPU);

var input = new Tensor(1, 1, 1, 10);
var output = engine.Execute(input).PeekOutput();

To analyze values from specific layers (intermediate nodes) during execution, you must specify the desired node names when creating the worker via WorkerFactory.CreateWorker. Once the worker is created with these names, you can retrieve them using PeekOutput just like the final model outputs.

// 1. Define the list of intermediate nodes to query
var additionalOutputs = new string[] {"layer0", "layer1"};

// 2. Pass them to the WorkerFactory during creation
m_Worker = WorkerFactory.CreateWorker(WorkerFactory.Type.ComputePrecompiled, m_RuntimeModel, additionalOutputs);

// ... after execution ...

// 3. Retrieve the intermediate values
var outputLayer0 = worker.PeekOutput("layer0");
var outputLayer1 = worker.PeekOutput("layer1");

To export TensorFlow models, you must first save the model in .pb format (using tf.saved_model.save or tf.train.write_graph) and then use the tf2onnx library to convert it.

Important Considerations:

• Layer Naming: TensorFlow appends :0 to layer names. When specifying inputs and outputs for conversion, you must include this suffix (e.g., input:0).
• Data Format: TensorFlow typically uses NHWC format, while ONNX uses NCHW. It is highly recommended to use the inputs_as_nchw option during conversion to facilitate compatibility with Barracuda.
• Version: When exporting from TensorFlow, it is recommended to use TF-1 instead of TF-2 for better compatibility.

python -m tf2onnx.convert --graphdef model.pb --inputs=input:0 --outputs=output:0 --output model.onnx

To use Barracuda execution results in a graphics pipeline without stalling the CPU or GPU, copy data from a Tensor into a RenderTexture using BarracudaTextureUtils.TensorToRenderTexture.

Note: You can reuse the same RenderTexture object for multiple conversions to save resources.

// Copy tensor to a new RenderTexture
var tensor = worker.PeekOutput();
var texture = BarracudaTextureUtils.TensorToRenderTexture(tensor);

// Reuse an existing RenderTexture
var texture = new RenderTexture(width, height, 0);
// ...
tensor = worker.PeekOutput();
BarracudaTextureUtils.TensorToRenderTexture(tensor, texture);