LiteRT-LM Documentation

To make tools available to the model, you must declare them within a Preface object. This object is passed to the ConversationConfig::Builder when creating a Conversation.

Tool declarations must be a JSON array where each tool is defined using a JSON schema containing the tool's name, description, and parameters (including type, properties, and required fields).

constexpr absl::string_view kToolString = R"([
{
  "name": "get_weather",
  "description": "Returns the weather for a given location.",
  "parameters": {
    "type": "object",
    "properties": {
      "location": {
        "type": "string",
        "description": "The location to get the weather for."
      }
    },
    "required": [
      "location"
    ]
  }
}
])";

JsonPreface preface;
preface.tools = nlohmann::ordered_json::parse(kToolString);

// Use preface in ConversationConfig::Builder
ABSL_ASSIGN_OR_RETURN(auto conversation_config, 
                   ConversationConfig::Builder()
                       .SetSessionConfig(session_config)
                       .SetPreface(preface)
                       .Build(*engine));

To use multimodal capabilities, the Engine must be initialized with appropriate vision and audio backends (e.g., litert::lm::Backend::GPU for vision).

When sending multimodal messages, the content field in the Message object must be an array of objects rather than a single string. Each object in the array defines a type (e.g., text, image, or audio) and its corresponding data (e.g., text or path).

// To use multimodality, the engine must be created with vision and audio
// backend depending on the multimodality to be used
auto engine_settings = EngineSettings::CreateDefault(
    model_assets,
    /*backend=*/litert::lm::Backend::CPU,
    /*vision_backend*/litert::lm::Backend::GPU,
    /*audio_backend*/litert::lm::Backend::CPU,
);

// ... Create Engine and Conversation ...

// Send message to the LLM with image data.
absl::StatusOr<Message> model_message = (*conversation)->SendMessage(
    Message{
        {"role", "user"},
        {"content", { // Now content must be an array.
          {{"type", "text"}, {"text", "Describe the following image: "}},
          {{"type", "image"}, {"path", "/file/path/to/image.jpg"}}
        }},
    });
CHECK_OK(model_message);

// Send message to the LLM with audio data.
model_message = (*conversation)->SendMessage(
    Message{
        {"role", "user"},
        {"content", { // Now content must be an array.
          {{"type", "text"}, {"text", "Transcribe the audio: "}},
          {{"type", "audio"}, {"path", "/file/path/to/audio.wav"}}
        }},
    });
CHECK_OK(model_message);