MLflow

website·Indexed 42 days ago

https://mlflow.org/docs/latest/rest-api.html

An open-source platform for managing the ML lifecycle, featuring specialized tools for GenAI including automatic and manual tracing for LLMs and AI agents, prompt optimization via mlflow.genai.optimize_prompts(), and an AI Gateway for managing model endpoints. It provides evaluation capabilities with built-in LLM judges, distributed tracing via OpenTelemetry, and an Agent Server for hosting AI agents.

Tokens: 39.3K
Snippets: 173
Records: 233
Agent score: 74%

What's inside MLflow

Overview of MLflow Prompt Engineering UI
Introduced in MLflow 2.7, the Prompt Engineering UI allows developers to iteratively build and evaluate LLM prompts without writing code. It integrates with the MLflow AI Gateway to test multiple LLMs, parameter configurations, and prompt templates. Every configuration (prompt templates, LLM choice, parameters) is tracked as an MLflow Run and stored in the MLflow Model format, making them ready for batch or real-time inference.
https://mlflow.org/docs/latest/genai/prompt-registry/prompt-engineeringwebsitellms
Overview of MLflow AI Issue Discovery
AI Issue Discovery is a tool that uses hypothesis-driven analysis to examine LLM application or AI agent traces logged in MLflow experiments. It identifies operational issues (errors, timeouts, rate limiting), quality issues (verbosity, inconsistency, repetitive content), success patterns (effective tool usage), and performance metrics (latency distributions, success rates). The output is a detailed markdown report containing quantitative evidence, specific trace examples, and actionable recommendations.
https://mlflow.org/docs/latest/genai/eval-monitor/ai-insights/ai-issue-discoverywebsitellms
Overview of MLflow Agent Server features
The MLflow Agent Server is a simple FastAPI server designed to host agents. Key features include:
- A standardized /invocations endpoint for agent interaction.
- Decorator-based registration using @invoke and @stream for easy development.
- Automatic request and response validation for agents following the Responses API schema.
- Built-in MLflow tracing integration and aggregation.
https://mlflow.org/docs/latest/genai/serving/agent-serverwebsitellms
Overview of MLflow AI Gateway supported model providers
MLflow AI Gateway natively supports a wide range of model providers for Chat and Embeddings. If a provider is not natively supported, users can optionally install LiteLLM to extend the gateway's capabilities.
Major Cloud Providers:
- OpenAI (Chat, Embeddings)
- Anthropic (Chat)
- Google Gemini (Chat, Embeddings)
- Azure OpenAI (Chat, Embeddings)
- AWS Bedrock (Chat, Embeddings)
- Vertex AI (Chat, Embeddings)
Additional Providers:
- Cohere, Mistral, Together AI, Fireworks AI, Ollama, Databricks, and Portkey (all support Chat; most support Embeddings except Groq).
https://mlflow.org/docs/latest/genai/governance/ai-gateway/endpoints/model-providerswebsitellms
Overview of MLflow Automatic Tracing
MLflow Tracing provides a one-line automatic tracing experience for various LLM and AI agent frameworks. It automatically captures application logic and intermediate steps based on the implementation of the supported SDK or framework, allowing developers to visualize and debug AI workflows without extensive manual instrumentation.
https://mlflow.org/docs/latest/genai/tracing/app-instrumentation/automaticwebsitellms
Track token usage and cost for LangChain
MLflow automatically tracks token usage and cost for LangChain. Token usage for each LLM call during a chain invocation is logged within each Trace/Span. Aggregated cost and time trends are available in the built-in MLflow dashboard.
https://mlflow.org/docs/latest/genai/tracing/integrations/listing/langchainwebsitellms
Evaluate AI Agents with MLflow
MLflow provides a toolkit to systematically evaluate AI agents—applications that use tools and execute multi-step workflows. The evaluation process involves building the agent, creating a dataset with expectations, defining scorers to analyze both final outputs and intermediate behaviors (via traces), and executing the evaluation to analyze results in the MLflow UI.
https://mlflow.org/docs/latest/genai/eval-monitor/running-evaluation/agentswebsitellms
Enable basic HTTP authentication in MLflow
MLflow supports basic HTTP authentication to provide access control over experiments, registered models, and scorers. When enabled, all visitors must authenticate before they can view or access any resources from the Tracking Server.
https://mlflow.org/docs/latest/self-hosting/security/basic-http-authwebsitellms
Implement SSO in MLflow using the OIDC Plugin
MLflow supports Single Sign-On (SSO) via custom plugins. The community-maintained mlflow-oidc-auth plugin enables OIDC-based authentication for the MLflow UI and API. It provides user and group-based access control, permissions management using regular expressions, and compatibility with the MLflow basic auth protocol for mlflow-client authentication.
https://mlflow.org/docs/latest/self-hosting/security/ssowebsitellms

Trace LangGraph agents in Python using MLflow

MLflow provides automatic tracing for LangGraph in Python as an extension of its LangChain integration. By calling mlflow.langchain.autolog(), MLflow automatically captures graph executions into traces and logs them to the active MLflow Experiment.

import mlflow
from langchain_openai import ChatOpenAI
from langchain.agents import create_agent
from langchain_core.tools import tool

# 1. Enable tracing
mlflow.langchain.autolog()

# 2. Optional: Configure tracking server and experiment
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("LangGraph")

@tool
def get_weather(city: str):
    """Use this to get weather information."""
    return "It's sunny in sf"

llm = ChatOpenAI(model="gpt-4o-mini")
tools = [get_weather]
graph = create_agent(llm, tools)

# 3. Invoke the graph to generate a trace
result = graph.invoke({"messages": [{"role": "user", "content": "what is the weather in sf?"}]})

Configure MLflow client for OIDC Basic Auth
The OIDC plugin supports the MLflow basic auth protocol for client connections. You must generate an access token from the OIDC permission page in the MLflow UI. Note that creating a new token invalidates the previous one.
```
# Set credentials for basic auth
export MLFLOW_TRACKING_USERNAME=your_oidc_username
export MLFLOW_TRACKING_PASSWORD=your_generated_access_token
```
```
import mlflow

mlflow.set_tracking_uri("http://localhost:8080")
with mlflow.start_run():
    mlflow.log_param("key", "value")
```
https://mlflow.org/docs/latest/self-hosting/security/ssowebsitellms
Trace Google Agent Development Kit (ADK) with MLflow
MLflow provides automatic tracing for Google ADK via OpenTelemetry integration. To enable tracing, you must install the necessary libraries, start an MLflow Tracking Server with a SQL-based backend (file-based stores are not supported for OpenTelemetry ingestion), and configure the OpenTelemetry tracer to export to the MLflow /v1/traces endpoint.
```
# 1. Install libraries
pip install 'mlflow>=3.6.0' google-adk opentelemetry-exporter-otlp-proto-http

# 2. Start MLflow Tracking Server (SQL backend required)
mlflow server --backend-store-uri sqlite:///mlflow.db --port 5000

# 3. Configure OpenTelemetry environment variables
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:5000
export OTEL_EXPORTER_OTLP_HEADERS=x-mlflow-experiment-id=123
```
https://mlflow.org/docs/latest/genai/tracing/integrations/listing/google-adkwebsitellms