RagaAI Catalyst

The agentic_tracing module provides specialized tracing functionality designed for agentic AI systems. It allows developers to track and analyze the complex behaviors of AI agents, specifically focusing on:

• LLM Interactions: Monitoring inputs, outputs, token usage, and costs.
• Tool Usage: Tracking how and when agents execute tools.
• Network Activities: Monitoring API calls and network-level interactions.
• Agent Behavior: Recording decisions, state changes, and user feedback loops.

RagaAI Catalyst is a platform for managing and optimizing LLM projects. Its core capabilities include:

• Project Management: Organizing LLM-related workflows.
• Dataset Management: Handling data used for LLM tasks.
• Evaluation Management: Assessing LLM performance.
• Trace Management: Monitoring execution flows.
• Agentic Tracing: Specialized tracing for agentic workflows.
• Prompt Management: Versioning and managing prompts.
• Synthetic Data Generation: Creating data for testing and training.
• Guardrail Management: Implementing safety and reliability constraints.
• Red-teaming: Testing LLM robustness against adversarial inputs.

The YouTube Summary Agent can be integrated with RagaAI Catalyst to enable agentic tracing. This allows you to:

• Track agent performance.
• Debug complex multi-agent workflows (e.g., interactions between the Clarifier agent and the Summarizer agent).
• Collect data for continuous improvement of the agentic logic.

Agentic Tracing uses structured data classes and utility modules to ensure consistency across traces:

Data Classes

Structured data types are used to represent:

• LLM calls
• Network requests
• Tool executions
• Trace components
• Agent states
• User interactions

Key Utilities

• LLM Utils: Handles model name extraction, token usage calculation, cost computation, and parameter sanitization.
• Unique Decorator: Generates unique identifiers for trace components to ensure traceability.
• Model Costs: A configuration system (model_costs.json) used to calculate the financial cost of LLM interactions.
• Upload Module: Provides functionality via upload_code.py to upload traced code and execution data for analysis.

The schema_mapping parameter is a dictionary used during CSV dataset creation to ensure data alignment with the system's expected format.

• Keys: The exact column names present in your CSV file.
• Values: The corresponding valid schema elements (retrieved via get_schema_mapping()) that define how the data is interpreted.

Example: If your CSV has user_id and response_time, but the system expects user_identifier and response_duration, your mapping should be: {'user_id': 'user_identifier', 'response_time': 'response_duration'}.

This example demonstrates how to use RagaAI Catalyst to monitor agentic workflows. By initializing the Catalyst client within your agent script, you can:

• Track agent performance during extraction tasks.
• Debug complex agent workflows by tracing interactions.
• Collect data for continuous improvement of your agent's extraction accuracy.

The summarizer uses a modular tool-based architecture. The following core functions/tools drive the pipeline:

• get_hugging_face_top_daily_paper(): Scrapes and retrieves the most upvoted paper from HuggingFace.
• get_paper_id_by_title(): Finds the corresponding arXiv ID for a given paper title.
• download_paper_by_id(): Downloads the paper PDF from arXiv using its ID.
• read_pdf_file(): Processes the PDF and extracts text from the first three pages for analysis.

This pipeline uses Qwen2.5-Coder-32B for LLM-powered summarization.

Auto-instrumentation allows RagaAI Catalyst to automatically trace your application by initializing a Tracer and calling init_tracing().

You must select a tracer_type that matches your framework. Supported types include:

• agentic/langgraph
• agentic/langchain
• agentic/smolagents
• agentic/openai_agents
• agentic/llamaindex
• agentic/haystack

from ragaai_catalyst import init_tracing, Tracer

# Initialize the tracer 
tracer = Tracer(
    project_name="Project_Name",
    dataset_name="Dataset_Name",
    tracer_type="agentic/langgraph"  
)

# Enable auto-instrumentation
init_tracing(catalyst=catalyst, tracer=tracer)

The module provides several specialized tracer implementations to capture different dimensions of an agent's lifecycle:

To use the Most Upvoted Paper Summarizer example, you need to install the required dependencies and configure your Hugging Face credentials.

1. Install Dependencies

Install all necessary packages using the provided requirements file:

pip install -r requirements.txt

2. Configure Hugging Face Token

You must provide a Hugging Face API token to access Hugging Face services. You can either:

• Replace the 'HF_API_TOKEN' placeholder directly in the source code with your actual token.
• Set your token as an environment variable.

Install the ragaai-catalyst package using pip to access the platform's management and optimization features for LLM projects.

pip install ragaai-catalyst