AdalFlow Documentation

LLM evaluation is a multifaceted process categorized into three main pillars:

1. What to evaluate: Defining the tasks and capabilities (e.g., NLU, NLG, Reasoning, Robustness, Fairness, Domain Adaptation, or Agent applications).
2. Where to evaluate: Selecting appropriate datasets and benchmarks (e.g., MMLU, HumanEval, HELM, or scenario-specific RAG datasets like RGB and CRAG).
3. How to evaluate: Choosing protocols and metrics, which can be either Automated (using mathematical metrics or an LLM as a judge) or Human (human-in-the-loop quality assessment).

The documentation is organized using Sphinx and reStructuredText (.rst).

• docs/source/conf.py: The main configuration file for Sphinx (extensions, themes, project info).
• docs/source/index.rst: The root document and homepage. It uses the toctree directive to define the hierarchy.
• docs/source/get_started/: Contains installation and quickstart guides.
• docs/source/tutorials/: Contains main tutorials.
• docs/source/use_cases/: Community-contributed use cases.
• docs/source/apis/: Source-code-related documentation generated via autodoc.
• docs/build/: The directory where generated HTML files reside. Do not commit this directory to version control.

The ReAct (Reasoning and Acting) agent in AdalFlow follows a sequential paradigm of interleaving three types of steps:

1. Thought: The reasoning behind taking a specific action.
2. Action: Selecting an action from a predefined set of tools.
3. Observation: The execution result of the action (typically a string) provided back to the agent to inform the next step.

Agents are implemented as generators that can use tools to complete user queries through multiple sequential or parallel steps.

In AdalFlow, an Agent is not a model or an LLM itself. Instead, it is a system that utilizes LLM models to plan and replan steps. These steps involve using various tools (such as function calls or other LLM-based models) based on context and history (memory) to complete tasks autonomously.

AdalFlow's implementation of the ReAct pattern is highly flexible and supports:

• Multi-hop reasoning: Dividing a query into subqueries and answering them sequentially.
• Flexible Tool Usage: Any function, such as a Retriever or Generator module, can be wrapped as a tool. This includes using an LLM as a tool to leverage world knowledge.
• Task Completion: A special 'finish' tool is used to conclude the task by joining all subquery answers.

AdalFlow incorporates several general design patterns for agents, inspired by industry research:

1. Reflection: Includes patterns like Self-Refine (iterative refinement with self-feedback) and Reflexion (language agents with verbal reinforcement learning).
2. Tool Use: Enables LLMs to connect with massive APIs (e.g., Gorilla) or perform multimodal reasoning and action (e.g., MM-REACT).
3. Planning: Utilizes reasoning techniques like Chain-of-Thought and frameworks like HuggingGPT or ReAct to solve complex tasks.
4. Multi-agent Collaboration: Supports systems where multiple agents communicate to solve problems (e.g., AutoGen or Communicative Agents).

AdalFlow's retriever design focuses on two main areas: high-precision retrieval methods that work locally/in-memory for efficient testing, and integration with cloud databases for large-scale data using built-in search and filter methods.

Supported retriever methods include:

• LLMAsRetriever
• Reranker (Cross-encoder)
• Semantic Search (Bi-encoder)
• BM25
• Database built-in search (e.g., full-text search/SQL-based search using Postgres or semantic search using PgVector).

Retrievers are designed to work alongside LocalDB or cloud SQL-based databases (via SQLAlchemy), specifically supporting Document and DialogTurn data models.

AdalFlow is built on two fundamental base classes designed for maximum customizability with minimum abstraction:

• Component: Used to define the task pipeline.
• DataClass: Used for data interaction with LLMs.

This modular approach allows developers full control over prompt templates, models, and output parsing.

AdalFlow provides a unified framework for optimizing LLM workflows using PyTorch-inspired patterns. Key components for optimization include:

• Parameter: Define trainable elements of your pipeline (e.g., prompts or demonstrations).
• Generator: The engine used to optimize parameters.
• PROMPT: Used for prompt tuning via textual gradient descent.
• DEMO: Used for few-shot bootstrap optimization.
• AdalComponent: Acts as the interpreter between the task pipeline and the trainer, managing training/validation steps, optimizers, evaluators, loss functions, and the backward engine for textual gradients.

The AdalFlow repository is organized into several key directories:

• /adalflow: Contains the core library source code, implementation, tests, and its own pyproject.toml.
• /docs: Houses the library documentation (written in reStructuredText) and includes its own pyproject.toml.
• /tutorials: Contains tutorials for each core feature or class.
• /use_cases: Covers end-to-end applications, including those with auto-optimization.
• /benchmarks: Includes benchmarks for evaluating the library against other frameworks.
• /notebooks: Contains all notebooks used across tutorials, use cases, and benchmarks.

AdalFlow uses two primary base classes to build pipelines:

1. Component: The fundamental building block used to construct modular LLM workflows.
2. DataClass: Used to define structured data interactions with LLMs, typically by inheriting from dataclasses.dataclass and adalflow.core.DataClass.