LazyLLM

Flow defines the data stream, describing how data passes from one callable object to another. Flows allow you to organize complex applications using predefined patterns. Supported Flow types include:

• Pipeline
• Parallel
• Diverter
• Warp
• IFS
• Loop

Flows enable easy combination of modules, reduce repetitive data transformation work, and support asynchronous/parallel execution for performance.

LazyLLM uses an inheritance-as-registration mechanism to manage extensible components like Flows, Nodes, Tools, Models, and Launchers. Instead of manual registration calls (e.g., registry.register(...)), registration happens automatically at class definition time by parsing inheritance relationships and naming conventions.

Key Principles:

• Inheritance-driven: Inheriting a Base class explicitly declares which capability system the class belongs to.
• Automatic: Registration occurs during class definition, not via manual triggers.
• Hierarchical: Supports multi-level capability structures (e.g., group.subgroup.xxx).
• Unified Access: Components are accessed via a stable, dot-notation interface (e.g., lazyllm.online.chat(...)) regardless of internal file structures.

LazyLLM unifies all datasets as List[dict]. The dataset size is the length of the list, and each dictionary represents a single data element.

There are two primary processing modes:

1. Single Data Processing Operator: Processes one dict at a time. The framework handles concurrency automatically. The return type determines the behavior:
   • Returns dict: Replaces the original data.
   • Returns List[dict]: Adds multiple new entries.
   • Returns None: Keeps the original data reference.
   • Returns List (empty): Deletes the data.
2. Full Data Processing Operator: Processes the entire List[dict] at once. The framework calls these sequentially without automatic concurrency; users must implement their own concurrency if needed (e.g., for deduplication).

LazyLLM applications are not static graphs. They are built using standard Python syntax, allowing for high flexibility and dynamic behavior. Key architectural characteristics include:

• Traditional Programming Style: Applications are constructed using basic Python syntax, making them feel like traditional programming rather than rigid graph definitions.
• Runtime Topology Changes: You can use Python's features to modify the application's topology (the structure of how modules connect) during runtime. Subsequent executions will follow the newly modified structure.
• Dynamic Hook Injection: You can inject hook functions at the connection points between modules. These hooks can even be defined at runtime to intercept or modify data flow.

The Reranker component performs a re-ranking process to refine the order of retrieved documents, ensuring that the most relevant documents appear at the top of the returned list. This is typically used in Retrieval-Augmented Generation (RAG) workflows to improve retrieval quality.

reranker = lazyllm.Reranker(
    name='ModuleReranker',
    model=lazyllm.OnlineEmbeddingModule(type="rerank"),
    topk=1
)

# Re-rank multiple retrieved results
doc_node_list = reranker(
    nodes=doc_node_list_1 + doc_node_list_2,
    query="用户问题"
)

To consume streaming outputs, wrap your model or pipeline with lazyllm.StreamCallHelper. This allows you to iterate over the model's output using a for loop.

If the model is part of a pipeline (Flow), wrap the outermost pipeline object instead of the model itself.

# For a standalone model
model = lazyllm.TrainableModule('qwen2-1.5b', stream=True)
model = lazyllm.StreamCallHelper(model)
for msg in model('hello'):
    print(msg)

# For a model within a pipeline (Flow)
model = lazyllm.TrainableModule('qwen2-1.5b', stream=True)
ppl = lazyllm.pipeline(model)
ppl = lazyllm.StreamCallHelper(ppl)
for msg in ppl('hello'):
    print(msg)

To build intelligent agents, install the lazyllm package and the httpx dependency for handling HTTP requests.

pip install lazyllm
pip install httpx

You can build a multi-agent dialogue system using a 'director-style scheduling' pattern. This involves using a DirectorDialogueAgent to control speaking order and conversation termination, while other DialogueAgent instances participate as controlled roles.

Key components include:

• DialogueAgent: Defines individual roles with independent memory (message_history) and persona (system_message).
• DirectorDialogueAgent: Acts as the central controller to select the next speaker and decide when to end the conversation.
• ChatPrompter: Used to build reusable prompt templates for different stages (e.g., continuing discussion, selecting speakers, or summarizing).
• OnlineChatModule: Used to simulate the multi-agent conversation flow.

from lazyllm import OnlineChatModule, ChatPrompter