DOCSEARCHbyAlgolia
Submit Docs

LOTUS Framework Documentation

repository·main·Indexed 23 days ago

https://github.com/lotus-data/lotus

LOTUS is a framework for optimized agentic and LLM bulk processing of structured and unstructured data. It provides a Pydantic-based Abstract Syntax Tree (AST) for building immutable data pipelines using semantic operators (map, reduce, filter, join, search) and pandas operations. The framework includes an execution engine with content-addressable caching, a GEPAOptimizer for LLM-guided evolutionary tuning of natural language instructions, and a PredicatePushdownOptimizer to improve efficiency.

Tokens
44.8K
Snippets
124
Records
180
Agent score
81%

What's inside LOTUS

  1. What is a Corpus in LOTUS?

    main

    A Corpus is the fundamental input for all agentic operators in LOTUS. It acts as a normalization layer that converts various data formats (in-memory strings, files, DataFrames, or large text blocks) into a stream of units. These units can then be sharded into batches for parallel processing by agents.

    Each unit is an atomic segment represented by a Unit dataclass containing:

    • id: A stable identifier (e.g., a file path or row index).
    • content: The text content visible to the agent.
    • metadata: A dictionary containing loader-specific information (e.g., file path, row number, or chunk index).
    @dataclass
class Unit:
    id: str                 # stable identifier (e.g. a file path or row index)
    content: str            # the text the agent sees
    metadata: dict          # loader-specific extras (path, row number, chunk index)

  2. How cache isolation works during evaluation

    main
    To ensure evaluation accuracy, LOTUS automatically disables operator caching while judge calls are running. This prevents the system from returning previously cached judgments for repeated trials. The original cache settings are automatically restored once the evaluation process completes.

  3. Understand `llm_as_judge` output columns

    main

    LOTUS generates new columns in the resulting DataFrame based on the trials performed. By default, columns are named using the pattern {suffix}_{trial} (e.g., _judge_0 if the suffix is _judge).

    Additional output options:

    • Set return_raw_outputs=True to add raw_output{suffix}_{trial} columns.
    • Set return_explanations=True to add explanation{suffix}_{trial} columns.

  4. Handle long context in sem_agg

    main

    When the input documents exceed the language model's context length, you can specify a long_context_strategy using lotus.types.LongContextStrategy.

    Strategies:

    • LongContextStrategy.TRUNCATE: (Default) Simply cuts off excess content at the token limit and appends "...". Use this when the most important information is at the beginning of documents.
    • LongContextStrategy.CHUNK: Intelligently identifies the largest column and splits it to preserve information. Use this when all parts of the document are potentially important.

    Note: While the documentation mentions TRUNCATE is default, the parameter description says CHUNK is default. Always verify your specific version's behavior.

    from lotus.types import LongContextStrategy

# Use TRUNCATE strategy
result_truncate = df.sem_agg(
    "Summarize the key points from {content}",
    long_context_strategy=LongContextStrategy.TRUNCATE
)

# Use CHUNK strategy
result_chunk = df.sem_agg(
    "Summarize the key points from {content}",
    long_context_strategy=LongContextStrategy.CHUNK
)

  5. Compose filter with map and reduce operations

    main

    The filter operator transforms a Corpus into a Corpus (Corpus → Corpus), allowing it to be chained in a pipeline. You can use filter to narrow down a corpus before passing the survivors to map or reduce operations.

    When filter is used as part of a multi-op pipeline, the result.output will contain the final reduced value (if reduce is present), while result.corpus will contain the units that survived the filtering stage.

    result = corpus.agent(
    task="Keep only functions with a bug, then write one summary of the bugs found.",
    ops=["filter", "reduce"],
    tools=[PythonREPLTool()],
)
print(result.output)     # a summary over only the units that survived the filter

  6. Understand the Predicate Pushdown Optimization

    main

    The PredicatePushdownOptimizer aims to reduce the number of rows processed by expensive LLM-based semantic operators by moving pandas filters earlier in the pipeline.

    Algorithm: It identifies PandasFilterNodes and bubbles them backward past consecutive SemFilterNodes.

    Safety Invariant: Because SemFilterNode only removes rows and never renames or adds columns, a pandas filter that depends on existing columns can safely be moved before the semantic filter.

  7. Control aggregation order with sem_partition_by

    main

    The sem_partition_by utility provides fine-grained control over how data is processed during semantic aggregation (sem_agg). It allows you to assign a partition number to each row in a DataFrame.

    During semantic aggregation, LOTUS aggregates over each partition separately before combining the intermediate results. The final combination follows the order of the partition numbers in increasing order. This is useful for tasks like summarization where LLM performance is sensitive to input ordering.

    By default, LOTUS uses a hierarchical reduce strategy assuming all records belong to the same partition. Using sem_partition_by overrides this behavior.

  8. How the GEPAOptimizer tunes instructions

    main

    The GEPAOptimizer uses LLM-guided evolutionary search to automatically tune natural language instructions for semantic nodes.

    Supported Parameters:

    • SemFilterNode: user_instruction, cascade_args.helper_filter_instruction
    • PairwiseJudgeNode: judge_instruction, cascade_args.helper_filter_instruction
    • SemMapNode: user_instruction
    • SemAggNode: user_instruction
    • SemTopKNode: user_instruction
    • SemJoinNode: join_instruction
    • SemSearchNode: query

    Optimization Process:

    1. Target Collection: Traverses the tree to find _OptTarget instances.
    2. Seed Candidate: Uses current values as the starting point.
    3. Evaluator: Executes the pipeline on example data and scores it using a user-provided eval_fn(output_df, example).
    4. Evolutionary Search: Runs mutation/reflection cycles to find better instructions.
    5. Application: Patches the original nodes with the best discovered values.

  9. Understand LOTUS evaluation caching behavior

    main
    During evaluation calls, LOTUS temporarily disables operator caching inside the judge loop. This ensures that repeated trials (such as those in pairwise_judge with n_trials > 1) produce independent judgments rather than returning cached results. The global cache setting is automatically restored once the evaluation call completes.

  10. Use Cascade Mode for cost-efficient comparisons

    main

    The pairwise_judge method supports a "Cascade Mode" via cascade_args. This uses semantic filtering to perform lower-cost comparisons, only escalating to expensive models when necessary.

    When using return_stats=True in cascade mode, the method returns a tuple of (DataFrame, stats) instead of just a DataFrame.

    from lotus.types import CascadeArgs

cascade_args = CascadeArgs(
    recall_target=0.9,
    precision_target=0.9,
    sampling_percentage=0.5,
    failure_probability=0.2,
)

results, stats = df.pairwise_judge(
    col1="model_a",
    col2="model_b",
    judge_instruction="Which response better answers {question}?",
    cascade_args=cascade_args,
    return_stats=True,
)

  11. How agentic map-reduce operators work in LOTUS

    main

    LOTUS agentic operators allow you to process a Corpus using a task and an ordered list of operations (map, filter, or reduce).

    1. Planning: A planner derives instructions, sharding, and strategies for each operation.
    2. Execution: Each unit is processed in parallel by a tool-using agent equipped with a sandboxed Python REPL.
    3. Tool Usage: Tool usage is handled transparently; the user-provided task does not need to explicitly mention tools.

    Operation Outputs:

    • map and filter operations return a new Corpus.
    • reduce operations return a single answer.

    Common patterns include:

    • map → reduce: For aggregations (e.g., calculating per-item totals then a grand total).
    • map → reduce: For analysis (e.g., per-file summaries then an architecture overview).
    • filter: For complex selection tasks that require code execution (e.g., identifying buggy code via REPL).

  12. Understand the LazyFrame API concept

    main

    A LazyFrame is LOTUS's lazy execution API for semantic operator programs. Instead of executing operations immediately (like pandas), a LazyFrame allows you to define a complete logical plan of semantic and pandas operations first. Nothing is executed until you call .execute().

    Why use LazyFrame?

    Using a LazyFrame provides a planning boundary that allows LOTUS to optimize the entire pipeline before making expensive LLM calls. Benefits include:

    • Inspection: View the full plan of semantic and pandas operations before running it.
    • Optimization: Move cheap pandas filters before expensive semantic filters and optimize prompts across the whole pipeline.
    • Efficiency: Pre-learn cascade thresholds so cheaper models can handle easy rows.
    • Persistence: Save an optimized pipeline (including learned thresholds) and reuse it later.

    When to use LazyFrame vs Eager execution

    • Eager execution: Best for data exploration where you want immediate feedback after each operator.
    • LazyFrame: Best for multi-step LLM programs where you want to optimize the global plan for cost and performance.