DOCSEARCHbyAlgolia
Submit Docs

TableGPT Agent

repository·main·Indexed 20 days ago

https://github.com/tablegpt/tablegpt-agent

A Langgraph-based agentic interface for TableGPT2, a multimodal model optimized for tabular data reasoning and question answering. It features an iterative code generation and execution workflow using an IPython sandbox, support for visual analysis via VLMs, and a safety mechanism to filter hazard categories. The package includes a Code Sandbox for isolated execution in local, remote, and cluster environments, as well as evaluation frameworks for benchmarks like RealTabBench, bird, and spider.

Tokens
20.9K
Snippets
62
Records
81
Agent score
70%

What's inside tablegpt-agent

  1. Overview of TableGPT Agent

    main
    TableGPT Agent is a pre-built agent designed for interacting with TableGPT2, a series of Large Multimodal Models (LMMs) specialized for table-based question answering. The agent is built using the Langgraph library, providing a structured interface for complex tabular reasoning tasks.

  3. What is Code Sandbox and how does it work?

    main

    Code Sandbox is a security layer for tablegpt-agent that isolates the execution of generated Python code. Since the agent generates code for data analysis, running it directly in production poses security and stability risks. Code Sandbox uses the pybox library to confine execution to a controlled, isolated environment.

    It supports three execution modes:

    1. Local Environment: Uses pybox to run code in a local sandbox. Best for development and debugging.
    2. Remote Environment: Uses RemotePyBoxManager and Jupyter Enterprise Gateway to create shared computing environments.
    3. Cluster Environment: Uses KubePyBoxManager to communicate directly with kernel pods in a Kubernetes cluster, bypassing proxy services like Enterprise Gateway.

  4. How the TableGPT Agent workflow works

    main

    The tablegpt-agent operates as a single-agent system that processes user queries through an iterative loop of code generation and execution. The core components are an agent powered by TableGPT2 and an IPython tool that executes code in a sandbox.

    Core Workflow Steps

    1. User Input: Receives the query.
    2. Security Assessment (optional): Evaluates the query for sensitive topics.
    3. Data Retrieval (optional): Uses a retriever plugin to fetch dataset metadata (columns, values, etc.).
    4. Code Generation: The agent writes Python code to solve the task.
    5. Code Execution: The code runs in an IPython sandbox.
    6. Result Generation: The agent processes execution results into a final response.
    7. Visual Analysis (optional): Uses a Visual Language Model (VLM) to summarize generated charts/images.

    Iterative Debugging

    If code execution fails, the system automatically enters an iterative debugging cycle. It repeats steps 4 through 7 (Code Generation $\rightarrow$ Result Generation) to resolve errors. The default maximum iteration limit is 25.

  5. How Incluster Code Execution works in tablegpt-agent

    main
    When using tablegpt-agent, the agent instructs tablegpt to generate Python code specifically for data analysis tasks. To maintain system security, this generated code is not executed in the main process. Instead, it is run within a sandboxed environment managed by the pybox library. This ensures that any data analysis code generated by the model is isolated from the core application process.

  6. Understand the TableGPT Evaluation dataset structure

    main

    In the context of TableGPT evaluation, a 'dataset' consists of samples used for testing. Each sample requires an input field representing the user query. For data analysis tasks, samples should also include an expected output field to serve as the ground truth for comparison.

    Note that the dataset being analyzed is referred to as 'reference data' to distinguish it from the evaluation samples. When generating expected output, ensure high accuracy as it is used as the ground truth for evaluation results.

  8. Install the TableGPT Evaluation environment

    main

    To set up the evaluation environment, create a virtual environment and install the required dependencies using pip.

    # Create and activate a virtual environment
python -m venv venv
source ./venv/bin/activate  # On Windows, use `.\venv\Scripts\activate`

# Install dependencies
pip install -r requirements.txt
    python -m venv venv
source ./venv/bin/activate
pip install -r requirements.txt

  9. Run the evaluation script

    main

    To execute an evaluation, you must provide a configuration file and ensure all necessary environment variables are set (either via exported shell variables or a .env file in the root directory). Run the evaluation using the agent_eval module.

    python -m agent_eval --config path/to/your/config.yaml
    python -m agent_eval --config path/to/your/config.yaml

  10. Run code in a Remote Sandbox environment

    main

    Remote mode uses RemotePyBoxManager to connect to a Jupyter Enterprise Gateway service. This allows multiple services to share access to the same remote environment.

    Prerequisites:

    • An active enterprise_gateway service.
    • The service address must be accessible (e.g., http://example.com).

    Advanced Configuration: RemotePyBoxManager supports two configuration options for environment variables:

    • env_file: Path to a file containing environment variables.
    • kernel_env: A dictionary of key-value pairs for environment variables.
    from uuid import uuid4
from pybox import RemotePyBoxManager, PyBoxOut

# Initialize the remote sandbox manager, replacing with the actual Enterprise Gateway service address
pybox_manager = RemotePyBoxManager(host="http://example.com")

# Assign a unique Kernel ID
kernel_id = str(uuid4())

# Start the remote sandbox environment
box = pybox_manager.start(kernel_id)

# Define the test code
test_code = """
import math
result = math.sqrt(16)
result
"""

# Run the code in the sandbox
out: PyBoxOut = box.run(code=test_code)

# Print the execution result
print(out)

  11. Run code in a Kubernetes Cluster environment

    main

    In a Kubernetes cluster, KubePyBoxManager communicates directly with Kernel Pods created by the jupyter-kernel-controller, removing the need for an intermediary gateway.

    Prerequisites:

    • The jupyter-kernel-controller service must be deployed in your cluster.

    Configuration: KubePyBoxManager uses the same environment configuration parameters as RemotePyBoxManager:

    • env_file: Path to an environment variable file.
    • kernel_env: A dictionary of environment variables.
    from uuid import uuid4
from pybox import KubePyBoxManager, PyBoxOut

# Initialize the cluster sandbox manager
pybox_manager = KubePyBoxManager(
    env_file="YOUR_ENV_FILE_PATH",  # Path to the environment variable file
    kernel_env="YOUR_KERNEL_ENV_DICT",  # Kernel environment variable configuration
)

# Assign a unique Kernel ID
kernel_id = str(uuid4())

# Start the cluster sandbox environment
box = pybox_manager.start(kernel_id)

# Define the test code
test_code = """
import math
result = math.sqrt(16)
result
"""

# Run the code in the sandbox
out: PyBoxOut = box.run(code=test_code)

# Print the execution result
print(out)

  12. Run table-related benchmark evaluations

    main
    The repository includes evaluation scripts and datasets for various academic and table-related benchmarks. These resources are located in the realtabbench directory. To perform evaluations, refer to the specific instructions in the realtabbench/README.md file.