TextGrad

When using the code optimization evaluation suite, be aware of the following constraints:

• Result Instability: Minor changes in code snippets can lead to varying results. It is recommended to run optimizations multiple times with different seeds and report averages.
• Platform Issues: Interfacing with the LeetCode platform via python-leetcode may result in timeouts, network issues, or rate limiting.
• Character Constraints: The LeetCode API may reject programs containing certain characters (like parentheses or brackets wrapping Python statements). The evaluation pipeline attempts to mitigate this by using GPT-4 to clean the code.
• Data Source: The evaluation uses data from LeetCodeHardGym. The dataset used is leetcode_with_tests.jsonl.

When using the code optimization evaluation tasks, be aware of the following behaviors:

• Result Instability: Code optimization results can be sensitive to minor changes in code snippets. To mitigate this, it is recommended to run optimizations multiple times with different seeds and report average results.
• LeetCode API Issues: The evaluation relies on pushing code to the LeetCode platform, which is subject to timeouts, network issues, and rate limiting.
• Character Constraints: The underlying LeetCode API (via python-leetcode) may reject programs containing certain characters like parentheses or brackets wrapping Python statements. The evaluation pipeline attempts to mitigate this by using GPT-4 to clean these characters.
• External Dependencies: The evaluation utilizes LeetCodeHardGym and the python-leetcode package.

You can run solution optimization experiments for multimodal benchmarks like MathVista and ScienceQA using the solution_optimization_mm.py script. The script allows you to specify the task, the LLM engine used for generation, the evaluation engine, the number of optimization iterations, and concurrency settings.

MathVista Experiment

To run an experiment on MathVista:

cd evaluation
python solution_optimization_mm.py --task mathvista \
--engine=gpt-4o \
--eval_engine=gpt-4o \
--max_iterations 4 \
--num_threads 10 \
--majority_voting

ScienceQA Experiment

To run an experiment on ScienceQA:

cd evaluation
python solution_optimization_mm.py --task scienceqa \
--engine=gpt-4o \
--eval_engine=gpt-4o \
--max_iterations 8 \
--num_threads 20

To use TextGrad for optimizing the Chameleon agentic system, set up a new Conda environment and install the required Python packages and system dependencies.

Python Packages:

• textgrad (installed in editable mode)
• matplotlib
• easyocr

System Dependencies:

• graphviz (required via apt)

conda create --name textgrad
conda activate textgrad
cd textgrad
pip install -e .
pip install matplotlib
pip install easyocr

sudo apt install graphviz

For users looking to replicate the code optimization evaluations, the LeetCodeHard dataset used in this project is available via the following external link:

leetcode_with_tests.jsonl

To use TextGrad, import the core components: get_engine, Variable, TextualGradientDescent, and TextLoss. Ensure you have an OPENAI_API_KEY set in your environment variables.

from textgrad.engine import get_engine
from textgrad import Variable
from textgrad.optimizer import TextualGradientDescent
from textgrad.loss import TextLoss
import os

# Ensure API key is set
os.environ["OPENAI_API_KEY"] = "your-key-here"

The PythonEvaluator is used to run model-generated code for evaluation.

Security Warning: This program executes untrusted model-generated code. It is strongly recommended to run this only within a robust security sandbox.

By default, the evaluation scripts (py_eval.py and utils.py) are commented out and configured to raise exceptions to prevent accidental execution of untrusted code. You must manually edit these files to enable execution.

TextGrad can be used to optimize the long chain of reasoning steps in the Chameleon agentic system through an interactive self-improvement loop. Chameleon is a system capable of planning, executing, and reasoning about tool sequences to answer questions.

A practical implementation example is available in the notebook: TextGrad_Chemeleon_ScienceQA.ipynb.

To perform optimization, follow the standard autograd pattern:

1. Compute the loss: l = loss(x)
2. Perform the backward pass: l.backward(engine)
3. Update parameters: optimizer.step()

To prevent gradient accumulation across steps, call optimizer.zero_grad() at the start of each iteration.

# Single optimization step
l = loss(x)
l.backward(engine)
optimizer.step()

# Loop pattern
for i in range(steps):
    optimizer.zero_grad()
    l = loss(x)
    l.backward(engine)
    optimizer.step()

The optimization process follows a pattern similar to PyTorch:

1. Forward Pass: Call your loss function to get the loss variable.
2. Backward Pass: Call loss.backward() to compute textual gradients.
3. Update: Call optimizer.step() to update the variables.
4. Reset: Call optimizer.zero_grad() before the next iteration to clear old gradients.

# Forward pass
loss = loss_fn(problem, code)

# Backward pass
loss.backward()

# Update parameters
optimizer.step()

# Next iteration
optimizer.zero_grad()
loss = loss_fn(problem, code)
loss.backward()
optimizer.step()