Reflexion

Ensure you are using Python 3.7 or later. It is recommended to use a conda environment for isolation.

1. Create and activate a new conda environment:

   conda create -n codex python=3.7
   conda activate codex

2. Clone and install the repository in editable mode:

   git clone https://github.com/openai/human-eval
   pip install -e human-eval

To run decision-making experiments using AlfWorld, navigate to the alfworld_runs directory and configure the parameters in ./run_reflexion.sh before execution.

Configuration Parameters in ./run_reflexion.sh

• num_trials: Number of iterative learning steps.
• num_envs: Number of task-environment pairs per trial.
• run_name: The name for this run.
• use_memory: Whether to use persisting memory to store self-reflections (set to false for baseline runs).
• is_resume: Whether to resume a previous run using the logging directory.
• resume_dir: The logging directory from which to resume.
• start_trial_num: The trial number to start from (if resuming).

Execution

Run the script:

./run_reflexion.sh

Logs are saved to ./root/<run_name>.

git clone https://github.com/noahshinn/reflexion && cd ./alfworld_runs
./run_reflexion.sh

To run reasoning experiments using the HotPotQA dataset, clone the repository, navigate to the hotpotqa_runs directory, install dependencies, and configure your OpenAI API key.

1. Clone and enter the directory:

git clone https://github.com/noahshinn/reflexion && cd ./hotpotqa_runs

2. Install dependencies:

pip install -r requirements.txt

3. Set your OpenAI API key:

export OPENAI_API_KEY=<your key>

To resume a run, use the --is_resume flag. You must provide the --resume_dir (the directory where previous logs exist) and the --start_trial_num (the index of the next trial to execute).

Requirements for Resuming:

1. The directory specified in --resume_dir must exist.
2. The script expects an environment configuration file from the previous trial to exist at: {resume_dir}/env_results_trial_{start_trial_num - 1}.json.

If you encounter the following error message:

malloc: can't allocate region

This indicates the system is running out of RAM. Because memory exhaustion can cause correct programs to fail during execution, it is recommended to free up system memory and retry the evaluation.

Reasoning experiments are executed via notebooks located in ./hotpot_runs/notebooks. The behavior is determined by the selected Agent Type and Reflexion Strategy.

Agent Types

• ReAct: ReAct Agent
• CoT_context: CoT Agent given supporting context about the question
• CoT_no_context: CoT Agent given no supporting context about the question

Reflexion Strategies

Strategies are defined via ReflexionStrategy Enum:

• ReflexionStrategy.NONE: No information about the last attempt is provided.
• ReflexionStrategy.LAST_ATTEMPT: Provides the reasoning trace from the last attempt as context.
• ReflexionStrategy.REFLEXION: Provides the self-reflection from the last attempt as context.
• ReflexionStrategy.LAST_ATTEMPT_AND_REFLEXION: Provides both the reasoning trace and self-reflection from the last attempt as context.

After running experiments, you can persist the results using save_agents and standard file writing. The logs are typically generated using log_react_trial.

# Save the text log
with open(os.path.join(root, 'ReAct', strategy.value, f'{len(agents)}_questions_{trial}_trials.txt'), 'w') as f:
    f.write(log)

# Save the agent objects
save_agents(agents, os.path.join('ReAct', strategy.value, 'agents'))

After running trials, you can save the accumulated log string to a text file and use save_agents to persist the agent instances. The file path typically includes the strategy value and the number of questions/trials.

# Save the text log
with open(os.path.join(root, 'CoT', 'context', strategy.value, f'{len(agents)}_questions_{trial}_trials.txt'), 'w') as f:
    f.write(log)

# Save the agent objects
save_agents(agents, os.path.join(root, 'CoT', 'context', strategy.value, 'agents'))

Execute multiple trials for a set of agents. In each trial, iterate through agents that have not yet produced a correct answer and call agent.run(reflexion_strategy=strategy). Use log_trial and summarize_trial from the util module to track progress and correctness across trials.

# n is the number of trials to run
n = 5
trial = 0
log = ''

for i in range(n):
    for agent in [a for a in agents if not a.is_correct()]:
        agent.run(reflexion_strategy = strategy)
        print(f'Answer: {agent.key}')
    trial += 1
    log += log_trial(agents, trial)
    correct, incorrect = summarize_trial(agents)
    print(f'Finished Trial {trial}, Correct: {len(correct)}, Incorrect: {len(incorrect)}')