Code Sandbox

Code Sandbox is a secure environment designed for running and judging code generated by Large Language Models (LLMs). It provides two primary capabilities:

1. Code Runner: Executes code snippets and returns the results. It supports a wide range of languages including Python (with pytest and GPU support), C++, C#, Go, Java (with junit), NodeJS, Typescript (with jest), Scala, Kotlin, PHP, Rust, Bash, Lua, R, Perl, D, Ruby, Julia, Verilog, and CUDA.
2. Online Judge: An implementation for evaluating and running Reinforcement Learning (RL) datasets. Supported datasets include HumanEval, MultiPL-E HumanEval, CodeContests, MBPP, MBXP, MHPP, CRUXEval, NaturalCodeBench, PAL-Math, and verilog-eval.

The sandbox provides two isolation modes to balance performance and security. Configuration for these modes is managed via the configuration API.

No Isolation

• Best for: High-performance requirements where the code is trusted.
• Behavior: Imposes no restrictions on the executing process. For certain languages, the process UID will be set to non-root.

Light Isolation

• Best for: Executing potentially dangerous code or code that needs to modify system files.
• Requirements: Requires privileged containers to operate.
• Isolation Features:
  • cgroups resource limits (CPU, memory)
  • Network namespace isolation
  • overlayfs + chroot filesystem isolation
  • PID namespace isolation

When running Go code in the sandbox (Version 1.21.6), the system uses two distinct modes depending on whether you are running tests or standard code execution.

In both modes, the sandbox initializes the environment by copying the built-in Go project template from runtime/go to a temporary folder before writing your code.

go_test mode

Used for running tests. The sandbox executes the following command: go test <filename>

go mode

Used for standard code execution. The sandbox performs a two-step process:

1. Compilation: go build -o out <filename>
2. Execution: ./out

# go_test mode
go test <filename>

# go mode
go build -o out <filename>
./out

Prompts are generated using content, labels.context, labels.fewshot, and labels.prompt_template. The latter three can be overridden via config.extra in the request.

If prompt_template is provided, it supports string substitution with the following variables:

• question: The content field.
• fewshot: The labels.fewshot field.
• context: The labels.context field.
• locale: The language used (e.g., en, zh).

In the sandbox service, a dataset is a specific collection of data (e.g., humaneval_python), whereas a dataset type is the underlying Python class that defines the data format, prompt generation logic, code extraction, and evaluation methods (e.g., HumanEvalDataset).

Multiple datasets often share a single dataset type to maximize code reuse. For example:

• HumanEvalDataset type covers humaneval_python, humaneval_cpp, humaneval_typescript, shadow_humaneval_python, and bigcodebench.
• CommonOJDataset type covers code_contests.

The CommonOJ dataset unifies competitive programming problem evaluation using three primary formats:

1. Standard (Supported): The LLM must output complete, executable code. The sandbox evaluates correctness by comparing the program's standard output (stdout) against the expected output under provided standard inputs (stdin).
2. Testlib (In Progress): Designed for problems requiring advanced validation, such as those with multiple valid outputs or permitted round-off errors. It uses specialized Testlib programs to verify correctness.
3. Code Interface (Under Discussion): Intended for platforms like LeetCode or TopCoder where problems require a specific solution class rather than standard stdio input/output.

To optimize performance and reduce compilation time, SandboxFusion uses a specific subset of the Mathlib foundation library sufficient for MiniF2F evaluation. This subset is defined in the project's Main.lean file.

When evaluating Lean code, users should be aware of the result mapping:

• compile_result: Not available for Lean.
• run_result: This is the primary indicator of success/failure, corresponding to the result of the lake build command. If lake build succeeds, the proof is considered correct.

A problem is considered passed if the run_result.return_code from the code execution is 0.

If the return code is non-zero, it indicates an assertion failure or a test tool error.

Preventing Python Hacks:

To prevent models from bypassing tests by explicitly calling exit(0), use the config.extra.append_flag option. When set to true, a command to print a random string is appended to the Python code. The evaluation will fail if this random string is not present in the final output, even if the return code is 0.

The CommonOJDataset type is used for tasks requiring the model to output complete, executable code. This is the standard format for most competitive programming problems.

Key Features:

• Testing Mechanism: It validates the model's output by checking if the stdout produced from a given stdin matches the expected stdout.
• Capabilities: Supports any programming language.

Data Format Requirements:

• id: Unique identifier (string or int).
• content: The problem description. The system automatically includes programming language requirements in the prompt based on this content.
• test (JSON list): A list of test cases, where each element contains:
  • input (JSON object): Contains stdin (Standard input).
  • output (JSON object): Contains stdout (Standard output).
• labels (JSON object): Metadata/labels for the dataset.

The sandbox follows a standard execution flow for every request. Unless otherwise specified, all code is executed in a temporary directory created under /tmp, which is destroyed after execution.

Execution Flow:

1. Create temporary directory
2. Write files passed through files
3. Write code passed through code to a temporary file
4. Set up environment according to the selected isolation mode
5. Execute compilation commands (if required)
6. Execute run commands
7. Retrieve files specified by fetch_files
8. Clean up environment

When using the Python language in the sandbox, you can specify one of two execution modes via the language parameter in the API:

1. python: Executes the code directly using the command python <filename>.
2. pytest: Executes the code as a test suite using the command pytest <filename>.

Neither mode requires a compilation step as they are interpreted directly.