DOCSEARCHbyAlgolia
Submit Docs

VerilogEval Documentation

repository·main·Indexed 19 days ago

https://github.com/nvlabs/verilog-eval

An evaluation harness for benchmarking Large Language Models (LLMs) on Verilog code generation. It supports code completion (code-complete-iccad2023) and specification-to-RTL tasks. The tool requires iverilog v12 and Python 3.11.0, utilizing a Makefile-driven workflow for configuration and execution.

Tokens
1K
Snippets
2
Records
4
Agent score
16%

What's inside VerilogEval

  1. Understand the differences in VerilogEvalV2

    main

    VerilogEvalV2 (released August 2024) introduces several changes compared to the original 1.0 version:

    • Task Scope: Includes specification-to-RTL tasks in addition to the original code completion tasks.
    • Prompting: Prompts are reframed to specify module interfaces more abstractly rather than explicitly including the Verilog module interface.
    • Data Management: The dataset is managed as plain text files instead of a single large JSONL file.
    • In-Context Learning: Supports adding in-context learning examples (0-4) via the --with-examples flag.
    • Analysis: Includes scripts to categorize common iverilog failures and output results in plain text or CSV.
    • Metrics: Pass@10 is no longer reported. Instead, it reports Pass@1 for two configurations:
      • Low temperature: $n=1$ (temperature=0, top_p=0.01)
      • High temperature: $n=20$ (temperature=0.85, top_p=0.95)
    • Limitations: MachineEval is not supported in V2; only Human Eval problem statements are included.

  2. Run the VerilogEval evaluation harness

    main

    The evaluation harness is driven by a Makefile. To run an evaluation, you must first create a build directory, run the configure script with your desired parameters, and then execute make.

    Workflow:

    1. Create a build directory: mkdir -p build/
    2. Configure the run: ../configure --with-task=$task --with-model=$model ...
    3. Execute: make

    You can speed up the evaluation by passing the -j flag to make (e.g., make -j4 to run 4 worker processes).

    Configuration Parameters:

    • --with-task: Specifies the task type. Valid tasks are code-complete-iccad2023 and spec-to-rtl.
    • --with-model: The LLM model to use (refer to scripts/sv-generate for a list of valid models).
    • --with-examples: Number of in-context learning examples (0-4).
    • --with-samples: Number of samples to collect per problem.
    • --with-temperature: Model temperature.
    • --with-top-p: Model top_p value.
    mkdir -p build/
../configure  --with-task=$task --with-model=$model --with-examples=$shots --with-samples=$samples --with-temperature=$temperature --with-top_p=$top_p
make

  3. Setup the Linux environment for VerilogEval

    main

    To use the evaluation harness, you must install specific versions of iverilog, verilator, and python3 along with required Python packages.

    Requirements:

    • iverilog (v12) — Note: iverilog v13 (development release) is NOT supported.
    • python3 (v3.11.0)

    Installation Steps:

    1. Install Python 3.11 using Conda:

      conda create -n codex python=3.11
conda activate codex

    2. Install ICARUS Verilog (v12):

      git clone https://github.com/steveicarus/iverilog.git && cd iverilog \
        && git checkout v12-branch \
        && sh ./autoconf.sh && ./configure && make -j4\
        && make install

    3. Install Python dependencies:

      pip install langchain langchain-openai langchain-nvidia-ai-endpoints
    conda create -n codex python=3.11
conda activate codex

git clone https://github.com/steveicarus/iverilog.git && cd iverilog \
        && git checkout v12-branch \
        && sh ./autoconf.sh && ./configure && make -j4\
        && make install

pip install langchain langchain-openai langchain-nvidia-ai-endpoints

  4. Reference available tasks and parameters in VerilogEvalV2

    main

    VerilogEvalV2 supports two primary task formats. While the underlying problem statements are identical, the task format differs.

    Available Tasks:

    • code-complete-iccad2023: Code completion task.
    • spec-to-rtl: Specification-to-RTL task.

    Configuration Options for ../configure:

    • --with-task: One of the tasks listed above.
    • --with-model: The target LLM (check scripts/sv-generate for supported models).
    • --with-examples: Integer (0-4) representing in-context learning examples.
    • --with-samples: Number of samples per problem.
    • --with-temperature: Floating point temperature.
    • --with-top-p: Floating point top_p value.