DOCSEARCHbyAlgolia
Submit Docs

ShinkaEvolve Documentation

repository·main·Indexed 20 days ago

https://github.com/sakanaai/shinkaevolve

ShinkaEvolve is a framework for automated scientific evolution that combines Large Language Models (LLMs) with evolutionary algorithms to discover and optimize scientific code. It utilizes LLMs as mutation operators to evolve populations of programs across generations. The framework supports multi-language optimization, including Python, Fortran, Go, Julia, and Verilog (via RTLLM for PPA optimization), providing tools for asynchronous evolution, program evaluation, and performance analysis.

Tokens
35.3K
Snippets
81
Records
188
Agent score
79%

What's inside ShinkaEvolve

  1. Overview of RTLLM × ShinkaEvolve PPA Optimization

    main

    This project uses ShinkaEvolve to optimize Verilog designs for PPA (Power, Performance, and Area) while keeping the functional specification fixed.

    Instead of a binary pass/fail correctness score, the fitness function is a continuous score based on the geometric mean of speedups over a human-designed reference:

    score = 100 · geomean( area_ref/area_cand , depth_ref/depth_cand , power_ref/power_cand )

    A reference design scores 100. A correct implementation that is smaller, faster, or lower-power will score > 100.

    Metrics measured:

    • Area (µm²): Measured via Yosys using Nangate45 standard cells (stat -liberty).
    • Performance (logic depth): Measured via Yosys longest topological path (ltp).
    • Power (µW): Measured via OpenSTA on the gate-level netlist (report_power).
    • Correctness: Verified via Icarus Verilog (testbench) and Yosys formal equivalence (equiv/sat) to ensure the candidate matches the reference function.

  2. What is ShinkaEvolve?

    main

    ShinkaEvolve is a framework developed by SakanaAI that combines Large Language Models (LLMs) with evolutionary algorithms. It works by proposing program mutations (patches, full rewrites, or crossovers) which are then evaluated and archived. The goal is to optimize code performance and discover novel scientific insights through an iterative evolution flow.

    Evolution Flow

    1. Selection: Select parent(s) from the archive or population.
    2. Mutation: LLM proposes a patch (diff, full rewrite, or crossover).
    3. Evaluation: Evaluate the candidate to determine its combined_score.
    4. Archiving: If valid, insert into the island archive (higher scores are prioritized).
    5. Migration: Periodically migrate top solutions between islands.
    6. Iteration: Repeat for $N$ generations.

  3. Overview of the ShinkaEvolve API surface

    main

    The ShinkaEvolve API is organized into several core modules that users compose to run evolution experiments. The primary modules are:

    • Core Runtime: Contains the main execution logic including EvolutionConfig, ShinkaEvolveRunner, and the run_shinka_eval entry point.
    • Database: Manages data persistence via DatabaseConfig, Program, ProgramDatabase, and prompt databases.
    • Launch: Handles job execution environments using LocalJobConfig, SLURM configurations, and the JobScheduler.
    • LLM: Provides interfaces for interacting with Large Language Models via LLMClient and AsyncLLMClient, including query helpers and model prioritization.
    • Embeddings: Manages vector representations through EmbeddingClient and AsyncEmbeddingClient with backend resolution.

    Note: For details on configuration layering and Hydra presets, refer to the Configuration documentation, as the API reference focuses on the runtime surface.

  4. How controlled oversubscription works

    main

    Controlled oversubscription is an adaptive mechanism used when proposal generation is slower than evaluation. It allows Shinka to maintain a small backlog of proposals to ensure evaluation workers are never idle, without creating an unbounded queue.

    Key Concept: Oversubscription never increases evaluation concurrency; max_evaluation_jobs remains the hard cap for evaluations. These settings only control how many proposal/sampling jobs are kept in flight ahead of the evaluation workers.

    Recommended Settings by Scale

    Scalemax_evaluation_jobsmax_proposal_jobsNotes
    Sequential-like1-41sync-like proposal behavior
    Small2-6eval + 1good default if eval waits on proposals
    Medium5-20eval + 1 to eval + 2use adaptive oversubscription
    Large20+eval + 2 to eval + 6keep bounded with caps

  6. Understand the Prime Counting evaluation metric

    main

    The evaluator uses a combined_score to drive the evolution process. The goal is to maximize accuracy while minimizing runtime.

    Metric Formula: combined_score = max(0.0, 100.0 * accuracy - runtime_seconds)

    Components:

    • accuracy: The exact-match rate across 10 fixed prime-count queries. Correctness is the primary driver; any incorrect answers reduce the score sharply.
    • runtime_seconds: A penalty term applied to the score once correctness is high. The theoretical maximum score is 100.0 (perfect accuracy with zero runtime overhead).

  7. Understand configuration precedence in shinka_run

    main

    When using shinka_run, configuration values are resolved based on the following priority (from lowest to highest):

    1. --config-fname YAML file
    2. --set overrides
    3. --results_dir (This flag always sets evo.results_dir)
    4. --num_generations (This flag always sets evo.num_generations)

  9. Inspect Evolution Results

    main

    Each run generates artifacts in a results directory, including:

    • Persisted program records and metrics.
    • Candidate code snapshots and diffs.
    • Timing metadata.
    • Prompt-evolution artifacts (if enabled).

    Results can be inspected using the built-in WebUI, notebooks in the examples/ directory, or custom post-processing scripts.

  10. Understand the configuration resolution order

    main

    Configuration values in shinka-evolve are resolved using a hierarchy where later definitions override earlier ones. The order of precedence (from lowest to highest) is:

    1. Dataclass defaults defined in the source code (EvolutionConfig, DatabaseConfig, or JobConfig classes).
    2. Hydra preset YAMLs located in shinka/configs/.
    3. Hydra composition overrides (task, cluster, or variant specific).
    4. CLI overrides via shinka_launch ... key=value or shinka_run --set ....
    5. Authoritative shinka_run flags such as --results_dir and --num_generations.

  11. Understand the Wolfram GCD-Sum optimization metric

    main

    The optimization goal is to maximize the combined_score. The evaluator uses the following logic:

    • Correctness Check: The candidate must return the correct integer (336784). If it does not, the score is -1.0.
    • Score Calculation: If correct, combined_score = baseline_time_ms / median(candidate_time_ms).
    • Baseline Calibration: The baseline_time_ms is calibrated on every run by timing the deoptimized seed (initial.wl) through the same RepeatedTiming inside TimeConstrained harness used for candidates. This ensures the baseline and candidate are measured identically on the host machine.

  12. Configure Controlled Oversubscription

    main

    If proposal generation is slower than evaluation, you can enable Controlled Oversubscription to keep extra proposal tasks in flight, preventing evaluation workers from idling.

    When enabled:

    • max_evaluation_jobs still caps evaluation concurrency.
    • max_proposal_jobs becomes the hard ceiling for proposal generation.
    • The controller increases the proposal target only when sampling_seconds > evaluation_seconds.
    • Oversubscription is bounded by proposal_buffer_max, proposal_target_ratio_cap, proposal_target_hard_cap, and max_proposal_jobs.

    Recommended starting configuration:

    max_evaluation_jobs: 5
max_proposal_jobs: 7
max_db_workers: 2

evo_config:
  enable_controlled_oversubscription: true
  proposal_target_mode: adaptive
  proposal_target_min_samples: 5
  proposal_target_ratio_cap: 2.0
  proposal_buffer_max: 2
  proposal_target_hard_cap: 7
  proposal_target_ewma_alpha: 0.3