DOCSEARCHbyAlgolia
Submit Docs

Ethereum Execution Spec Tests (EEST)

repository·main·Indexed 42 days ago

https://github.com/ethereum/execution-spec-tests

A test authoring framework used to validate Ethereum execution client implementations. It includes a fuzzer bridge for converting JSON fuzzer outputs (v2.0) into blockchain tests and tools for verifying results against clients like Geth, Besu, and Nethermind. Note: This repository is archived and has migrated to ethereum/execution-specs.

Tokens
88K
Snippets
281
Records
429
Agent score
52%

What's inside Execution Spec Tests

  1. Overview of EEST testing commands: consume and execute

    main

    EEST provides two primary commands for running test cases against Execution Layer (EL) clients:

    1. consume: Runs JSON test fixtures against a client. The client is said to "consume" the test case fixture.
    2. execute: Runs test cases derived from Python source against a client. The test case is "executed" against the client.

    Use consume for fixture-based testing (e.g., JSON files) and execute for running tests via JSON RPC (e.g., eth_sendRawTransaction) against Hive or live networks.

  2. Use the Fuzzer Bridge to convert fuzzer output to blockchain tests

    main

    The Fuzzer Bridge integrates Blocktest fuzzers with the execution-spec-tests framework. It automatically handles complex blockchain test requirements such as state root computations, RLP encoding, block header generation, system contract interactions, and genesis block derivation from fuzzer-generated data.

    To use it, provide fuzzer output in JSON v2 format. The bridge will transform this into valid blockchain test fixtures that can be used to test Ethereum client implementations.

    # Basic conversion
uv run fuzzer_bridge --input fuzzer_output.json --output blocktest.json

  3. Available pytest plugins for execution-spec-tests

    main

    The execution-spec-tests repository uses a custom suite of pytest plugins to extend testing capabilities for the execution specification. When writing or running tests, you may interact with the following plugins:

    • Filler Plugin: Provides utilities for filling data structures or state.
    • Forks Plugin: Manages or simulates state forks during testing.
    • Spec Version Checker Plugin: Ensures that the tests are running against the correct version of the execution specification.

  4. Explore the Execution Spec Tests library components

    main

    The execution-spec-tests repository is composed of several specialized packages designed to simplify the implementation of Ethereum execution spec test cases. Depending on your goal (defining types, using fixtures, or interacting with clients), you should refer to the specific library documentation:

    • Core Types & Exceptions:
      • ethereum_test_base_types: Basic types used as the foundation for all other testing libraries.
      • ethereum_test_types: Ethereum-specific types built on top of the base types for defining test cases.
      • ethereum_test_exceptions: Standardized exception definitions used across all tests.
    • Test Definitions & Specs:
      • ethereum_test_specs: Definitions for spec types used to define test cases and generate test fixtures.
      • ethereum_test_forks: Definitions for the supported Ethereum forks used within the test suite.
      • ethereum_test_vm: Definitions for the Ethereum Virtual Machine (EVM) used to define bytecode in test cases.
    • Fixtures & Tools:
      • ethereum_test_fixtures: Definitions of all test fixture types produced by this repository for client consumption.
      • ethereum_test_tools: Primitives and helpers for testing Ethereum execution clients.
      • ethereum_clis: A wrapper for the transition (t8n) tool.
    • Testing Framework Integration:
      • pytest_plugins: Customizations for pytest that provide enhanced functionality for generating test fixtures.

  5. Understand the repository structure

    main

    The repository is organized into several key directories that separate test definitions, expected outputs (fixtures), and the underlying testing framework logic:

    • tests/: Contains the actual implementation of the Ethereum consensus test cases, organized by type (e.g., eips/, vm/).
    • fixtures/: Stores the default expected output files used for test assertions, categorized by test type (e.g., blockchain_tests/, state_tests/).
    • src/: Contains the core library and framework packages. This includes tools for defining test cases, interfacing with the evm t8n command, and custom pytest extensions used to execute the tests.
    • docs/: Contains all markdown documentation and configuration files.
    • .vscode/: Contains recommended settings and launch configurations for Visual Studio Code users.
    📁 execution-test-specs/
├─╴📁 tests/                     # test cases
│   ├── 📁 eips/
│   ├── 📁 vm/
│   └── 📁 ...
├─╴📁 fixtures/                  # default fixture output dir
│   ├── 📁 blockchain_tests/
│   ├── 📁 blockchain_tests_engine/
│   ├── 📁 state_tests/
│   └── 📁 ...
├─╴📁 src/                       # library & framework packages
│   ├── 📁 ethereum_test_fork/
│   ├── 📁 ethereum_test_tools/
│   └── 📁 ...
├─╴📁 docs/                      # markdown documentation
│   ├── 📁 getting_started
│   ├── 📁 dev
│   └── 📁 ...
├─╴📁 .vscode/                   # visual studio code config
│   ├── 📄 settings.recommended.json # copy to settings.json
│   ├── 📄 launch.recommended.json
│   └── 📄 extensions.json
└── 📄 whitelist.txt             # spellcheck dictionary

  6. What is Hive?

    main
    @ethereum/hive is a containerized testing framework designed to orchestrate test execution against Ethereum clients. It is highly extensible through a module-based system where new test suites are implemented as "simulators" that interact with clients to verify specific behaviors. The Execution Spec Tests (EEST) repository utilizes several of these simulators to validate the execution specification.

  7. Handling failing or invalid transactions

    main

    The behavior of invalid transactions depends on whether you are using a StateTest or a BlockchainTest:

    • StateTest: Transactions included here are expected to be intrinsically valid (e.g., the sender has enough funds for gas, the max fee is $\ge$ the block base fee). However, an intrinsically valid transaction can still revert during execution.
    • BlockchainTest: Blocks can contain intrinsically invalid transactions. If a block contains such a transaction, the entire block (including all other valid transactions within it) is expected to be completely rejected.

  8. Use address stubs to map test contracts to pre-deployed addresses

    main

    When testing on networks with existing contracts (like mainnet, testnets, or bloat-net), you can use the --address-stubs flag to map contract labels used in tests to actual deployed addresses. This avoids the gas and time required to redeploy large contracts and allows testing against real-world state (especially useful on bloat-net).

    How it works: When a test calls pre.deploy_contract(..., stub="NAME"), the framework checks if NAME exists in your stubs. If it does, the framework uses the existing address, skips the deployment transaction, and uses the existing code/state.

    Failure conditions:

    • The stub name is not in the provided list.
    • The address in the list does not contain code on the remote chain.

    Supported formats for --address-stubs:

    • A JSON string.
    • A path to a JSON file.
    • A path to a YAML file.
    # Example of using a stubbed contract in a test
my_stubbed_contract = pre.deploy_contract(code, stub="uniswap")
# Accessing live information from the chain
print(pre[my_stubbed_contract].nonce)
print(pre[my_stubbed_contract].balance)

  9. Use Fork Markers to specify test validity

    main

    Fork markers are used to categorize tests based on which Ethereum hard fork they are valid for. These markers ensure that tests only run during the appropriate network state.

    Available fork markers:

    • @pytest.mark.valid_from("FORK_NAME"): Valid for the specified fork and all subsequent forks.
    • @pytest.mark.valid_until("FORK_NAME"): Valid for the specified fork and all preceding forks.
    • @pytest.mark.valid_at("FORK_NAME_1", "FORK_NAME_2", ...): Valid only for the specific list of forks provided.
    • @pytest.mark.valid_at_transition_to("FORK_NAME"): Valid specifically for the transition to the specified fork.

  10. Understand State Test Fixture Format

    main

    State tests are used to verify the Ethereum Virtual Machine (EVM) state transition function. They define a transaction, a pre-execution state (pre), and a post-execution state (post), ensuring that executing the transaction on the pre-state results in the expected post-state.

    Key characteristics:

    • File Structure: A single JSON file contains multiple test objects. Each key in the JSON object is a unique test name, and the combination of the file path and test name serves as the unique identifier.
    • Multi-vector Support: While the format allows a single test object to contain multiple possible outcomes (via a list in the post field), tests generated by the execution-spec-tests repository currently use only one test vector per object.

  11. Configure documentation navigation

    main

    Navigation for the documentation is controlled via navigation.md using the mkdocs-literate-nav plugin.

    Manual Navigation

    All pages intended for the documentation and the navigation bar must be explicitly included in navigation.md.

    Automatic Navigation

    The "Test Case Reference" entries are generated automatically and appended to navigation.md during the build process. Because of this, you should place your manual navigation entries before the Test Case Reference section to ensure correct ordering.

  12. Understand Transaction Test fixtures

    main

    Transaction Test fixtures are used to verify a client's ability to parse transaction RLP (Recursive Length Prefix) bytes without executing them on the EVM.

    These tests define a transaction's binary representation and specify whether it should be accepted or rejected based on the current fork schedule. A single JSON fixture file contains multiple test objects, where each key is a unique test name. The unique identifier for a test is the combination of the JSON file path and the test name.