Frontier Evals

repository·main·Indexed 23 days ago

https://github.com/openai/frontier-evals

A collection of evaluation suites for measuring frontier AI model capabilities, featuring benchmarks such as PaperBench, SWE-Lancer, and EVMBench. It includes the nanoeval framework for defining and running evaluations via Eval, EvalSpec, Task, and Solver abstractions, as well as the ploit toolset for smart contract security evaluation, including environment setup, transaction re-execution, and grading.

Tokens
462.1K
Snippets
709
Records
1.1K
Agent score
79%

What's inside frontier-evals

  1. Introduction to PaperBench

    main

    PaperBench is an evaluation framework designed to test an AI agent's ability to replicate research from 20 Spotlight and Oral papers from ICML 2024 from scratch. Each evaluation sample consists of a research paper and a rubric defining the requirements for successful replication.

    The evaluation process follows a three-stage lifecycle:

    1. Agent Rollout: The agent runs in an Ubuntu container to create a codebase that replicates the paper.
    2. Reproduction: The submitted codebase is executed in a fresh, second container (with GPU access) to generate results (the executed submission).
    3. Grading: A third container runs a judge to grade the executed submission against the paper's rubric.
  2. Vulnerability Analysis: NFT Theft and Payment Freezing in re-nft

    main

    This document details a critical vulnerability (H-05) in the re-nft smart contracts where a malicious actor can steal actively rented NFTs and freeze rental payments in the escrow contract. The exploit leverages three primary bugs:

    1. Delayed Validation: The stopRent and stopRentBatch functions in Stop.sol do not validate the existence of a RentalOrder until after the lender has been sent the specified NFT. This allows an attacker to supply a non-existent order that is created during the onERC721Received/onERC1155Received callback.
    2. Predictable Signatures: The signature digest signed by the protocol signer is predictable and not unique to a specific order, allowing a generic signature to be reused for multiple fulfillments until metadata.expiration >= block.timestamp.
    3. EIP-712 Non-compliance: Core functions in Signer.sol are not EIP-712 compliant, allowing valid signatures to be used across different metadata.orderType values, rentDuration values, and arbitrary safe wallet addresses.

    Impact: An attacker can steal and freeze any/all NFTs currently being rented (active or expired) as long as the rental has not been stopped.

  3. Manage ploit evaluation configurations with `ploit-config`

    main

    ploit-config provides configuration management for ploit evaluation environments. It allows you to store setup state and deployed contracts using specific configuration types, and supports loading/saving these configurations via TOML files.

    Core Components

    • Configuration Types:
      • PloitConfig: Stores the overall setup state.
      • DeployedContract: Stores information about deployed contracts.
    • File I/O: Supports loading and saving configurations using TOML format.
    • Config Manager: Provides a CLI-driven way to manage configuration values using ConfigManager, ConfigKey, ConfigAction, and ActionType.
    • Constants: Includes default values for RPC ports, wallet addresses, and file paths.
  4. Core features of `veto-core`

    main

    The veto-core runtime includes the following capabilities:

    • Runtime: The run function bootstraps the Axum server, binds to the requested socket, and awaits Ctrl+C for shutdown.
    • Proxy Engine: The router wires handlers around AppState to allow the proxy to either forward or block JSON-RPC calls.
    • JSON-RPC Validation: Implements strict parsing to guard against malformed payloads and rejects batch requests upfront.
    • Error Reporting: Provides deterministic error payloads and rich ProxyError diagnostics for callers.
  5. Use `ploit-utils` for common workspace helpers

    main

    The ploit-utils crate provides shared utilities for the ploit workspace, including debugging tools, logging, signal handling, CLI styling, and process management.

    Key features include:

    • Backtrace: Helpers to enable Rust backtraces for easier debugging.
    • Logging: Structured logging utilities to ensure consistent output across the workspace.
    • Signal Handlers: A SIGSEGV handler specifically for crash diagnostics.
    • CLI Styles: Pre-defined styles for consistent command-line interface output.
    • Process Management: Utilities like KillService, KillConfig, and kill_port for managing processes and freeing up ports.
    use ploit_utils::{backtrace, logging, styles, kill_port};
    
    // Enable backtrace for debugging
    backtrace::install();
    
    // Initialize logging
    logging::init(true); // verbose mode
    
    // Kill a process on a specific port
    kill_port(8545)?;
    
    // Use CLI styles for consistent output
    use styles::*;
    println!("{}", SUCCESS.paint("Operation completed"));
  6. Avoid using tx.origin for authorization to prevent phishing attacks

    main

    In smart contract development, using tx.origin for authorization checks instead of msg.sender creates a vulnerability where a malicious contract can trick a user into authorizing a transaction that redirects assets to the attacker.

    The Vulnerability Pattern

    A phishing attack succeeds when a contract follows this pattern:

    1. Authorization via tx.origin: The contract checks if tx.origin (the original EOA/user) is the owner.
    2. Asset transfer to msg.sender: The contract transfers assets to msg.sender (the caller of the function).

    In this scenario, if a user calls a malicious PhishingProxy contract, the tx.origin remains the user (passing the check), but msg.sender becomes the PhishingProxy contract, which receives the assets.

    • Use msg.sender exclusively: Always use msg.sender for authorization checks to ensure the immediate caller is the intended actor.
    • Explicit Recipient Parameters: Instead of defaulting to msg.sender, require a recipient address as a parameter and verify that the recipient is the authorized owner.
    • Trusted Forwarders: If implementing meta-transactions or relayers, use the EIP-2771 trusted forwarder pattern rather than relying on tx.origin.
  7. Vulnerability: Non-EIP-712 compliant hash derivation in reNFT

    main

    The _deriveOrderMetadataHash and _deriveRentPayloadHash functions in the Signer contract do not fully comply with the EIP-712 standard. Specifically, the OrderMetadata struct contains fields that are omitted from its derived hash, allowing for potential manipulation of order parameters without invalidating the signature.

    Affected Structs and Fields

    OrderMetadata struct:

    • orderType: Omitted from the hash.
    • rentDuration: Included.
    • hooks: Included.
    • emittedExtraData: Omitted from the hash.

    Because orderType and emittedExtraData are not part of the hash, an attacker can change these values in an order while using a signature originally generated for a different order type, provided the rentDuration and hooks remain the same.

    Impact on Integrators

    Third-party integrators attempting to follow the EIP-712 standard strictly (e.g., via an SDK) will find that signatures generated using 'correct' EIP-712 digests are rejected by the reNFT protocol. This is because the protocol's internal validation relies on these non-compliant derivation functions.

    function _deriveOrderMetadataHash(
        OrderMetadata memory metadata
    ) internal view returns (bytes32) {
        // ...
        return
            keccak256(
                abi.encode(
                    _ORDER_METADATA_TYPEHASH,
                    metadata.rentDuration,
                    keccak256(abi.encodePacked(hookHashes))
                )
            );
    }
  8. Impact: Reward accounting inconsistencies in AgentRewardV2

    main

    Duplicate validator entries caused by the addValidator() vulnerability impact the AgentRewardV2::_distributeValidatorRewards() logic.

    While individual validator rewards are simply overwritten, the shared variable validatorPoolRewards accumulates residuals multiple times for the same validator. This leads to validatorPoolRewards becoming overstated relative to the actual token amount deposited. Consequently, when withdrawValidatorPoolRewards() is called, the contract may transfer an excess amount, leaving the balance insufficient to cover valid validator rewards and causing claim failures.

    // AgentRewardV2::_distributeValidatorRewards() logic affected by duplicates
    for (uint256 i = 0; i < validatorCount; i++) {
        address validator = nft.validatorAt(virtualId, i);
        // ... calculation ...
        _validatorRewards[validator][rewardId] = participationReward;
        validatorPoolRewards += validatorRewards - participationReward;
    }
  9. Vulnerability in ValidatorRegistry score initialization

    main

    In the ValidatorRegistry contract, the _initValidatorScore function initializes new validators with a base score derived from the total number of existing proposals for a virtual (_getMaxScore(virtualId)). This base score is added to the validator's actual participation score in both validatorScore and getPastValidatorScore functions.

    This creates a vulnerability where new validators can earn full rewards without participating in the protocol. Because the score is pre-filled from historical proposals, a new validator's score can actually exceed that of an older validator who has participated in some, but not all, proposals.

    Impact:

    • New validators can earn full or near-full rewards without voting.
    • Stakers can game the system by repeatedly delegating to newly-registered validators to maintain a near-perfect participation multiplier.
    • Active, long-term validators are unfairly penalized by the dilution of rewards.
  10. Vulnerability: Arbitrary minting of claimable TITN via `onTokenTransfer`

    main

    In the MergeTgt contract, the onTokenTransfer function is used as an ERC677-like callback to account for incoming TGT tokens. However, because the function is external and lacks a check to ensure msg.sender is the authorized TGT token contract, any address can call it directly.

    An attacker can call onTokenTransfer with an arbitrary amount to inflate their internal entitlement to TITN without actually transferring any TGT tokens. This allows them to subsequently call claimTitn to drain the entire TITN reserve held by the contract.

  11. Vulnerability: Reentrancy in Size protocol liquidation

    main

    The Size protocol's executeLiquidateWithReplacement and executeRepay functions are vulnerable to reentrancy attacks. This occurs because the protocol performs external calls to untrusted contracts (like borrowOffer.getRatePerTenor or borrowAToken.transferFrom) before finalizing critical state updates (such as updating the borrower or futureValue).

    An attacker can deploy a malicious borrowOffer that uses the getRatePerTenor callback to re-enter the Size contract and trigger nested liquidations on the same debtPositionId. Because the state is not yet updated, these nested calls execute against stale data, allowing the attacker to siphon borrowAToken and collateral rewards multiple times until the pool is drained.