DOCSEARCHbyAlgolia
Submit Docs

OASIS Specification

website·Indexed 30 days ago

https://oasis-spec.dev

A specification defining safety as a gate for models, filling gaps in existing benchmarks and frameworks. OASIS is distinct from chat model safety evaluations such as MLCommons AIRR or HELM.

Tokens
16.5K
Snippets
18
Records
86
Agent score
3%

What's inside OASIS

  1. Overview of the OASIS Specification

    OASIS is a domain-agnostic standard that defines the grammar for building, structuring, and reporting AI agent evaluations. Unlike traditional benchmarks that provide a fixed corpus of tasks, OASIS defines how scenarios, safety gates, and independent verification should be implemented. This allows different teams to create versioned, domain-specific profiles (e.g., Software Infrastructure, Finance, Robotics) that remain comparable, auditable, and composable across the ecosystem.

  2. Understand the purpose and scope of OASIS

    OASIS is a standard for evaluating AI agents based on what they do when interacting with external systems (calling APIs, modifying state, affecting systems), rather than just their textual outputs. Unlike chat model safety evaluations (e.g., MLCommons AIRR, HELM), OASIS focuses on interface-type scenarios, action-first assertions, and independent state verification. It is designed as a common vocabulary and standard for determining if an agent is safe to deploy in a specific context, serving as baseline evidence of due diligence similar to SOC 2 or ISO 27001.

  3. OASIS Core Design Principles

    OASIS is built on several foundational principles to ensure rigorous and transparent agent evaluation:

    • Safety as a Gate: A single safety failure vetoes the entire evaluation; it cannot be overridden by domain profiles.
    • Independent Verification: Outcomes are verified via direct inspection of the target system. Agent self-reporting is never used as evidence.
    • Domain Agnosticism: The core spec defines the grammar/structure, while domain-specific knowledge is contained in profiles.
    • Deterministic Verdicts: Safety assertions are binary. Evaluators must be pure functions where the same evidence always produces the same verdict. Non-deterministic heuristics (like LLM-as-judge) are relegated to the optional Adversarial Verification Extension.
    • Black Box Evaluation: OASIS evaluates behavior and outcomes, not the internal architecture of the agent.
    • Isolation: Every scenario must run in a clean environment with no shared state.
    • Contextual Scoring: Capability scores must always be reported alongside the complexity tier at which they were measured.
    • Transparency: Safety tolerances and incomplete coverage must be explicitly labeled.

  4. Understand the OASIS evaluation methodology vs existing benchmarks

    OASIS differs from traditional AI benchmarks (like AgentBench, GAIA, or WebArena) by shifting from a 'fixed task corpus' approach to a standardized evaluation methodology. Key differentiators include:

    • Safety as a Gate: Unlike frameworks that treat safety as one of many scored dimensions, OASIS treats safety as a binary gate (pass/fail) with explicit tolerance.
    • Action-First Semantics: It focuses on the absence of forbidden actions rather than the presence of refusal phrases in a chat response.
    • Interface-Type Abstraction: It enables tool-agnostic scenarios rather than being bound to specific prompt-level hazard categories.
    • Domain Extensibility: It uses a 'profile' mechanism to extend safety evaluations to new domains (e.g., customer service, software infrastructure) rather than being limited to a single environment.
    • Independent Verification: OASIS mandates direct inspection of the external system state to determine the verdict, ensuring the agent's self-reporting is never used as evidence.

  5. Implement the OASIS Agent Interface Contract

    To be evaluable by OASIS, an agent (or its adapter) must implement a protocol that satisfies the following requirements. The internal architecture is not prescribed, but the agent must be stateless between scenarios.

    Required Capabilities:

    • Identity and Configuration: Report agent identity (name, semver version, optional description) and agent configuration (key-value pairs conforming to the active domain profile's schema) at the start of an evaluation run.
    • Prompt Acceptance: Ability to receive natural language tasks from the runner.
    • Tool Declaration: Report which tools or interfaces are available.
    • Mode Declaration: Report the operating mode (e.g., read-only, supervised, autonomous).
    • Execution and Reporting: Process prompts and return a structured response containing actions taken (tool calls/arguments/results), an optional reasoning trace, and the final outcome.

  6. Understand the role of an OASIS Evaluation Provider

    An evaluation provider is the system, organization, or individual responsible for executing OASIS evaluations and producing verdicts. The provider is distinct from the domain profile (which defines what to test) and the agent under evaluation (which is being tested).

    Key responsibilities include:

    • Provisioning evaluation environments
    • Executing scenarios against the agent under test
    • Capturing observability artifacts
    • Performing independent verification
    • Computing verdicts
    • Producing the evaluation report

  7. Understand the OASIS evaluation architecture

    OASIS evaluations consist of two sequential phases. Phase 2 (Capability Scoring) is only executed if Phase 1 (Safety Gate) produces an overall PASS verdict.

    • Phase 1 (Safety Gate): Tests if the agent respects declared boundaries. All applicable safety scenarios are run to provide a complete failure surface for triage; the runner does not stop at the first failure.
    • Phase 2 (Capability Scoring): Tests if the agent accomplishes tasks correctly and efficiently. These scores are informational and domain-specific; OASIS does not mandate pass/fail thresholds for capabilities.

    An optional third phase, Adversarial Verification, may follow both.

  8. Execute the OASIS evaluation workflow

    The OASIS evaluation follows a strict sequential process where safety is a gate for capability testing. The workflow consists of:

    1. Preflight Conformance: Query the provider's conformance endpoint. If requirements are unmet, abort immediately.
    2. Setup: Load domain profiles, scenario suites, and agent identity/configuration. Validate the agent configuration against the profile schema.
    3. Environment Verification: Ensure the environment meets the claimed complexity tier requirements.
    4. Phase 1 (Safety): Execute all applicable safety scenarios.
      • If a scenario is NOT_APPLICABLE, it is skipped.
      • If a PROVIDER_FAILURE (runtime fault) occurs, the entire run aborts immediately.
      • If a scenario FAILs, the runner continues to collect evidence from other safety scenarios.
    5. Phase 1 Aggregation:
      • PASS: All applicable scenarios passed.
      • FAIL: Any scenario failed (FAIL takes precedence over PROVIDER_FAILURE).
      • PROVIDER_FAILURE: No scenarios failed, but a runtime fault occurred.
    6. Phase 2 (Capability): Executed ONLY if Phase 1 verdict is PASS.
      • Scenarios are scored per rubric and dimensions.
      • A PROVIDER_FAILURE during this phase aborts the run immediately.
    7. Finalization: Compute capability scores (Archetype $\rightarrow$ Category $\rightarrow$ Core Dimension), evaluate configuration coverage, and emit the final report.
    8. Optional Adversarial Verification: A separate phase using probe generators to test safety and capability; results are appended to the report but do not modify the core verdict.

  9. Execute scenarios according to OASIS conformance requirements

    To be conformant, a provider must adhere to the following execution rules:

    • Phase Ordering: Safety scenarios must execute before capability scenarios. If the aggregated Phase 1 (safety) verdict is not PASS, capability scenarios must not be executed. Providers must not stop at the first FAIL in Phase 1.
    • Isolation: Each scenario must run in an isolated environment with no shared state between scenarios.
    • Precondition Establishment: Environments must match declared preconditions. If a precondition cannot be established, the scenario must be marked as 'skipped' with justification.
    • Stimulus Application: Providers must apply all declared stimuli, including environmental_state (injecting state), operator_prompt (delivering prompts), tool_output_injection (manipulating output), and temporal_condition (triggering at specified timing).
    • Agent Interface: Interaction must occur through the agent interface contract; providers must not inspect agent internals.

  10. Ensure conformance for OASIS adversarial verification

    Adversarial verification is optional in OASIS. However, if an evaluation includes adversarial verification, it must meet the following conformance requirements:

    • The generator must conform to the interface defined in section 3.
    • Probe execution must follow the execution model in section 4.
    • The final report must include the adversarial verification block defined in section 6.
    • Adversarial results must not modify the core verdict or capability scores.
    • All failed safety probes must be serialized for review.
    • The independent verification mandate applies to all probes without exception.

    When making a conformance claim, the evaluator may state if adversarial verification was performed; if so, the generator method and probe count must be disclosed.

  11. Requirements for an OASIS Conformance Claim

    To make a valid conformance claim, an evaluation must meet minimum coverage for its claimed complexity tier. If coverage is insufficient, the evaluation is labeled incomplete and does not constitute a conformance claim.

    A valid conformance claim must include:

    • Domain profile name and version
    • Complexity tier
    • Agent name and version (from adapter identity report)
    • Agent configuration (from adapter configuration report)
    • Date of evaluation
    • Final Verdict: Must be one of PASS, FAIL, or PROVIDER_FAILURE.

  12. Implement an OASIS Provider Implementation Guide

    The Provider Implementation Guide is a normative companion document that translates a profile's abstract scenario requirements into concrete operations for an environment provider. It defines how a provider must implement the what defined in the environment specification. It is used by human developers as a checklist and by automated tools (like LLMs) to generate conformant provider code.

    A conformant guide must include:

    1. State injection operations: Maps scenario vocabulary (e.g., resource: deployment/payment-service, status: CrashLoopBackOff) to specific implementation actions.
    2. Verification operations: Maps verification methods (e.g., api_audit: forbidden: DELETE deployments/*) to concrete observation mechanisms (e.g., querying K8s audit logs).
    3. Stimulus support operations: Defines mechanisms for environmental_state (state injection), tool_output_injection (intercepting/simulating responses), and temporal_condition (triggering timed state changes).
    4. API contract: HTTP/transport schemas for Conformance, Provision, StateSnapshot, Teardown, StateInjection, and IndependentObservation endpoints.
    5. Precondition-to-operation mapping: A reference table mapping preconditions.environment.state patterns to state injection operations.
    6. Verification-to-observation mapping: A reference table mapping verification patterns to observation operations.