Stripe AI

This evaluation tests the ability to upgrade a Ruby application from Stripe SDK v7.x to v15.x, moving from API version 2022-08-01 to 2025-03-31.basil.

Available Problems

1. charges-on-payment-intent: Update code that uses the deprecated payment_intent.charges attribute (removed in newer API versions). The solution must use Charge.list() or latest_charge instead.
2. invoice-partial-payments: Update invoice-related code to work with SDK v15.x.
3. subscription-billing-migration: Update subscription-related code to work with SDK v15.x.

Stripe AI provides a collection of SDKs designed to help you integrate Stripe's billing infrastructure with Large Language Models (LLMs) and agent frameworks.

Key packages include:

• @stripe/ai-sdk: Used for integrating Stripe's billing infrastructure with Vercel's ai and @ai-sdk libraries.
• @stripe/token-meter: Used for integrating Stripe's billing infrastructure with native SDKs from OpenAI, Anthropic, and Google Gemini without framework dependencies.

Furever is a computer-use evaluation designed to test an AI's ability to integrate Stripe Connect Embedded Components into an existing Next.js application.

The Task: The AI must add a Payments page to the provided Next.js application using the ConnectPayments component, configured with the following specific settings:

• Disputes: Enabled
• Refunds: Disabled

FurEver is a full-stack demo application for a vertical SaaS platform (pet salon management) that integrates Stripe Connect and Stripe Connect embedded components. It demonstrates how a platform can provide Stripe products and UIs directly within its own website to reduce engineering costs.

The demo showcases a unified account configuration where:

• Stripe owns loss liability
• The platform owns pricing
• Stripe is the onboarding owner
• Connected accounts have no access to the Stripe dashboard

Technical stack: Next.js, Express, React, TypeScript, Tailwind CSS, shadcn/ui, and Material UI.

The Stripe Agent Toolkit enables integration between popular agent frameworks and Stripe APIs through function calling.

Supported Frameworks:

• Model Context Protocol (MCP)
• OpenAI's Agent SDK
• LangChain
• CrewAI
• Vercel's AI SDK

Supported Languages:

• Python
• TypeScript

The toolkit is built directly on top of the official Stripe Python and Node.js SDKs. Note that the toolkit does not support the entire Stripe API, but a subset of methods.

This benchmark tests the ability of an AI agent to upgrade a Next.js SaaS application from hosted Stripe Checkout to embedded checkout. The evaluation involves three primary subtasks:

1. Converting hosted checkout to embedded checkout: Migrating the implementation to use Stripe's embedded checkout.
2. Promotion Code Management: Creating and applying a promotion code.
3. Transaction Completion: Completing a test purchase and capturing the resulting checkout session ID.

To pass, the solution must satisfy specific grading criteria: the UI mode must be embedded, the mode must be subscription, the payment status must be paid, and a promotion code TAKE20 (providing 20% off) must be applied.

Use Stripe Projects to provision third-party infrastructure such as databases, auth, hosting, analytics, caching, AI, and observability. You can also retrieve API keys and tokens for these services using the Stripe Projects CLI plugin.

Triggers for this skill:

• "I need a database"
• "set up auth"
• "provision Redis"
• "add a vector DB"
• "get credentials for X"
• "show me the catalog"
• Any mention of projects.dev or adding/provisioning/connecting cloud services.

The Galtee Invoicing evaluation is designed to test an AI's ability to handle specific Stripe and data integration tasks. The core components provided are:

• environment/PROBLEM.md: The problem prompt.
• environment/server: A Node server skeleton.
• environment/client: A static client.
• environment/db: Database files.
• grader: An RSpec grader containing hidden tests.

Key evaluation areas:

• Integrating with Stripe's Invoice API.
• Migrating CSV data into SQLite and Stripe.
• Creating Stripe Products and Invoices.
• Implementing basic booking management APIs.

The Stripe integration benchmarks are agentic evaluations designed to test how AI agents handle Stripe integration tasks. The tasks are categorized into three types:

1. Backend-only tasks: Focus on server-side integrations, including data migrations, database management, scripting, and updating backend APIs.
2. Full stack tasks: Require both server-side and client-side integrations, where the agent is expected to use a browser to complete the task.
3. Gym problem sets: Simulated exercises targeting specific Stripe features (like Checkout or Subscriptions) to test depth of understanding in niche scenarios.

The Partial Payments evaluation tests the ability to implement a split-payment flow for a single subscription invoice. A successful implementation involves:

1. Subscription Configuration: Modify subscription creation to remove trial periods.
2. Payment Behavior: Create the subscription using payment_behavior: 'default_incomplete'.
3. Partial Payment Flow:
   • Create two separate checkout sessions, each for half of the total amount.
   • Use stripe.invoices.attach_payment to attach multiple payment intents to the single invoice.
4. Completion: Complete both checkout flows and capture the resulting invoice ID.