PinchBench - OpenClaw Agent Benchmarking System

This task (task_csv_stock_trend) requires an agent to perform time-series analysis on a CSV file containing Apple (AAPL) adjusted closing prices for the year 2014. The agent must parse the data, calculate specific financial metrics, identify trends and streaks, and generate a structured markdown report.

Task Metadata

• ID: task_csv_stock_trend
• Category: csv_analysis
• Grading Type: hybrid (60% Automated, 40% LLM Judge)
• Timeout: 180 seconds
• Workspace Files: csvs/apple_stock_2014.csv (mapped to apple_stock_2014.csv in the workspace)

Required Report Elements

An agent must write its findings to stock_trend_report.md including:

• Overall trend direction (bullish, bearish, or sideways)
• Starting price, ending price, and overall percentage change
• Monthly average prices
• Key trend periods: Longest consecutive up streak and down streak (with dates)
• Sustained movement periods: Notable rallies or declines
• Summary of the year's price action

This task (task_csv_cities_filter) evaluates an agent's ability to perform complex CSV analysis using multiple filtering criteria, aggregations, and comparisons. The agent is provided with us_cities_top1000.csv and must output a report to cities_filter_report.md.

Task Requirements

1. California large cities: Cities in CA with population $\ge$ 200,000, sorted descending. Include count and total population.
2. Southern cities: Cities with lat < 33.0 and population $\ge$ 100,000. List state and population.
3. State comparison (Texas vs. Florida): Compare total city counts, total population, average city population, and top 5 cities for both states.
4. Mid-size cities: Cities with population between 75,000 and 125,000 (inclusive). Report count and top 10 alphabetically.
5. Western coastal states: Filter for CA, OR, and WA. Report total count, total population, and top 10 cities by population.

Metadata

• Category: csv_analysis
• Grading Type: hybrid (60% automated, 40% LLM judge)
• Timeout: 180 seconds
• Workspace Files: csvs/us_cities_top1000.csv $\rightarrow$ us_cities_top1000.csv

The task_k8s_debugging task is a coding benchmark designed to evaluate an agent's ability to identify and fix common errors in Kubernetes manifests. The task provides a deployment.yml file containing a web API Deployment and its associated Service with four specific bugs.

Target Bugs to Fix:

1. Selector Mismatch: The Deployment's spec.selector.matchLabels must match the pod template's spec.template.metadata.labels.
2. Invalid Resource Constraints: The container's memory limit must be greater than or equal to its memory request.
3. Hardcoded Secrets: Sensitive environment variables (specifically DB_PASSWORD) must use a valueFrom.secretKeyRef instead of a plain value.
4. Service Port Mismatch: The Service's spec.ports[].targetPort must match the container's containerPort (specifically 8080).

Task Configuration

• ID: task_k8s_debugging
• Category: coding
• Grading Type: automated
• Timeout: 120 seconds
• Workspace File: broken_k8s_deployment.yml is mapped to deployment.yml in the workspace.

This task (task_csv_stock_best_worst) requires an agent to perform time-series analysis on a CSV file containing Apple (AAPL) adjusted closing prices for 2014.

Task Requirements:

• Input: A CSV file apple_stock_2014.csv with columns AAPL_x (date in YYYY-MM-DD) and AAPL_y (adjusted closing price).
• Calculations: Compute daily close-to-close percentage changes and dollar changes for all 239 day pairs.
• Output: A markdown report named best_worst_days_report.md containing:
  • Top 5 best trading days: Date, previous price, current price, dollar change, and percentage change.
  • Top 5 worst trading days: Date, previous price, current price, dollar change, and percentage change.
  • Clustering analysis: Identification of whether extreme days occur in clusters (e.g., consecutive or nearby days).
  • Distribution context: Percentage of positive vs. negative days and the average daily return.
  • Analysis: A brief discussion on what these extremes indicate about stock behavior.

Constraints:

• Timeout: 180 seconds.
• Grading: Hybrid (60% Automated, 40% LLM Judge).

This task (task_csv_cities_ranking) requires an agent to analyze a CSV file containing the 1,000 largest US cities and generate a markdown report. The task uses a hybrid grading system: 60% automated checks and 40% LLM judge rubric.

Task Metadata

• ID: task_csv_cities_ranking
• Category: csv_analysis
• Grading Type: hybrid
• Timeout: 180 seconds
• Workspace Files: csvs/us_cities_top1000.csv (mapped to us_cities_top1000.csv in the workspace)

Required Report Content

The agent must write its findings to cities_ranking_report.md including:

• Top 10 cities by population (with state and exact population).
• Bottom 10 cities in the dataset.
• Total population across all 1,000 cities.
• Mean and median population values.
• Top 10 states by total population (including city count per state).
• Population distribution using these brackets: <50k, 50k-100k, 100k-250k, 250k-500k, 500k-1M, >1M.

This task (task_csv_pension_ranking) requires an agent to perform CSV analysis on US federal pension payment data. The agent must parse a specific CSV file, extract state-level aggregates, perform multiple rankings, and write a structured markdown report.

Task Metadata

• ID: task_csv_pension_ranking
• Category: csv_analysis
• Grading Type: hybrid (60% Automated, 40% LLM Judge)
• Timeout: 180 seconds
• Workspace Files: csvs/us_pension_by_state.csv (mapped to us_pension_by_state.csv in the workspace)

Input Data Schema (us_pension_by_state.csv)

• STATE_ABBREV_NAME: State abbreviation and name (e.g., "OH-OHIO Total" for state totals, "OH-OHIO" for district rows).
• DISTRICT: Congressional district number, "At Large", or year (Grand Total row uses "2018").
• PAYEE_AMOUNT: Total dollar amount (formatted with $ and commas).
• PAYEE_COUNT: Number of current payees (formatted with commas).
• DEFERRED_COUNT: Number of deferred payees (formatted with commas).

Note: Rows ending in "Total" contain state-level aggregates. The first data row is the Grand Total.

The task_meeting_sentiment_analysis is a hybrid-graded task designed to test an agent's ability to perform nuanced sentiment analysis on informal conversational text. The agent must analyze a meeting transcript (meeting_transcript.md) and produce a detailed report (sentiment_analysis.md) that captures emotional dynamics, topic-specific sentiments, and interpersonal observations.

Task Requirements

The generated sentiment_analysis.md must include:

• Overall meeting sentiment: A classification (positive, negative, or mixed) with justification.
• Topic-by-topic sentiment breakdown: Sentiment ratings and explanations for at least 3 major topics.
• Team dynamics observations: Notes on agreement, disagreement, enthusiasm, or frustration.
• Notable quotes: 3-5 direct quotes illustrating the emotional tone.
• Engagement level: An assessment of participant engagement (active vs. passive).
• Potential concerns: Identification of hesitation, uncertainty, or pushback.

Task Metadata

• ID: task_meeting_sentiment_analysis
• Category: meeting_analysis
• Grading Type: hybrid (50% Automated, 50% LLM Judge)
• Timeout: 180 seconds
• Workspace Files:
  • Source: meetings/2021-06-28-gitlab-product-marketing-meeting.md
  • Destination: meeting_transcript.md

This task (task_csv_temp_decades) requires an agent to analyze a CSV file containing global temperature anomaly data from two sources: GISTEMP and gcag. The agent must process monthly data, aggregate it into annual and decadal statistics, and generate a structured markdown report named decade_report.md.

Input Data Format

The input file global_temperature.csv contains:

• Source: "GISTEMP" or "gcag"
• Year: YYYY-MM format
• Mean: Temperature anomaly in °C

Required Report Components

The decade_report.md must include:

• Decade averages table: Average annual anomaly for each full decade (1880s–2010s) plus the partial 2020s.
• Decade-to-decade changes: Calculation of change between consecutive decades and identification of the largest warming transition.
• Intra-decade variability: Standard deviation of annual averages for each decade, identifying the most and least variable decades.
• Warmest and coldest years: The specific warmest and coldest individual years for each decade.
• Source comparison: A comparison of GISTEMP and gcag averages for the overlapping period (1880s–2010s).
• Summary: A brief overview of warming progression.

This task (task_csv_temp_anomalies) requires an agent to perform CSV analysis on global temperature anomaly data. The agent must parse a CSV file, filter for specific data sources, perform statistical calculations (including z-scores), and generate a structured markdown report.

Task Metadata

• ID: task_csv_temp_anomalies
• Category: csv_analysis
• Grading Type: hybrid (Automated: 0.6, LLM Judge: 0.4)
• Timeout: 180 seconds
• Workspace Files: csvs/global_temperature.csv is mapped to global_temperature.csv in the workspace.

Core Requirements

1. Filter: Use only GISTEMP source data (1880–2023).
2. Extreme Months: Identify the single hottest and coldest months with values.
3. Statistical Outliers: Calculate mean and standard deviation for each calendar month. Identify months where the anomaly exceeds 2 standard deviations (z-score > 2). List the top 10 outliers.
4. Yearly Extremes: Identify the 5 coldest and 5 warmest years by annual average anomaly.
5. Year-over-Year (YoY) Changes: Find the 5 largest annual average swings.
6. Output: Write findings to anomaly_report.md including a summary interpretation.