RD-Agent: Research & Development Agent

BioProBench is a multi-task benchmark for biological protocol understanding and reasoning. It includes four specific fine-tuning tasks:

1. PQA (Protocol Question Answering): Multiple-choice QA regarding protocol content (parameters, reagents, operations).
2. ORD (Step Ordering): Ordering protocol steps (top-level or sub-steps) into the correct chronological sequence.
3. ERR (Error Correction): Detecting and correcting errors in protocol text using local context.
4. GEN (Protocol Generation): Generating concise, numbered, single-level protocol steps from a prompt.

All tasks support <think>...</think> tags for Chain-of-Thought (CoT) reasoning, which the evaluator will strip before comparison.

The PANORAMA dataset is a patent examination benchmark that captures decision trails and rationales. It is used for legal reasoning tasks involving patent claims and prior art. It is available on Hugging Face via the LG-AI-Research/PANORAMA repository.

Key Tasks Supported:

• PAR4PC (Prior-Art Retrieval for Patent Claims): Multi-label classification to select relevant prior-art documents from 8 candidates. Metrics: Exact Match Accuracy and Custom Score.
• PI4PC (Paragraph Identification for Patent Claims): Single-choice task to identify the most relevant paragraph in a prior-art document. Metric: Exact Match Accuracy.
• NOC4PC (Novelty and Non-Obviousness Classification): Ternary classification to determine if a claim should be ALLOW, 102 rejection (Novelty), or 103 rejection (Non-Obviousness). Metrics: Macro F1-score and Per-class Accuracy.

The DeepScaleR dataset is a curated collection of approximately 40,000 unique mathematics problem-answer pairs designed for training mathematical reasoning models via reinforcement learning (RL). It is specifically optimized for scaling RL algorithms (like GRPO) on competition-level mathematics ranging from high school to olympiad levels.

Key characteristics:

• Target Task: Mathematical reasoning.
• Scale: ~40,000 examples.
• Use Case: Training models to achieve high Pass@1 accuracy on benchmarks like AIME 2024 through distributed RL and context length scaling (8K→16K→24K).

R&D-Agent is designed for automated Data-Driven R&D, focusing on three core capabilities:

1. Extraction: Reading real-world materials (reports, papers) to extract formulas, features, and models.
2. Implementation: Converting extracted formulas into runnable code, using an evolving process to improve performance via feedback.
3. Ideation: Proposing new ideas based on existing knowledge and observations.

The system operates in two modes:

• Copilot (🦾): Automates repetitive tasks following human instructions.
• Agent (🤖): Acts autonomously to propose ideas for future improvements.

Supported domains include Finance (Model Implementation and Data Building), Medical (Model Implementation), and General (Paper reading, Kaggle Model Tuning, and Kaggle Feature Engineering).

The FinanceIQ dataset is a collection of approximately 6,179 multiple-choice questions derived from Chinese financial professional certification exams. It is designed to benchmark financial reasoning capabilities in Chinese language models across 10 distinct financial domains.

Key Domains Covered:

• CPA (注册会计师)
• Banking Practitioner (银行从业资格)
• Securities Practitioner (证券从业资格)
• Fund Practitioner (基金从业资格)
• Insurance (CICE)
• Economist (经济师)
• Tax Advisor (税务师)
• Futures Practitioner (期货从业资格)
• Financial Planner (理财规划师)
• Actuary (精算师-金融数学)

Repository Source: LlamaFactory/FinanceIQ License: CC-BY-NC-SA-4.0

The TableBench ecosystem consists of two primary components for Table Question Answering (TableQA) evaluation and training:

1. TableBench (Test): A high-quality evaluation benchmark containing 886 curated test cases across 4 major reasoning categories (Fact Checking, Numerical Reasoning, Data Analysis, and Visualization).
2. TableInstruct (Train): A large-scale instruction tuning dataset designed for training models on diverse table QA examples and reasoning patterns.

Key Task Categories:

• Fact Checking: Verifying statements (e.g., simple verification, cross-table validation).
• Numerical Reasoning: Mathematical operations (e.g., arithmetic, aggregations, comparisons).
• Data Analysis: Complex reasoning (e.g., impact analysis, correlation, trend forecasting).
• Visualization: Chart generation and interpretation (e.g., bar, line, pie, scatter plots).

FT-Agent is an autonomous LLM fine-tuning scenario that automates a benchmark-driven loop. It is designed to:

1. Inspect target benchmarks and raw datasets.
2. Generate data processing code and LLaMA-Factory training configurations.
3. Perform fail-fast validation before full training.
4. Execute the fine-tuning process.
5. Evaluate results using OpenCompass and use the feedback to iterate.

Note: This is a research-oriented implementation. Running it involves downloading large datasets, building environments, calling LLM APIs, and consuming GPU resources.

R&D-Agent is a multi-agent framework designed for autonomous machine learning engineering, quantitative finance research, and data science. It is a top-performing agent on the MLE-bench benchmark and includes specialized implementations such as:

• FT-Agent: For autonomous LLM fine-tuning.
• RD-Agent(Q): A data-centric multi-agent framework for quantitative finance strategy development.
• Data Science Agent: For general data science tasks (including Kaggle scenarios).
• Agent² RL-Bench: A benchmark for evaluating LLM agents on end-to-end post-training engineering pipelines.

The project supports LiteLLM as a default backend for integrating various LLM providers.

R&D-Agent is designed to automate industrial R&D processes, specifically focusing on data-driven scenarios. The framework is built on two core components: 'R' (proposing new ideas) and 'D' (implementing those ideas).

Key application scenarios include:

• Automatic Quant Factory: Automating quantitative trading research.
• Data Mining Agent: Iteratively proposing and implementing data and models by extracting knowledge from data.
• Research Copilot: Automatically reading research papers or financial reports to implement model structures or build datasets.
• Kaggle Agent: Automating model tuning and feature engineering for machine learning competitions.
• FT-Agent: Autonomous LLM fine-tuning for benchmark-driven domain adaptation.

RD-Agent's research is focused on enabling continuous evolution of R&D capabilities and acquiring specialized domain knowledge through deep exploration. The research framework is divided into three primary areas:

1. Benchmark: Evaluating and benchmarking R&D abilities.
2. Research: Idea proposal, which involves exploring new ideas or refining existing ones.
3. Development: The ability to realize ideas through implementation and execution.