DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report

If you like our project, please give us a star ⭐ on GitHub for the latest update.

---

✨ News

• [Aug 2026] 🔄 Official Evaluator Migrated to GPT-5.5

  • Google has scheduled gemini-2.5-pro for retirement on Vertex AI on October 20, 2026.
  • To keep the official evaluation pipeline reproducible and available, we have migrated the default evaluator from Gemini 2.5 Pro to GPT-5.5.
  • The previous Gemini 2.5 Pro evaluator remains available on the gemini-25 branch.

• [May 2026] 📜 Per-task License Metadata Added

  • Each entry in tasks_and_rubrics.jsonl now includes a license field indicating the license of its source article.
  • Except for idx=26 and idx=110 (CC BY-NC 4.0) and idx=119 (CC0), all other tasks are derived from CC BY 4.0 sources.
  • Updated DATA_LICENSE: each task's license now follows its source article individually, rather than applying a blanket CC-BY-NC-4.0 to the entire dataset.

• [Feb 24, 2026] 📦 DeepResearch Bench II Dataset Released

  • We have released the DeepResearch Bench II Dataset on Hugging Face, containing articles generated by a subset of the models we evaluated.

• [Feb 2026] 🌐 Official Website & Leaderboard Released

  • The official DeepResearch Bench II Website is now live!
  • Check out the Leaderboard to see how SOTA deep research agents compare across 9,430 expert-written rubrics.
  • 🎯 Welcome to submit your model! See the submission requirements and contact us at imlrz@mail.ustc.edu.cn or dumingxuan@mail.ustc.edu.cn.

• [Jan 2026] 📄 Paper Released on arXiv

  • Our paper is now available on arXiv (2601.08536).

• [Nov 2025] 🎉 DeepResearch Bench II Evaluation Pipeline Released

  • The initial official evaluation pipeline for DeepResearch Bench II used Gemini 2.5 Pro with fine-grained, verifiable rubrics derived from expert-written research reports.
  • It supported multimodal inputs (PDF/DOCX/images/text) and batched rubric-based evaluation for information recall, analysis, and presentation.

For complete experimental results, model comparisons, and ablation studies, please refer to the main paper (paper/main.pdf).

---

📖 Overview

DeepResearch Bench II addresses key limitations of existing deep research benchmarks by combining:

• Real-world, expert-authored research reports as the grounding signal.
• Fine-grained, fully verifiable rubrics that do not rely on the judge model’s internal domain knowledge.
• Three core dimensions of deep research quality:
  • 🔍 Information Recall – Can the agent identify, retrieve, and cross-check all key information needed to answer the task?
  • 🧠 Analysis – Can the agent synthesize retrieved information into higher-level conclusions and insights?
  • 📝 Presentation – Can the agent present the information in a structured, readable, and easily verifiable way?

This repository (DeepResearch-Bench-II) contains a lightweight evaluation pipeline that:

• Takes model-generated research reports (DOCX/Markdown),
• Uses tasks_and_rubrics.jsonl to load task descriptions and rubrics, and
• Invokes GPT-5.5 to score each rubric item in batches, producing:
  • Per-task, per-dimension rubric scores, and
  • Aggregated CSVs summarizing model performance.

---

Benchmark Construction

Topic and Task Design

DeepResearch Bench II is built on top of the original DeepResearch Bench topic distribution and task design:

• We start from real-world user queries and task themes collected in the original benchmark.
• For each seed task, we search for expert-written review reports addressing similar research questions in:
  • Reputable journals and top conferences,
  • High-quality institutional or governmental reports.

These source reports are:

• Written by domain experts over weeks or months,
• Validated by reviewers, editors, and the broader community,
• Released under CC-BY-4.0 / CC-BY-4.0-NC licenses.

After license filtering and quality screening, we retain 132 expert-authored reports, which become the basis for:

• Task formulations, and
• Ground-truth, expert-aligned rubrics.

Rubric Design from Expert Articles

From each expert article, we construct:

• One or more deep research tasks that require both information collection and analysis.
• A set of binary rubrics decomposed across the three dimensions:
  • Information Recall,
  • Analysis,
  • Presentation.

Each rubric is:

1. Essential – captures information necessary to correctly answer the task.
2. Atomic – checks a single fact or inference; complex points are split into smaller rubrics.
3. Content-bearing – encodes the actual answer, not just a vague topic (e.g., “states that X increased from A to B between years Y and Z”).
4. Numerically precise – numerical rubrics explicitly specify values and tolerated error ranges.

Rubrics are built through a four-stage pipeline:

1. LLM extraction from expert articles, guided by carefully designed prompts.
2. Self-evaluation iteration – rejecting hallucinated or inconsistent rubrics using the source article as reference.
3. Manual revision – human annotators refine wording, remove redundancy, and enforce atomicity.
4. Expert review & refinement – domain experts ensure that rubrics faithfully represent the article’s core content.

---

Evaluation Framework

DeepResearch Bench II uses LLM-as-judge with verifiable rubrics:

1. The task + rubric are serialized into a structured JSON prompt.
2. The model report (DOCX/Markdown) is extracted as text and provided as the passage.
3. GPT-5.5 is prompted to output, for each rubric item:
   • score ∈ {1, 0, -1},
   • reason, and
   • evidence (supporting sentences from the report).

Scoring semantics:

• 1 – rubric satisfied with valid evidence and no use of blocked references,
• 0 – rubric not mentioned at all,
• -1 – rubric mentioned but evidence relies on explicitly blocked references.

The evaluation pipeline in this repo:

• Handles text-only inputs:
  • DOCX files are parsed into text and tables (Markdown); embedded images are ignored.
  • Markdown files are loaded as plain text.
• Supports batched evaluation:
  • Rubric items are split into batches of size CHUNK_SIZE (default 50).
  • Each batch is evaluated independently; results are merged and re-grouped by dimension.
• Aggregates token usage statistics:
  • Per batch (usageMetadata),
  • Per file, and
  • Per model across the whole run.

Evaluator consistency study

We compared three candidate evaluators against human annotations on the same 10 reports, covering 738 matched rubric-level judgments. All evaluators used the same task definitions, rubric items, three-way labels, and evaluation prompt.

Evaluator	3-way accuracy	Cohen's κ	Info recall	Analysis	Presentation	Task-level Pearson r	Reported tokens
GPT-5.5	91.19%	0.7993	92.10%	87.69%	90.20%	0.9587	328,333
Gemini 3.1 Pro Preview	91.06%	0.7896	92.46%	85.38%	90.20%	0.8689	440,459
Claude Opus 4.8	83.33%	0.6479	84.56%	77.69%	84.31%	0.8376	428,222

GPT-5.5 achieved the highest overall exact-match accuracy, Cohen's κ, and task-level correlation while using 25.5% fewer reported tokens than Gemini 3.1 Pro Preview. This is a small-scale evaluator agreement study rather than a leaderboard comparison; token counts cover accepted evaluation outputs.

---

📊 Evaluation Results

This repository focuses on the evaluation pipeline.
Aggregated scores (per-task, per-dimension, and per-model) can be produced locally via aggregate_scores.py.

For full experimental details, including:

• Cross-model comparison,
• Dimension-wise analysis,
• Error cases and ablations,

please refer to the paper (paper/main.pdf) and any public leaderboard associated with DeepResearch Bench II.

---

🏆 Submit to Leaderboard

To request an official leaderboard entry, contact:

• imlrz@mail.ustc.edu.cn
• dumingxuan@mail.ustc.edu.cn

Please prepare the following materials:

1. A working temporary API key with access to GPT-5.5
   • The key is required so that we can reproduce and verify the official evaluation under the same evaluator configuration.
   • It must remain valid and have sufficient quota for the agreed evaluation window.
   • Official OpenAI keys and keys for OpenAI-compatible endpoints are supported. For a compatible provider, also include the endpoint URL and exact model name.
   • Use a scoped, revocable key intended only for this evaluation, and revoke it after we confirm that evaluation is complete.
2. Raw generated reports
   • Provide one idx-<task_idx>.docx or idx-<task_idx>.md file per task, following the report/<model_name>/ layout used by this repository.
3. Reproducibility information
   • For an open-source model or agent, provide the repository and instructions needed to reproduce the reports.
   • For a closed-source system, provide its product page and/or API link.
4. Model metadata
   • Model and project names, project link, organization, and open-source license or proprietary status.

If available, you may also include your local result.jsonl and aggregated CSVs to speed up verification. Official leaderboard scores are recomputed by the maintainers.

API key safety: Do not commit a key to Git, attach it to a public issue or pull request, or include it in the report files. Contact us first by email to arrange private transfer of the temporary credential. The key is used only for submission verification and evaluation.

---

🛠️ Installation

Prerequisites

• Python 3.9+
• An OpenAI or OpenAI-compatible API endpoint and key with GPT-5.5 access

---

1. Environment configuration (.env)

Create a .env file in the project root DeepResearch-Bench-II to store API configuration and runtime parameters:

cd DeepResearch-Bench-II
touch .env
vim .env  # or use your favorite editor

Required config (replace with your own values):

OPENAI_API_URL=https://api.openai.com/v1/chat/completions
OPENAI_API_KEY=your-api-key
OPENAI_MODEL=gpt-5.5
OPENAI_REASONING_EFFORT=medium
OPENAI_MAX_OUTPUT_TOKENS=32768
OPENAI_TIMEOUT=600

PDF_DIR=report
OUT_JSONL=result.jsonl
TASKS_JSONL=tasks_and_rubrics.jsonl
CHUNK_SIZE=50
MAX_WORKERS=10
MAX_RETRIES=5
MAX_PAPER_CHARS=150000
LOG_FILE=run_evaluation.log

---

2. Install dependencies (supports uv / conda)

Option A: Use uv (recommended)

The project ships with pyproject.toml, so you can manage the virtual environment and dependencies via uv:

# Install uv (if not installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create/sync virtual environment and install dependencies
cd DeepResearch-Bench-II
uv sync

How to check whether uv is installed correctly

Run any of the following commands in your terminal:

# 1. Check version (recommended)
uv --version

# 2. Check executable path
which uv

# 3. Show help
uv --help

• If uv --version prints something like uv 0.x.y, it is installed correctly.
• If you see command not found or similar, uv is not installed or not on your PATH.

Option B: Use conda

# Create and activate a conda environment
conda create -n drbench-II python=3.10 -y
conda activate drbench-II

# Install Python dependencies
cd DeepResearch-Bench-II
pip install requests python-docx

You can then run all commands inside this conda environment.

---

Project Structure

DeepResearch-Bench-II/
├── assets/                    # Images and figures for README
│   ├── distribution.png
│   ├── intro.png
│   ├── main_result.png
│   └── method.png
├── report/                    # Input directory for model-generated reports
│   └── <model_name>/         # Per-model subdirectories
│       ├── idx-1.docx        # Model output for task 1
│       ├── idx-2.md          # Model output for task 2
│       └── ...
├── gpt_client.py              # OpenAI-compatible GPT-5.5 API client
├── run_evaluation.py          # Main evaluation script (batched rubric scoring logic)
├── aggregate_scores.py        # Score aggregation utility (produces CSV summaries)
├── tasks_and_rubrics.jsonl    # Tasks and rubrics (132 expert-derived tasks)
├── pyproject.toml             # Dependency management (uv / pip / conda)
├── .env_example               # Example configuration file
├── .env                       # Local configuration (user-created, ignored by Git)
├── .gitignore                 # Git ignore rules
└── README.md                  # This documentation

Note: Place DOCX or Markdown reports under report/<model_name>/idx-*.docx|md. The subdirectory name becomes the model identifier in output files.

---

Quick Start

1. Prepare your model outputs

Organize your model-generated reports under report with the following structure:

report/
├── ModelA/
│   ├── idx-1.docx
│   ├── idx-2.docx
│   └── ...
└── ModelB/
    ├── idx-1.md
    ├── idx-2.md
    └── ...

• Subdirectory name = model name (used in output JSONL).
• File name pattern = idx-<task_idx>.<ext> where <ext> is docx or md.

2. Run the evaluator

Run via uv (recommended)

cd DeepResearch-Bench-II
uv run python run_evaluation.py

Run directly with python

cd DeepResearch-Bench-II

# Use configuration from .env
python run_evaluation.py

# Or override configuration via CLI arguments
python run_evaluation.py \
    --pdf_dir grok \
    --out_jsonl result.jsonl \
    --chunk_size 50

This produces a JSONL file where each line has the form:

{"model": "ModelA", "idx": 1, "result": {...}}

3. Aggregate scores

After you have a merged JSONL of evaluation results (e.g., merged.jsonl), run:

python aggregate_scores.py \
  --input result.jsonl \
  --tasks-file tasks_and_rubrics.jsonl

This will generate multiple CSVs:

• agg_scores_inforecall.csv
• agg_scores_analysis.csv
• agg_scores_presentation.csv
• agg_scores_total.csv
• agg_scores_blocked.csv

Each CSV summarizes model performance by task (idx), including:

• Per-dimension scores,
• Overall averages,
• Blocked-rate statistics.

---

📜 License

This project uses a dual-license structure:

• Code (evaluation pipeline, scripts, utilities): Apache License 2.0
• Data (tasks, rubrics, and annotations): Per-task license follows the source article (see DATA_LICENSE)

The data in this benchmark is derived from expert-authored reports released under CC-BY-4.0 / CC-BY-NC-4.0 / CC0 licenses. Each entry in tasks_and_rubrics.jsonl carries a license field that records the license of its corresponding source article:

License	Count	Tasks (by idx)
CC BY 4.0	129	All except those listed below
CC BY-NC 4.0	2	idx=26, idx=110
CC0	1	idx=119

Accordingly, each task's rubrics and descriptions inherit the license of their source article. For the 129 CC BY 4.0 tasks, commercial use is permitted with attribution; for idx=26 and idx=110 (CC BY-NC 4.0), commercial use is not permitted.

If you use this benchmark in academic research, you are free to do so under all applicable licenses. Commercial use of the code is always permitted. Commercial use of the data depends on the per-task license — check the license field in tasks_and_rubrics.jsonl.

---

Acknowledgements

DeepResearch Bench II builds on the ideas and infrastructure of DeepResearch Bench and related benchmarks.
We thank all authors and annotators involved in collecting tasks, source articles, and rubrics.

---

Citation

If you use DeepResearch Bench II or this evaluation pipeline in your research, please cite:

@misc{li2026deepresearchbenchiidiagnosing,
      title={DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report}, 
      author={Ruizhe Li and Mingxuan Du and Benfeng Xu and Chiwei Zhu and Xiaorui Wang and Zhendong Mao},
      year={2026},
      eprint={2601.08536},
      archivePrefix={arXiv},
      url={https://arxiv.org/abs/2601.08536}, 
}