# Datasets and benchmarks

> links + original short summaries; content belongs to publishers

6 entries.

JSON: https://agentpaywire.com/api/library/datasets-benchmarks.json
Markdown: https://agentpaywire.com/library/datasets-benchmarks.md
JSONL: https://agentpaywire.com/library.jsonl
Feed: https://agentpaywire.com/library/feed.json
Schema: https://agentpaywire.com/schemas/library.schema.json

### 2024-06-17

- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://arxiv.org/abs/2406.12045)
  - id: tau-bench
  - publisher: arXiv
  - authors: Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik Narasimhan
  - date_published: 2024-06-17
  - type: research-paper
  - tags: benchmark, tools, research
  - summary: Benchmark of dynamic conversations between a language-model-simulated user and an agent equipped with domain-specific API tools and policy guidelines. Success is scored by comparing the final database state with an annotated goal state, and the paper introduces the pass^k reliability metric.

### 2023-11-21

- [GAIA: a benchmark for General AI Assistants](https://arxiv.org/abs/2311.12983)
  - id: gaia-benchmark
  - publisher: arXiv
  - authors: Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, Thomas Scialom
  - date_published: 2023-11-21
  - type: research-paper
  - tags: benchmark, research
  - summary: Benchmark of real-world questions for general AI assistants that require reasoning, multimodality, web browsing and tool use.

### 2023-10-10

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://arxiv.org/abs/2310.06770)
  - id: swe-bench-paper
  - publisher: arXiv
  - authors: Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan
  - date_published: 2023-10-10
  - type: research-paper
  - tags: benchmark, research
  - summary: Evaluation framework of 2,294 software engineering problems drawn from real GitHub issues and pull requests across 12 popular Python repositories.

### 2023-07-25

- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://arxiv.org/abs/2307.13854)
  - id: webarena
  - publisher: arXiv
  - authors: Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, Graham Neubig
  - date_published: 2023-07-25
  - type: research-paper
  - tags: benchmark, web-agents, research
  - summary: Highly realistic and reproducible web environment for building and testing language-guided agents that perform tasks on the web. The authors list shows the first 12 of a longer list.

### Undated

- [SWE-bench Leaderboards](https://www.swebench.com/)
  - id: swe-bench-leaderboards
  - publisher: SWE-bench
  - date_published: null
  - type: docs
  - tags: benchmark, leaderboard
  - summary: Official SWE-bench site with leaderboards for SWE-bench, Verified, Multilingual, Multimodal and Lite variants, plus related tools.

- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://robotics-transformer-x.github.io/)
  - id: open-x-embodiment
  - publisher: Open X-Embodiment Collaboration
  - date_published: null
  - type: docs
  - tags: dataset, robotics
  - summary: Project page of the Open X-Embodiment Collaboration presenting robotic learning datasets and RT-X models. It asks whether the consolidation around large pretrained models seen in NLP and computer vision can happen in robotics.