Agentic Evaluation Tasks
This category contains annotation task designs for evaluating LLM agents: coding agents, web and GUI agents, multi-agent systems, process reward modeling, and trajectory review.
Run these designs in Potato. See the agent evaluation documentation for trace displays, process-reward annotation, and multi-agent schemes.
Tasks
| Design | Description | Reference |
|---|---|---|
| agentboard-progress-scoring | Assess multi-turn LLM agent progress by identifying achieved milestones, scoring overall progress,… | Ma et al., NeurIPS 2024 |
| agentrewardbench-trajectory-scoring | Evaluate web agent trajectories by rating step-level quality across multiple dimensions, judging overall… | Lù et al., arXiv 2025 |
| aitw-mobile-trajectory-review | Step-by-step review of Android device-control episodes in the style of Android in the Wild | Rawles et al., NeurIPS 2023 |
| androidworld-mobile-agent-eval | Evaluation of autonomous agents performing tasks in dynamic Android environments | Rawles et al., ICLR 2025 |
| apibank-tool-call-review | Per-call review of API (tool) calls in tool-augmented assistant dialogues, based on API-Bank | Li et al., EMNLP 2023 |
| bigcodebench-human-baseline | Evaluate agent-generated code solutions for BigCodeBench tasks | Zhuo et al., ICLR 2025 |
| codeprm-code-process-reward | Process reward annotation for step-by-step code generation with execution feedback | Li et al., ACL 2025 Findings |
| codeultrafeedback-code-preference | Pairwise comparison of code responses with multi-dimensional quality rating | Weyssow et al., arXiv 2024 |
| devbench-repo-eval | Evaluate AI-generated repositories across the full software development lifecycle | Li et al., arXiv 2024 |
| mast-failure-taxonomy | Annotate multi-agent system traces to identify failure modes from the MAST taxonomy, rate severity,… | Cemri et al., arXiv 2025 |
| mind2web-web-agent-tasks | Web agent task annotation | Deng et al., NeurIPS 2023 |
| multiagentbench-collaboration-scorecard | Coordination-quality rating of multi-agent LLM runs, based on MultiAgentBench | Zhu et al., ACL 2025 |
| osworld-desktop-agent-eval | Evaluation of multimodal agents performing open-ended tasks in real desktop environments | Xie et al., NeurIPS 2024 |
| prm800k-step-verification | Step-by-step verification of math reasoning chains based on OpenAI’s PRM800K dataset | Lightman et al., ICLR 2024 |
| processbench-math-error-steps | Earliest-error identification in step-by-step math solutions, based on ProcessBench | Zheng et al., ACL 2025 |
| r-judge-agent-safety | Safety risk evaluation of LLM agent interaction records across diverse risk scenarios | Yuan et al., EMNLP Findings 2024 |
| refactorbench-multi-file-eval | Evaluate multi-file refactoring operations generated by coding agents | Gautam et al., ICLR 2025 |
| swe-prm-coding-process-reward | Step-level process reward dataset for coding agents from the SWE-PRM paper | Gandhi et al., arXiv 2025 |
| swebench-code-agent-eval | Evaluation of code agent solutions to real GitHub issues | Jimenez et al., ICLR 2024 |
| swebench-plus-patch-screening | Screen and compare model-generated patches against gold patches for SWE-Bench+ instances | Aleithan et al., arXiv 2024 |
| swebench-verified-validation | Manually validate GitHub issues from SWE-bench to ensure they are well-specified, have adequate test… | Chowdhury et al., OpenAI 2024 |
| tau-bench-agent-eval | Evaluate tool-agent-user interactions in customer service domains by judging task success, conversation… | Yao et al., arXiv 2024 |
| trajeval-staged-evaluation | Evaluate code agent trajectories decomposed into search, edit, and verification stages, rating quality of… | Kim et al., arXiv 2026 |
| visualwebarena-visual-agent-eval | Evaluation of multimodal web agents on visually grounded web tasks | Koh et al., ACL 2024 |
| webarena-web-agent-eval | WebArena is a reproducible, self-hosted web environment with 812 tasks for testing autonomous language agents | Zhou et al., ICLR 2024 |
| whowhen-failure-attribution | Failure attribution for LLM multi-agent systems, based on the Who&When dataset | Zhang et al., ICML 2025 |
Quick Start
# Navigate to a specific task
cd agentic/<task-name>
# Run with Potato
potato start config.yaml
Task Count
Total: 26 agentic evaluation tasks