Benchmark Methodologyv2.0.0

Jimenez, C. E., Yang, J., Wettig, A., et al. · ICLR 2024 · 2024

Evaluates language models on real GitHub issue resolution, measuring end-to-end task completion on production codebases. Informs our emphasis on outcome-based scoring over process metrics.

Zhou, S., Xu, F. F., Zhu, H., et al. · ICLR 2024 · 2023

Creates a reproducible web environment for evaluating agents on realistic tasks like flight booking and form filling. Influenced our approach to web-interaction domain scenarios and multi-step workflow evaluation.

Mialon, G., Fourrier, C., Swift, C., et al. · arXiv preprint · 2023

Proposes tiered difficulty levels for evaluating general-purpose AI assistants with unambiguous, verifiable answers. Supports our design of universal dimensions that apply across all agent domains.

Yao, S., Wang, Y., et al. · arXiv preprint · 2024

Measures reliability of tool-calling agents in conversational workflows, focusing on trial consistency across repeated runs. Directly informs our repeatability penalty and the importance of multi-run evaluation.

Xu, F. F., Zhou, Y., et al. · arXiv preprint · 2024

Evaluates agents on enterprise-style tasks across organizational workflows requiring cross-application coordination. Influenced our enterprise and automation domain templates.

Alzahrani, N., Barnett, S., et al. · arXiv preprint · 2025

Demonstrates that small changes in seeds and dataset splits produce measurable benchmark fluctuations, reinforcing the need for repeatability disclosure and confidence adjustments in published scores.

Liu, X., Zhu, Y., et al. · arXiv preprint · 2025

Comprehensive survey of agent evaluation across software engineering, web interaction, tool use, and scientific tasks. Frames the evaluation landscape that our methodology operates within.

Zhang, T., et al. · arXiv preprint · 2025

Finds that 75% of production teams bypass benchmarks in favor of A/B tests and user feedback. Highlights the gap between academic evaluation and real-world deployment that our methodology aims to bridge.

Zheng, L., Chiang, W.-L., Sheng, Y., et al. · NeurIPS 2023 · 2023

Foundational study showing strong LLM judges reach over 80% agreement with human evaluators — the same level as human–human agreement — while documenting position, verbosity, and self-enhancement biases. Grounds our panel judging design: single-answer rubric grading, anti-verbosity instructions, and a vendor-diverse panel with every judge’s individual score published.

Verga, P., Hofstätter, S., Althammer, S., et al. · arXiv preprint · 2024

Shows that a panel of smaller models from different families correlates better with human judgment than a single large judge, at lower cost, and materially reduces intra-model bias — a judge favouring outputs resembling its own. Direct basis for scoring every agent with eight judges drawn from eight competing labs rather than one.

Ye, S., Kim, D., Kim, S., et al. · ICLR 2024 · 2024

Demonstrates that decomposing evaluation into fine-grained skill-level rubrics correlates better with human judgment than a single holistic score. Grounds our requirement-level verdicts: judges mark each requirement met, partial or unmet with an evidence citation, and the numeric scores are recomputed from those verdicts.

Liang, P., Bommasani, R., Lee, T., et al. · TMLR · 2023

Argues evaluation should be a matrix of scenarios × metrics, with generic metrics measured inside every scenario, and declines to collapse results into a single number. Shapes our two-axis design — universal dimensions measured across domain scenarios — and the rule that both sub-scores are always published, not just the composite.

Burnell, R., Schellaert, W., Burden, J., et al. · Science · 2023

Argues aggregate scores hide where a system actually fails and calls for disaggregated, instance-level reporting. Why every judge’s individual scorecard, per-dimension medians and the panel’s spread (σ) are published alongside the headline number rather than summarized away.

Hendrycks, D., Burns, C., Basart, S., et al. · ICLR 2021 · 2021

Establishes the standard for multi-domain knowledge evaluation across 57 subjects. While designed for static models, its domain-stratified scoring approach influenced our per-domain evaluation structure.

Chen, M., Tworek, J., Jun, H., et al. · arXiv preprint · 2021

Introduces functional correctness as the evaluation criterion for code generation, using pass@k as a metric. Informs our coding domain template where output correctness is paramount.

Li, Y., et al. · arXiv preprint · 2025

Benchmarks agents on workflows averaging 1 million tokens and 90 tool calls, revealing how performance degrades on long-horizon tasks. Supports our multi-step evaluation dimension (U3) and the need for execution endurance testing.

Wang, Z., et al. · arXiv preprint · 2025

Tests agents on parallel processes, resource constraints, and unexpected disruptions. Validates our error handling dimension (U4) and the importance of testing agent behavior under adversarial or constrained conditions.

Siegel, Z., et al. · arXiv preprint · 2025

Evaluates agents on scientific paper reproducibility — following multi-step procedures with precision. Reinforces the methodology's emphasis on instruction interpretation (U2) and task completion fidelity (U1).

Park, J., et al. · arXiv preprint · 2025

Finds efficiency metrics appeared in only 14 of 23 benchmark papers; fairness in only 1. Underscores the gap in holistic evaluation that our output quality dimension (U6) and transparency disclosures address.