it contains 418 real-world tasks across 6 domains and 3 difficulty levels to evaluate capability synergy, featuring over 2,000 stepwise checkpoints that average 10+ person-hours of manual annotation per task.
大多数人认为AI评估可以通过相对简单的自动化流程完成。然而，作者提出的评估基准需要每个任务超过10小时的人工标注和2000多个检查点，这暗示了真正评估AI代理能力的复杂性和成本远超行业普遍认知。这一观点挑战了AI评估领域的效率优先思维，强调了高质量评估需要大量人工投入的现实。
Study: Myocarditis risk 37 times higher for children with COVID-19 than uninfected peers | American Academy of Pediatrics. (n.d.). Retrieved October 10, 2021, from https://www.aappublications.org/news/2021/08/31/covid-myocarditis-risk-children-083121
Zhou, F., Shefer, A., Wenger, J., Messonnier, M., Wang, L. Y., Lopez, A., Moore, M., Murphy, T. V., Cortese, M., & Rodewald, L. (2014). Economic Evaluation of the Routine Childhood Immunization Program in the United States, 2009. Pediatrics, 133(4), 577–585. https://doi.org/10.1542/peds.2013-0698
Sun, W., McCroskery, S., Liu, W.-C., Leist, S. R., Liu, Y., Albrecht, R. A., Slamanig, S., Oliva, J., Amanat, F., Schäfer, A., Dinnon, K. H., Innis, B. L., García-Sastre, A., Krammer, F., Baric, R. S., & Palese, P. (2020). A Newcastle disease virus (NDV) expressing membrane-anchored spike as a cost-effective inactivated SARS-CoV-2 vaccine. BioRxiv, 2020.07.30.229120. https://doi.org/10.1101/2020.07.30.229120