existing benchmarks often overlook these non-functional requirements, rewarding functionally correct but structurally arbitrary solutions.
大多数人认为现有的LLM代码生成评估已经足够全面，但作者指出当前基准测试忽略了非功能性需求，只奖励功能正确但结构随意的解决方案，这挑战了当前评估方法的充分性。
2 Matching Annotations
- Last 7 days
-
arxiv.org arxiv.org
-
- Apr 2026
-
-
Tests reject correct solutions: We audited a 27.6% subset of the dataset that models often failed to solve and found that at least 59.4% of the audited problems have flawed test cases that reject functionally correct submissions
大多数人认为代码测试是客观公正的，能够准确评估模型的真实能力。但作者发现，近60%的测试案例存在缺陷，会拒绝功能上正确的解决方案。这一发现挑战了AI评估领域的共识，表明我们广泛使用的基准测试可能存在系统性问题，无法准确反映模型的实际编程能力。
-