SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code r

Related evidence

These records share source-supplied organisations, an exact publisher byline, automatic topics or regions. The reason is shown on every link; related does not mean supporting, agreeing with or verifying this record.

How to cite this record

ethics.ai (9 August 2026), “SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring,” evidence record 17973, https://ethics.ai/record/17973 (originally published by HuggingFace Daily Papers).

Use and limitations

This page is a stable index and citation surface for a source record. ethics.ai did not author the underlying report and has not independently verified every claim. Automatic topics may be imperfect. For consequential use, quote and cite the original publisher.