arXiv cs.AIbenchmark6h agoenergy 97

Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages

image: arXiv cs.AI

arXiv:2611.10304v2 Announce Type: replace-cross Abstract: Challenging repository-level benchmarks remain limited for many programming language ecosystems, where filtering scarce candidates with ambiguous descriptions or defective tests can discard valuable engineering problems. We introduce Longtail-SWE, a benchmark of 202 issue-resolution tasks across 13 languages with limited coverage in existing repository-level benchmarks: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift. Its construction uses a solver-judge-repairer pipeline: a solver attempts each candidate task, and a judge reviews the task together with the solver's trajectory and test results to guide revisions to tests and task descriptions. These revisions aim to correct tests that reject valid alternative implementations or accept incorrect solutions while retaining challenging tasks and preserving their intended requirements. We evaluate nine model configurations, including frontier models, and analyze task-solving performance, consistency across repeated attempts, and resource usage. We release all benchmark instances, executable environments, complete construction and evaluat

read the original at arXiv cs.AI →
This source publishes only a summary in its feed, so the full article lives at the outlet. The link above and below goes to the original.

more from the stream

Control Reinforcement Learning: Interpretable Token-Level Steering of LLMs via Sparse… (arXiv cs.AI) Reward-Driven Collider Physics Challenges for Self-Improving Agentic AI via Reinforcement… (arXiv cs.AI) What Roles Do System Prompts Really Assign? Validating and Steering Occupational Role Vectors… (arXiv cs.AI) ← back to upsy.si