Microsoft-backed OpenAI announces launch of SWE-Lancer

In an X post, Open-AI said that, “Today we’re launching SWE-Lancer – a new, more realistic benchmark to evaluate the coding performance of AI models. SWE-Lancer includes over 1,400 freelance software engineering tasks from Upwork (UPWK) (UPWK), valued at $1 million USD total in real-world payouts.” “SWE-Lancer tasks span the full engineering stack, from UI/UX to systems design, and include a range of task types, from $50 bug fixes to $32,000 feature implementations. SWE-Lancer includes both independent engineering tasks and management tasks, where models choose between technical implementation proposals… As AI research advances, more realistic software engineering benchmarks are critical to assess model performance and understand socioeconomic implications. To facilitate future research, we open-source a unified Docker image and a public evaluation split, SWE-Lancer Diamond.”