← Back to News Paper Release · KDD 2026 Main Conference (CCF-A) · February 2026

SWE-Bench Mobile: We Asked Coding Agents to Ship a Real iOS Feature. The Best Score Is 12%.

Accepted to KDD 2026's Applied Data Science Track at the main conference (CCF-A). Built with Xiaohongshu on a real, shipping iOS app.

Muxin Tian, Zhe Wang, Blair Yang, Zhenwei Tang, Kunlun Zhu, Honghua Dong, Hanchen Li, Xinni Xie, Guangjing Wang, Jiaxuan You
Coolwei AI Lab × University of Toronto × Xiaohongshu Inc. × University of Illinois Urbana-Champaign × University of California, Berkeley
February 20265 min read
Research Detail Project & Leaderboard arXiv
50real iOS feature tickets
449human-written tests
~500Klines of shipping code
12%best task pass rate

Can today's strongest coding agents actually ship a real mobile feature? We took 50 feature tickets straight out of Xiaohongshu's production iOS app — PRDs, Figma designs, ~500K lines of mixed Swift / Objective-C, and 449 human-written tests grading them — and pointed every leading agent × model combo at the work. The best score is 12%. The paper is now in KDD 2026's Applied Data Science Track at the main conference (CCF-A).

The takeaway: today's coding agents can complete meaningful parts of real production work, but independent delivery of industrial mobile features remains clearly out of reach. Even the strongest commercial agent + model combos clear only 12% of tasks under the strict all-tests-pass bar.

KDD 2026 · Jeju Island, Korea

Three members of the SWE-Bench Mobile author team holding the poster at the KDD 2026 venue on Jeju Island.
Members of the author team with the SWE-Bench Mobile poster at the KDD 2026 venue.
The screen · before → afterFigma referenceGround-truth diffBeforeAfter14pxFeedItemFooter.swift 35override func layout() { 36  super.layoutSubviews()+38  if cfg.timeEmphasis {+39    likeBtn.hidden = true+43    timeLabel.font = .cap+46    timeLabel.frame = r+55  } 57}Graded by 449 hand-written tests — about 9 per task.
Redrawn from the paper, schematic. What one task looks like. A product change, a Figma reference, and the diff the agent has to land — graded against a hand-written test suite.

Why mobile is the honest hard test

Most coding-agent benchmarks ask a model to fix a bug it can see, against a test that already exists, inside a repo that may well have been in pretraining. Real mobile work is none of that. Tasks come in as feature tickets, not bug reports. The spec lives in a PRD and a Figma file, not a GitHub issue. The fix usually touches UI, data, interaction, feature flags, and engineering conventions at the same time. And the codebase is hundreds of thousands of lines that the model has not seen.

SWE-Bench Mobile puts all of that in one evaluation. The question it answers: can a complete agent system take a real product requirement and land it cleanly inside a production iOS app?

What's actually in it

Task sourceXiaohongshu's production iOS app
Task typeFeature additions, not bug fixes
Codebase size~500K lines of mixed Swift / Objective-C
Task inputsPRD + Figma design + reference images + codebase snapshot
Evaluation scale50 tasks, 449 human-written tests
Release formatHosted challenge with a public leaderboard
VenueKDD 2026 Main Conference (CCF-A)

The leaderboard, in one chart

Task success rate · 22 agent + model combos04812task success rate (%)Cursor + Opus 4.512.0%Cursor + Sonnet 4.512.0%Codex + GLM 4.612.0%Codex + Sonnet 4.510.0%Codex + GPT 510.0%Claude Code + GLM 4.610.0%Claude Code + Sonnet 4.510.0%Cursor + GPT 5.28.0%Claude Code + Opus 4.58.0%Claude Code + Haiku8.0%OpenCode + GLM 4.68.0%Cursor + Gemini 3 Pro6.0%OpenCode + GPT 5.16.0%Codex + Opus 4.54.0%OpenCode + Sonnet 4.54.0%OpenCode + GLM 4.74.0%OpenCode + Gemini 3 Pro4.0%OpenCode + GPT 5.24.0%Cursor + GPT 5.12.0%OpenCode + Opus 4.52.0%OpenCode + GPT 52.0%Codex + GPT 5.10.0%three tied at the 12% ceiling
Redrawn from the paper. 22 combos, three tied at 12%, a long tail in the low single digits. Live leaderboard.
Claude Opus 4.5 in four agents · task success rate (%)0481212%Cursor4%Codex8%Claude Code2%OpenCodesame model, 6× spreadSonnet 4.5 spans 12% → 4% across the same four agents; GLM 4.6, 12% → 8%.
Redrawn from the paper. Same model, four agents. The scaffold decides as much of the outcome as the model itself.
The SWE-Bench Mobile poster during the KDD 2026 poster session, with attendees reading and discussing the results.
The poster session at KDD 2026. Applied Data Science Track, Main Conference (CCF-A).

Five findings backed by the data

  • Same model, different agent — up to 6× spread. It's the scaffold, not just the LLM. Reports that name only the model are missing half the story.
  • Top score 12%. Top test pass rate 28.1%. Agents land partial wins, then trip over production details.
  • Simpler beats fancier. A plain "Defensive Programming" prompt outperforms more elaborate strategies by +7.4 pp on test pass rate.
  • Cross-module work remains the weakest link. Tasks touching 7+ files drop to 2% success. Localized changes are far easier.
  • What breaks the patch isn't the code — it's the conventions. Missing feature flags, half-built data models, the one file no one remembered, UI components that don't match the rest of the app.

Submit your stack to the leaderboard

SWE-Bench Mobile is a hosted challenge for coding-agent teams, foundation-model vendors, and mobile-development researchers. Submissions run server-side, so the test set never leaks into anyone's training data. What it offers is an evaluation coordinate system that stays close to real shipping mobile engineering.

Submit your agent to the leaderboard

The hosted challenge takes submissions today; one submission places you alongside every other team. Project page and public leaderboard are live; the paper is on arXiv and now in the KDD 2026 Main Conference proceedings (CCF-A).

Project & Leaderboard arXiv:2602.09540