Qwen-UI-Agent Claims It Beats GPT-5.6 and Claude Opus 4.8. The Benchmark Table Says Otherwise
Alibaba's Qwen-UI-Agent report claims a win over GPT-5.6 and Claude Opus 4.8, but the full nine-benchmark table is a 4-4-1 split, and the 27B model behind the headline scores has no public weights, API, or release yet.
Contents5
What Alibaba's Qwen-UI-Agent actually claims
On August 20, 2026, Alibaba's Tongyi Lab published the Qwen-UI-Agent technical report and framed it as a clean win over the field's best closed models. Coverage ran with headlines like "surpassing GPT-5.6 and Claude 4.8," built on a 27B-parameter model tested on real phones and a simulated desktop environment.
The individual numbers are real. Qwen-UI-Agent hits 82.1% on MobileWorld, 12 points ahead of GPT-5.6 Sol, and 92.2% on MobileWorld-Real, ahead of Claude Opus 4.8 and Gemini 3.1 Pro. On AndroidDaily it scores 97.5%. Those are legitimate, strong mobile-agent numbers.
What most of the coverage left out is the rest of the scoreboard, and the fact that the model producing these scores isn't something you can actually go use.
The benchmark table nobody published in full
Qwen-UI-Agent's own technical report runs nine benchmarks, not three. Mobile and Android tasks are where it wins clearly. Desktop control, the OSWorld family, is where it loses to Claude Opus 4.8, and open-web research tasks are where it loses badly to GPT-5.5.
| Benchmark | Qwen-UI-Agent | Best rival | Winner |
|---|---|---|---|
| MobileWorld | 82.1% | Claude Opus 4.8: 67.5% | Qwen |
| MobileWorld-Real | 92.2% | Seed 2.1 Pro: 88.7% | Qwen |
| AndroidDaily | 97.5% | Seed 2.1 Pro: 95.2% | Qwen |
| WebArena | 73.6% | Prior best: 71.9% | Qwen |
| OSWorld-Verified (desktop) | 79.5% | Claude Opus 4.8: 83.4% | Claude |
| OSWorld-v2, partial credit | 40.0% | Claude Opus 4.8: 54.8% | Claude |
| OSWorld-v2, binary | 13.9% | Claude Opus 4.8: 20.6% | Claude |
| BrowseComp (English research) | 64.1% | GPT-5.5: 90.1% | GPT |
| BrowseComp-ZH (Chinese) | 75.0% | Apodex-1.0-mini: 80.6% | Apodex |
That's a 4-4-1 split, not a sweep. Alibaba's own report supports the "beats GPT-5.6 and Claude on phones" claim. It does not support "surpasses GPT-5.6 and Claude," full stop, which is the version that ended up in most headlines. On the single benchmark where it goes head to head with GPT-5.5 on open-ended web research, it loses by 26 points.
This is a familiar shape if you've followed [how Alibaba's Wan 3.0 video launch played out against actual testing](https://questloops.com/blog/alibaba-s-wan-3-0-promises-30-second-video-testing-found-a-15-second-wall): a real, sometimes genuinely impressive result gets marketed as a category-wide win, and the caveats live three paragraphs into the technical report instead of the press release.
You can't download the model that set these scores
Here's the detail almost no coverage mentioned: the 27B model behind the headline numbers isn't public.
The report site at `tongyi-mai.github.io` is just that, a website, and its own GitHub repo says so directly: "this is not the Qwen-UI-Agent implementation repository" and it "does not contain the model, training code, or agent implementation." The actual code lives in a separate repo, Tongyi-MAI/MAI-UI, which has 1.8k stars and 178 forks but zero GitHub releases.
What you can actually download from that repo is MAI-UI-2B and MAI-UI-8B on Hugging Face. The 27B model that scored 82.1% on MobileWorld, and the 235B-A22B variant referenced elsewhere in the report, are not distributed. There are no weights, no API, and no rate card. If you want the model that produced these benchmark numbers, your only option right now is to wait.
This matters for anyone deciding whether to plan around Qwen-UI-Agent this quarter: it's a research artifact, not a shipping product, and the gap between "Alibaba published a benchmark" and "you can deploy this" is currently indefinite.
Where the failures actually come from
The report's failure analysis is the most useful part and the least quoted. On real-device testing across more than 100 physical phones and 150-plus apps, Alibaba breaks down what causes a task to fail:
- 52% of failures come from the screen itself: ads, paywalls, CAPTCHAs, blank loading states, or UI elements the model simply misreads
- 40.3% are the agent's own fault: wrong action, wrong target, lost plan
- The rest fall into ambiguous or unscoreable cases
That 52% number is arguably more useful than any of the headline scores, because it tells you where the ceiling on GUI agents currently sits. Even a model that reasons perfectly still loses to a cookie banner or a CAPTCHA a meaningful chunk of the time. That's not a Qwen-specific problem; it's true of [Osaurus](https://questloops.com/tools/osaurus) and every other agent trying to drive a real screen, and it's worth remembering the next time a vendor claims a benchmark score close to 100%.
What this means if you're evaluating GUI agents
Treat the MobileWorld and AndroidDaily numbers as real and specific to mobile, on-device tasks. Treat "surpasses GPT-5.6 and Claude" as marketing shorthand for a benchmark table that actually splits 4-4-1. And treat the whole report as a preview of where Alibaba's agent research is headed rather than something you can put into production this month, since the model behind the headline score has no public weights or API yet.
If you need a GUI or coding agent you can actually run today, the field already has usable open options: [Kilo](https://questloops.com/tools/kilo) routes across 500+ hosted models for IDE and CLI work, and Osaurus runs agent loops entirely on-device for Mac users who want the same local-first idea Qwen-UI-Agent is gesturing at, minus the wait for weights.
The mobile numbers here are legitimate and worth watching. The "beats everything" framing isn't supported by Alibaba's own benchmark table, and until the 27B model actually ships, this is a research report worth bookmarking, not a tool worth switching to.