Open the "Project Phoenix" room and send the message "status update".
success1.0
The final screenshot shows the Project Phoenix room with Alice's sent message 'status update' visible with timestamp/checkmark and an empty composer.
Created by'%3e%3cg%20id='BG'%3e%3c/g%3e%3cpath%20id='Border'%20fill-rule='evenodd'%20clip-rule='evenodd'%20d='M0.199951%200.200012H1.69995H101.4H102.024L102.463%200.642199L121.063%2019.3422L121.5%2019.781V20.4V64V65.5H120H20.4H19.7786L19.3393%2065.0607L0.639291%2046.3607L0.199951%2045.9213V45.3V1.70001V0.200012ZM3.19995%205.32133V44.6787L18.8999%2060.3786V21.0213L3.19995%205.32133ZM5.32127%203.20001L21.0213%2018.9H116.392L100.776%203.20001H5.32127ZM118.5%2062.5H21.8999V21.9H118.5V62.5Z'%20fill='var(--fill-0,%20%23001A72)'/%3e%3cpath%20id='Letter%2003'%20d='M46.2%2046.5C46.2%2049.8%2044.1%2052.6%2038.8%2052.6C36.5%2052.6%2034.7%2052.1%2033%2051.4V46.9C34.7%2047.8%2036.8%2048.3%2038.7%2048.3C40.4%2048.3%2041.3%2047.8%2041.3%2046.8C41.3%2043.8%2033%2044.7%2033%2038.5C33%2034.7%2036.1%2032.3%2040.3%2032.3C42.1%2032.3%2043.7%2032.7%2045.4%2033.7V38.5C43.1%2037.1%2041.5%2036.6%2040.2%2036.6C38.6%2036.6%2037.8%2037.2%2037.8%2038.1C37.7%2041%2046.2%2039.9%2046.2%2046.5Z'%20fill='var(--fill-0,%20%23001A72)'/%3e%3cpath%20id='Letter%2002'%20d='M79.3%2032.8L73%2052.2H68.2L64%2038.8L59.7%2052.1H54.9L48.6%2032.7H54L57.5%2045.7L61.6%2032.7H66.4L70.5%2045.7L74%2032.7H79.3V32.8Z'%20fill='var(--fill-0,%20%23001A72)'/%3e%3cpath%20id='Letter%2001'%20d='M109.4%2039.4V52.2H104.6V40.2C104.6%2038.1%20103.1%2036.9%20101.5%2036.9C99.8%2036.9%2098.4%2038.1%2098.4%2040.2V52.2H93.6V40.2C93.6%2038.1%2092.1%2036.9%2090.5%2036.9C88.8%2036.9%2087.4%2038.1%2087.4%2040.2V52.2H82.6V32.8H87.1V34.2C88.5%2032.8%2090%2032.3%2091.8%2032.3C93.8%2032.3%2095.6%2033.2%2096.8%2034.8C98.4%2033.1%20100.4%2032.3%20102.7%2032.3C106.6%2032.4%20109.4%2035.2%20109.4%2039.4Z'%20fill='var(--fill-0,%20%23001A72)'/%3e%3c/g%3e%3cdefs%3e%3cclipPath%20id='clip0_0_712'%3e%3crect%20width='122'%20height='66'%20fill='white'/%3e%3c/clipPath%3e%3c/defs%3e%3c/svg%3e)
AppControlBench
Compare models, tools, cost, and test runs across real iOS app-control tasks.
Leaderboard
Completion assigns 1 to success, 0.5 to partial, and 0 to failure. Ties are broken by mean time on successful runs.
| Rank | Configuration | Tool | Completion | Time / run | Cost / run | Task outcomes |
|---|---|---|---|---|---|---|
| 01 | qwen3.8-max (high)· argent v0.15.0 | argentv0.15.0 | 98.3% | 2m 51s | $0.32 | Passed 97%, Partial 3%, Failed 0% Time / run2m 51sCost / run$0.32 |
| 02 | haiku-4.5 (high)· argent v0.15.0 | argentv0.15.0 | 97.5% | 1m 07s | $0.22 | Passed 95%, Partial 5%, Failed 0% Time / run1m 07sCost / run$0.22 |
| 03 | qwen3.8-max (low)· argent v0.15.0 | argentv0.15.0 | 95.8% | 1m 14s | $0.27 | Passed 93%, Partial 5%, Failed 2% Time / run1m 14sCost / run$0.27 |
| 04 | gpt-5.4-mini (high)· argent v0.15.0 | argentv0.15.0 | 95.0% | 1m 14s | $0.13 | Passed 92%, Partial 7%, Failed 2% Time / run1m 14sCost / run$0.13 |
| 05 | haiku-4.5 (low)· argent v0.15.0 | argentv0.15.0 | 94.2% | 58s | $0.20 | Passed 92%, Partial 5%, Failed 3% Time / run58sCost / run$0.20 |
| 06 | qwen3.8-max (high)· agent-device v0.17.6 | agent-devicev0.17.6 | 92.5% | 3m 20s | $0.23 | Passed 88%, Partial 8%, Failed 3% Time / run3m 20sCost / run$0.23 |
| 07 | qwen3.8-max (low)· agent-device v0.17.6 | agent-devicev0.17.6 | 89.2% | 2m 51s | $0.26 | Passed 83%, Partial 12%, Failed 5% Time / run2m 51sCost / run$0.26 |
| 08 | gpt-5.4-mini (high)· agent-device v0.17.6 | agent-devicev0.17.6 | 85.0% | 1m 29s | $0.10 | Passed 77%, Partial 17%, Failed 7% Time / run1m 29sCost / run$0.10 |
| 09 | haiku-4.5 (low)· agent-device v0.17.6 | agent-devicev0.17.6 | 84.2% | 1m 53s | $0.13 | Passed 80%, Partial 8%, Failed 12% Time / run1m 53sCost / run$0.13 |
| 10 | haiku-4.5 (high)· agent-device v0.17.6 | agent-devicev0.17.6 | 83.3% | 1m 25s | $0.15 | Passed 75%, Partial 17%, Failed 8% Time / run1m 25sCost / run$0.15 |
Tool comparison
Completion score by model and tool, partial outcomes at half credit. Every judged run counts, failures included; judge errors and stale reruns do not.Completion score by model and tool, partial outcomes at half credit. Every judged run counts, failures included; judge errors and stale reruns do not.
Average completion score
86.1%n=360
- low89.2%n=60
- high92.5%n=60
- low82.5%n=60
- high85.0%n=60
- low84.2%n=60
- high83.3%n=60
| model | argentv0.15.0 | agent-devicev0.17.6 | no tool |
|---|---|---|---|
| gpt-5.4-mini | 88.8%n=120 | 83.8%n=120 | 30.8%n=120 |
| low | 82.5%n=60 | 82.5%n=60 | 15.8%n=60 |
| high | 95.0%n=60 | 85.0%n=60 | 45.8%n=60 |
| haiku-4.5 | 95.8%n=120 | 83.8%n=120 | 13.3%n=120 |
| low | 94.2%n=60 | 84.2%n=60 | 15.0%n=60 |
| high | 97.5%n=60 | 83.3%n=60 | 11.7%n=60 |
| qwen3.8-max | 97.1%n=120 | 90.8%n=120 | 57.5%n=120 |
| low | 95.8%n=60 | 89.2%n=60 | 53.3%n=60 |
| high | 98.3%n=60 | 92.5%n=60 | 61.7%n=60 |
| overall | 93.9%n=360 | 86.1%n=360 | 33.9%n=360 |
Breakdowns
Average time uses successful runs only - failures, partials and timeouts are excluded. Average cost uses the same runs as the completion score, failures included, skipping runs with no price data.Average time uses successful runs only - failures, partials and timeouts are excluded. Average cost uses the same runs as the completion score, failures included, skipping runs with no price data.
Cost by model
Cost by tool
Time by model
Time by tool
Cost efficiency
Average price per task vs completion score, both over every judged run - failures included, so a model that fails cheaply is not flattered. Colour identifies the model, shape the tool; higher and cheaper is better.Average price per task vs completion score, both over every judged run - failures included, so a model that fails cheaply is not flattered. Colour identifies the model, shape the tool; higher and cheaper is better.
Model:
Tool:
Methodology
1ArrangeA fresh simulator clone and re-seeded app state; the agent gets only the task text and its tool
2ActIt drives the device until it decides it is done or the clock runs out
3CaptureThe final screenshot
4JudgeA vision model matches that against the judge screen description
The judge is never told which model produced a screenshot, though the action names reveal which tool drove the device.