Open the "Project Phoenix" room and send the message "status update".
success1.0
The final screenshot shows the Project Phoenix room with Alice's sent message 'status update' visible with timestamp/checkmark and an empty composer.
Created by'%3e%3cg%20id='BG'%3e%3c/g%3e%3cpath%20id='Border'%20fill-rule='evenodd'%20clip-rule='evenodd'%20d='M0.199951%200.200012H1.69995H101.4H102.024L102.463%200.642199L121.063%2019.3422L121.5%2019.781V20.4V64V65.5H120H20.4H19.7786L19.3393%2065.0607L0.639291%2046.3607L0.199951%2045.9213V45.3V1.70001V0.200012ZM3.19995%205.32133V44.6787L18.8999%2060.3786V21.0213L3.19995%205.32133ZM5.32127%203.20001L21.0213%2018.9H116.392L100.776%203.20001H5.32127ZM118.5%2062.5H21.8999V21.9H118.5V62.5Z'%20fill='var(--fill-0,%20%23001A72)'/%3e%3cpath%20id='Letter%2003'%20d='M46.2%2046.5C46.2%2049.8%2044.1%2052.6%2038.8%2052.6C36.5%2052.6%2034.7%2052.1%2033%2051.4V46.9C34.7%2047.8%2036.8%2048.3%2038.7%2048.3C40.4%2048.3%2041.3%2047.8%2041.3%2046.8C41.3%2043.8%2033%2044.7%2033%2038.5C33%2034.7%2036.1%2032.3%2040.3%2032.3C42.1%2032.3%2043.7%2032.7%2045.4%2033.7V38.5C43.1%2037.1%2041.5%2036.6%2040.2%2036.6C38.6%2036.6%2037.8%2037.2%2037.8%2038.1C37.7%2041%2046.2%2039.9%2046.2%2046.5Z'%20fill='var(--fill-0,%20%23001A72)'/%3e%3cpath%20id='Letter%2002'%20d='M79.3%2032.8L73%2052.2H68.2L64%2038.8L59.7%2052.1H54.9L48.6%2032.7H54L57.5%2045.7L61.6%2032.7H66.4L70.5%2045.7L74%2032.7H79.3V32.8Z'%20fill='var(--fill-0,%20%23001A72)'/%3e%3cpath%20id='Letter%2001'%20d='M109.4%2039.4V52.2H104.6V40.2C104.6%2038.1%20103.1%2036.9%20101.5%2036.9C99.8%2036.9%2098.4%2038.1%2098.4%2040.2V52.2H93.6V40.2C93.6%2038.1%2092.1%2036.9%2090.5%2036.9C88.8%2036.9%2087.4%2038.1%2087.4%2040.2V52.2H82.6V32.8H87.1V34.2C88.5%2032.8%2090%2032.3%2091.8%2032.3C93.8%2032.3%2095.6%2033.2%2096.8%2034.8C98.4%2033.1%20100.4%2032.3%20102.7%2032.3C106.6%2032.4%20109.4%2035.2%20109.4%2039.4Z'%20fill='var(--fill-0,%20%23001A72)'/%3e%3c/g%3e%3cdefs%3e%3cclipPath%20id='clip0_0_712'%3e%3crect%20width='122'%20height='66'%20fill='white'/%3e%3c/clipPath%3e%3c/defs%3e%3c/svg%3e)
AppControlBench
Compare models, tools, cost, and test runs across real iOS app-control tasks.
Leaderboard
Completion assigns 1 to success, 0.5 to partial, and 0 to failure. Ties are broken by mean time on successful runs.
| Rank | Configuration | Tool | Completion | Time / run | Cost / run | Task outcomes |
|---|---|---|---|---|---|---|
| 01 | haiku-4.5 (high)· argent v0.15.0 | argentv0.15.0 | 98% | 1m 07s | $0.22 | Passed 95%, Partial 5%, Failed 0% Time / run1m 07sCost / run$0.22 |
| 02 | gpt-5.4-mini (high)· argent v0.15.0 | argentv0.15.0 | 95% | 1m 14s | $0.13 | Passed 92%, Partial 7%, Failed 2% Time / run1m 14sCost / run$0.13 |
| 03 | haiku-4.5 (low)· argent v0.15.0 | argentv0.15.0 | 94% | 58s | $0.20 | Passed 92%, Partial 5%, Failed 3% Time / run58sCost / run$0.20 |
| 04 | gpt-5.4-mini (high)· agent-device v0.17.6 | agent-devicev0.17.6 | 85% | 1m 29s | $0.10 | Passed 77%, Partial 17%, Failed 7% Time / run1m 29sCost / run$0.10 |
| 05 | haiku-4.5 (low)· agent-device v0.17.6 | agent-devicev0.17.6 | 84% | 1m 53s | $0.13 | Passed 80%, Partial 8%, Failed 12% Time / run1m 53sCost / run$0.13 |
| 06 | haiku-4.5 (high)· agent-device v0.17.6 | agent-devicev0.17.6 | 83% | 1m 25s | $0.15 | Passed 75%, Partial 17%, Failed 8% Time / run1m 25sCost / run$0.15 |
| 07 | gpt-5.4-mini (low)· agent-device v0.17.6 | agent-devicev0.17.6 | 82% | 55s | $0.06 | Passed 73%, Partial 18%, Failed 8% Time / run55sCost / run$0.06 |
| 08 | gpt-5.4-mini (low)· argent v0.15.0 | argentv0.15.0 | 82% | 1m 17s | $0.14 | Passed 78%, Partial 8%, Failed 13% Time / run1m 17sCost / run$0.14 |
| 09 | gpt-5.4-mini (high)· no tool | no tool | 46% | 8m 32s | $0.62 | Passed 42%, Partial 8%, Failed 50% Time / run8m 32sCost / run$0.62 |
| 10 | gpt-5.4-mini (low)· no tool | no tool | 16% | 5m 13s | $0.10 | Passed 8%, Partial 15%, Failed 77% Time / run5m 13sCost / run$0.10 |
Tool comparison
Completion score by model and tool, partial outcomes at half credit. Every judged run counts, failures included; judge errors and stale reruns do not.Completion score by model and tool, partial outcomes at half credit. Every judged run counts, failures included; judge errors and stale reruns do not.
Average completion score
84%n=240
- low82%n=60
- high85%n=60
- low84%n=60
- high83%n=60
| model | argentv0.15.0 | agent-devicev0.17.6 | no tool |
|---|---|---|---|
| gpt-5.4-mini | 89%n=120 | 84%n=120 | 31%n=120 |
| low | 82%n=60 | 82%n=60 | 16%n=60 |
| high | 95%n=60 | 85%n=60 | 46%n=60 |
| haiku-4.5 | 96%n=120 | 84%n=120 | 13%n=120 |
| low | 94%n=60 | 84%n=60 | 15%n=60 |
| high | 98%n=60 | 83%n=60 | 12%n=60 |
| overall | 92%n=240 | 84%n=240 | 22%n=240 |
Breakdowns
Average time uses successful runs only - failures, partials and timeouts are excluded. Average cost uses the same runs as the completion score, failures included, skipping runs with no price data.Average time uses successful runs only - failures, partials and timeouts are excluded. Average cost uses the same runs as the completion score, failures included, skipping runs with no price data.
Cost by model
Cost by tool
Time by model
Time by tool
Cost efficiency
Average price per task vs completion score, both over every judged run - failures included, so a model that fails cheaply is not flattered. Colour identifies the model, shape the tool; higher and cheaper is better.Average price per task vs completion score, both over every judged run - failures included, so a model that fails cheaply is not flattered. Colour identifies the model, shape the tool; higher and cheaper is better.
Model:
Tool:
Methodology
1ArrangeA fresh simulator clone and re-seeded app state; the agent gets only the task text and its tool
2ActIt drives the device until it decides it is done or the clock runs out
3CaptureThe final screenshot
4JudgeA vision model matches that against the judge screen description
The judge is never told which model produced a screenshot, though the action names reveal which tool drove the device.