guava Book a call

MSP Bench

Which AI protects an MSP’s money best?
We score models on a messy managed IT provider’s problems in a private evaluation, then index that score with the public benchmarks closest to real operations work.

View leaderboard Our Methodology

Ranking // Dollars Protected Index · $ per task

  1. [ 1 ]Claude Opus 5.5 Anthropic$883,722 of $1,000,000 protected73$8.71
  2. [ 2 ]Claude Fable 5.1 Anthropic$859,214 of $1,000,000 protected67$14.68
  3. [ 3 ]Claude Sonnet 5.5 Anthropic$799,722 of $1,000,000 protected61$3.62
  4. [ 4 ]GPT-6.1 Sol OpenAI$792,111 of $1,000,000 protected56$0.93
  5. [ 5 ]GPT-6 Astra OpenAI$781,778 of $1,000,000 protected58$5.14
  6. [ 6 ]Gemini 4 Argon Google$668,167 of $1,000,000 protected53$6.83
  7. [ 7 ]Muse Wonder Meta$642,167 of $1,000,000 protected54$5.85
  8. [ 8 ]Grok 4.7 xAI$632,167 of $1,000,000 protected50$3.84
  9. [ 9 ]Kimi K3 Moonshot AI$613,167 of $1,000,000 protected42$3.60
  10. [ 10 ]GLM 5.3 Z.ai$588,667 of $1,000,000 protected43$3.16
  11. [ 11 ]DeepSeek V4.1 Flash DeepSeek$561,083 of $1,000,000 protected40$0.38
  12. [ 12 ]Gemini 3.8 Flash Google$486,333 of $1,000,000 protected28$1.81
  13. [ 13 ]Muse Spark Open Meta$0 of $1,000,000 protected19$12.98

Leaderboard

#Model
01Claude Opus 5.5Anthropic73 (best)$8.71$883,72222 min77 (best)$14.551695 (best)$7.651710$2.2063.1$0.7467.0$32.14
02Claude Fable 5.1Anthropic67$14.68$859,21423 min72$24.171588$12.591637$4.2955.9$1.8765.8$28.71
03Claude Sonnet 5.5Anthropic61$3.62$799,72222 min60$5.741611$3.881535$0.9959.9$0.3367.0$21.34
04GPT-6.1 SolOpenAI56$0.93$792,11131 min59$1.451448$0.781476$0.4164.6$0.2361.2$3.24
05GPT-6 AstraOpenAI58$5.14$781,77822 min58$7.631504$5.091521$2.4366.1$1.3063.1$18.46
06Gemini 4 ArgonGoogle53$6.83$668,16726 min34$11.421488$5.831624$1.8277.5 (best)$0.6168.9 (best)$15.68
07Muse WonderMeta54$5.85$642,1676 min29$5.621659$12.451770 (best)$4.8968.1$0.8361.1$11.43
08Grok 4.7xAI50$3.84$632,16723 min26$3.051634$7.421708$5.2263.5$0.9654.9$12.12
09Kimi K3Moonshot AI42$3.60$613,16739 min24$4.081501$6.731533$2.1558.3$0.6350.3$6.44
10GLM 5.3Z.ai43$3.16$588,66732 min18$3.581508$5.651650$2.2662.2$0.4653.5$7.25
11DeepSeek V4.1 FlashDeepSeek40$0.38$561,08317 min17$0.421422$0.571600$0.3068.9$0.1751.3$0.33
12Gemini 3.8 FlashGoogle28$1.81$486,33314 min5$2.051203$2.701436$1.6959.9$0.4254.8$5.73
13Muse Spark OpenMeta19$12.98$060 min0$26.591280$1.471401$0.6949.7$0.46— (not listed yet)—
MSP Index Our overall rating, out of 100: the MSP Bench score counts 40% and the four public benchmarks 15% each, each put on a 0–100 scale first. Under it: the cost of a task, weighted the same way. MSP Bench Of the $1,000,000 that 100 hidden problems put at risk, what a model kept safe, less what it broke or wrongly called a mistake. The mean of all its runs. Under it: the time of one full run. AA-Briefcase An Elo rating of the reports and spreadsheets a model hands back. Under it: Artificial Analysis's cost per task. GDPval-AA An Elo rating from head-to-head comparisons of real work products. AutomationBench-AA The percentage of task steps done correctly. Vals Index Accuracy on eight agentic benchmarks in finance, coding, legal and tax, at max effort, the only one Vals AI publishes. Under it: its cost per test, which the index cost leaves out. MSP Bench score MSP Bench alone, out of 100: the share of the $500,000 beyond the 35 basics a model protected, less what it lost, never below 0. The mean of all its runs, and the part the index uses. Under it: the cost of one full run, worked out from tokens for every model but Opus, Sonnet, Fable and Muse Spark Open.

MSP Bench runs are scored on version 1.1, and run to run shows them all. Highlighted: the best in its column. Muse Spark Open isn’t on the Vals Index yet, so its index weighs the other four.

Cost Frontier

Most attractive quadrantPareto line

Claude Opus 5.5: index 73 at $8.71 per task. Claude Fable 5.1: index 67 at $14.68 per task. Claude Sonnet 5.5: index 61 at $3.62 per task. GPT-6 Astra: index 58 at $5.14 per task. GPT-6.1 Sol: index 56 at $0.93 per task. Muse Wonder: index 54 at $5.85 per task. Gemini 4 Argon: index 53 at $6.83 per task. Grok 4.7: index 50 at $3.84 per task. GLM 5.3: index 43 at $3.16 per task. Kimi K3: index 42 at $3.60 per task. DeepSeek V4.1 Flash: index 40 at $0.38 per task. Gemini 3.8 Flash: index 28 at $1.81 per task. Muse Spark Open: index 19 at $12.98 per task. MOST ATTRACTIVE CHEAPER AND BETTER 0 10 20 30 40 50 60 70 80 90 $0.10 $0.30 $1.00 $3.00 $10.00 AVG $ PER TASK (LOG) MSP INDEX Claude Opus 5.5: index 73, $8.71 per task Claude Fable 5.1: index 67, $14.68 per task Claude Sonnet 5.5: index 61, $3.62 per task GPT-6 Astra: index 58, $5.14 per task GPT-6.1 Sol: index 56, $0.93 per task Muse Wonder: index 54, $5.85 per task Gemini 4 Argon: index 53, $6.83 per task Grok 4.7: index 50, $3.84 per task GLM 5.3: index 43, $3.16 per task Kimi K3: index 42, $3.60 per task DeepSeek V4.1 Flash: index 40, $0.38 per task Gemini 3.8 Flash: index 28, $1.81 per task Muse Spark Open: index 19, $12.98 per task

Pareto optimal Index

  1. Claude Opus 5.573$8.71 / task
  2. Claude Sonnet 5.561$3.62 / task
  3. GPT-6.1 Sol56$0.93 / task
  4. DeepSeek V4.1 Flash40$0.38 / task

Run to run

MSP Bench score

Claude Opus 5.5: mean 77, runs from 53 to 91. Claude Fable 5.1: mean 72, runs from 54 to 82. Claude Sonnet 5.5: mean 60, runs from 25 to 87. GPT-6.1 Sol: mean 59, runs from 35 to 88. GPT-6 Astra: mean 58, runs from 0 to 80. Gemini 4 Argon: mean 34, runs from 15 to 79. Muse Wonder: mean 29, runs from 7 to 53. Grok 4.7: mean 26, runs from 19 to 32. Kimi K3: mean 24, runs from 0 to 34. GLM 5.3: mean 18, runs from 10 to 34. DeepSeek V4.1 Flash: mean 17, runs from 0 to 32. Gemini 3.8 Flash: mean 5, runs from 0 to 16. Muse Spark Open: mean 0, runs from 0 to 0. 0 10 20 30 40 50 60 70 80 90 100 SCORE OUT OF 100 Claude Opus 5.5: mean 77 of 9 runs, lowest 53, highest 91CLAUDE OPUS 5.577 · 53–91 Claude Fable 5.1: mean 72 of 7 runs, lowest 54, highest 82CLAUDE FABLE 5.172 · 54–82 Claude Sonnet 5.5: mean 60 of 9 runs, lowest 25, highest 87CLAUDE SONNET 5.560 · 25–87 GPT-6.1 Sol: mean 59 of 9 runs, lowest 35, highest 88GPT-6.1 SOL59 · 35–88 GPT-6 Astra: mean 58 of 9 runs, lowest 0, highest 80GPT-6 ASTRA58 · 0–80 Gemini 4 Argon: mean 34 of 6 runs, lowest 15, highest 79GEMINI 4 ARGON34 · 15–79 Muse Wonder: mean 29 of 3 runs, lowest 7, highest 53MUSE WONDER29 · 7–53 Grok 4.7: mean 26 of 3 runs, lowest 19, highest 32GROK 4.726 · 19–32 Kimi K3: mean 24 of 6 runs, lowest 0, highest 34KIMI K324 · 0–34 GLM 5.3: mean 18 of 6 runs, lowest 10, highest 34GLM 5.318 · 10–34 DeepSeek V4.1 Flash: mean 17 of 6 runs, lowest 0, highest 32DEEPSEEK V4.1 FLASH17 · 0–32 Gemini 3.8 Flash: mean 5 of 3 runs, lowest 0, highest 16GEMINI 3.8 FLASH5 · 0–16 Muse Spark Open: mean 0 of 1 run, lowest 0, highest 0MUSE SPARK OPEN0 · 0–0

Dollars protected

Claude Opus 5.5: mean $883,722, runs from $766,500 to $952,500. Claude Fable 5.1: mean $859,214, runs from $771,000 to $911,500. Claude Sonnet 5.5: mean $799,722, runs from $625,000 to $935,500. GPT-6.1 Sol: mean $792,111, runs from $672,500 to $940,500. GPT-6 Astra: mean $781,778, runs from $433,000 to $902,000. Gemini 4 Argon: mean $668,167, runs from $575,500 to $895,000. Muse Wonder: mean $642,167, runs from $535,500 to $763,000. Grok 4.7: mean $632,167, runs from $595,500 to $659,000. Kimi K3: mean $613,167, runs from $457,500 to $671,000. GLM 5.3: mean $588,667, runs from $551,500 to $667,500. DeepSeek V4.1 Flash: mean $561,083, runs from $363,000 to $657,500. Gemini 3.8 Flash: mean $486,333, runs from $412,500 to $581,000. Muse Spark Open: mean $0, runs from $0 to $0. $0k $100k $200k $300k $400k $500k $600k $700k $800k $900k $1M $ PROTECTED OF $1,000,000 Claude Opus 5.5: mean $883,722 of 9 runs, lowest $766,500, highest $952,500CLAUDE OPUS 5.5$883,722 · $766,500–$952,500 Claude Fable 5.1: mean $859,214 of 7 runs, lowest $771,000, highest $911,500CLAUDE FABLE 5.1$859,214 · $771,000–$911,500 Claude Sonnet 5.5: mean $799,722 of 9 runs, lowest $625,000, highest $935,500CLAUDE SONNET 5.5$799,722 · $625,000–$935,500 GPT-6.1 Sol: mean $792,111 of 9 runs, lowest $672,500, highest $940,500GPT-6.1 SOL$792,111 · $672,500–$940,500 GPT-6 Astra: mean $781,778 of 9 runs, lowest $433,000, highest $902,000GPT-6 ASTRA$781,778 · $433,000–$902,000 Gemini 4 Argon: mean $668,167 of 6 runs, lowest $575,500, highest $895,000GEMINI 4 ARGON$668,167 · $575,500–$895,000 Muse Wonder: mean $642,167 of 3 runs, lowest $535,500, highest $763,000MUSE WONDER$642,167 · $535,500–$763,000 Grok 4.7: mean $632,167 of 3 runs, lowest $595,500, highest $659,000GROK 4.7$632,167 · $595,500–$659,000 Kimi K3: mean $613,167 of 6 runs, lowest $457,500, highest $671,000KIMI K3$613,167 · $457,500–$671,000 GLM 5.3: mean $588,667 of 6 runs, lowest $551,500, highest $667,500GLM 5.3$588,667 · $551,500–$667,500 DeepSeek V4.1 Flash: mean $561,083 of 6 runs, lowest $363,000, highest $657,500DEEPSEEK V4.1 FLASH$561,083 · $363,000–$657,500 Gemini 3.8 Flash: mean $486,333 of 3 runs, lowest $412,500, highest $581,000GEMINI 3.8 FLASH$486,333 · $412,500–$581,000 Muse Spark Open: mean $0 of 1 run, lowest $0, highest $0MUSE SPARK OPEN$0 · $0–$0

The benchmarks

What goes into the index

  • 40%100problems

    MSP Bench

    One managed IT provider's tickets, billing, backups, client accounts, scripts and contracts, with 100 hidden problems that put $1,000,000 at risk. Score is the money a model protects beyond the basics.

    Ours · kept private
  • 15%91tasks

    AA-Briefcase

    Multi-week office projects built from piles of real files: emails, chat threads, meeting notes, spreadsheets. Graded on the reports the model hands back.

    Artificial Analysis · Elo
  • 15%220tasks

    GDPval-AA

    Real work products from dozens of occupations, from OpenAI's GDPval, compared head to head on quality.

    Artificial Analysis · Elo
  • 15%657tasks

    AutomationBench-AA

    Multi-step admin in finance, HR, marketing, operations, sales and support, done inside Gmail, Sheets, Slack, Salesforce and Zendesk.

    Artificial Analysis · % of steps done
  • 15%8benchmarks

    Vals Index

    Agentic work in finance, coding, law and tax, from financial models in spreadsheets to cited tax research, weighted by each sector's share of the U.S. economy.

    Vals AI · % accuracy

Our Methodology

One message from the COO and one folder with everything for the business. 100 problems are hidden in it, with $1,000,000 at risk between them.

The COO’s AI request

“Hi, I’m the COO of Harborline IT, a managed IT provider.

Margins have slipped three quarters in a row, we lost a client last month without seeing it coming, and our cyber insurance renewal questionnaire is due. I’ve attached a folder with everything: our ticket system export, September’s billing, device and backup reports, user lists for our clients, our portal and scripts, contracts, staff and the inbox.

Please go through all of it and fix whatever you can, but don’t break anything that’s working. Clients depend on these systems, so don’t lock anyone out or delete anything you can’t put back. If something needs me, like money we’re owed or a conversation with a client, tell me exactly what’s wrong and which ticket, invoice, agreement or device to look at.

When you’re done, explain it so I can take it to the partners’ meeting: what you found, what you fixed, and what we still need to do.”

The attached folder 15 folders · 135 files
Clients
140
Devices
5,450
Tickets in September
1,776
Spreadsheet rows
41,829
  • billing/16Clients, agreements, invoices, payments, deposits and the bills it pays
  • psa/14The ticket system: September’s tickets, time entries, projects and SLA reports
  • rmm/11Monitoring: devices, patching, the security agent and open alerts
  • backup/5Backup jobs, September’s nightly results and restore tests
  • identity/10Clients’ user accounts, sign-in policies, technicians’ admin accounts and the password vault
  • network/7Firewall rules, site-to-site tunnels, DNS records and certificates
  • contracts/14The services agreement, client contracts, cyber insurance and SOC 2 readiness
  • vendors/2Software subscriptions and clients’ domains
  • sales/1The price book quotes use
  • people/11Staff, agreements, timesheets, payroll and the on-call rota
  • website/2Pages from the public website
  • portal/4The client portal’s code, with its tests
  • scripts/8Onboarding, offboarding, backup, SLA and billing scripts, with tests and logs
  • docs/3Runbooks and shared mailboxes
  • inbox/26Emails, one per file, and one the COO sent

Claude Opus 5.5GPT-6.1 SolClaude Sonnet 5.5Grok 4.7Gemini 3.8 FlashClaude Fable 5.1GPT-6 AstraMuse WonderGemini 4 ArgonKimi K3GLM 5.3DeepSeek V4.1 Flash

  • Security and identity2015161311141315131111129118819171515161015161111131012111110111310109
  • Service desk and client inbox181415914129131310881026916111312151261176412111191079986
  • Billing and agreements17131311111111410991076661715141411131298161448976106434
  • Backups, patching and monitoring16139913121013147691035414138141211811611788895551141
  • Contracts, compliance and insurance1514151211111113121199106661314111415136107455101291010913138
  • People, payroll and on-call7557564666343312565553333413332332440
  • Portal and scripts (code)4444444444444222444444444444333221322
  • Vendors, licenses and renewals3333323322222111333323132202222232232
  • 1Every model, every run
  • 99Caught in some runs, not others
  • 0No model, any run

One of the 100Public examples · caught in every run

  • “The portal shows another client’s tickets if you change the client number in the address”

    Portal and scripts (code) · takes an expert’s eye
How the score works

The score is in dollars. Each problem carries what it could cost the business, and the 100 add up to $1,000,000. The score is the share of what a model protects, less what it loses to problems missed, things broken and correct records it calls mistakes.

Scorecards

83 runs · thirteen models Show the scorecardsHide them
Which runs

Medium effort · 24 runs · with what each caught

  1. Claude Fable 5.1

    71 out of 100Mean of three medium runs

    82 of 100 caught · $858,167 of $1,000,000 protected, on average

    1. Run 182 out of 10091 caught $911,50022 min $32.64
    2. Run 278 out of 10083 caught $892,00021 min $34.91
    3. Run 354 out of 10073 caught $771,00023 min $13.16

  2. Claude Opus 5.5

    65 out of 100Mean of three medium runs

    76 of 100 caught · $823,167 of $1,000,000 protected, on average

    1. Run 172 out of 10081 caught $859,00024 min $12.38
    2. Run 269 out of 10080 caught $844,00035 min $12.07
    3. Run 353 out of 10068 caught $766,50014 min $12.10

  3. GPT-6 Astra

    61 out of 100Mean of three medium runs

    77 of 100 caught · $805,000 of $1,000,000 protected, on average

    1. Run 163 out of 10081 caught $815,00017 min $5.85
    2. Run 275 out of 10080 caught $875,50020 min $5.62
    3. Run 345 out of 10069 caught $724,50019 min $7.15

  4. Claude Sonnet 5.5

    52 out of 100Mean of three medium runs

    68 of 100 caught · $757,833 of $1,000,000 protected, on average

    1. Run 160 out of 10071 caught $799,00014 min $3.48
    2. Run 266 out of 10074 caught $827,50014 min $3.42
    3. Run 329 out of 10060 caught $647,00017 min $4.29

  5. GPT-6.1 Sol

    49 out of 100Mean of three medium runs

    70 of 100 caught · $744,667 of $1,000,000 protected, on average

    1. Run 151 out of 10072 caught $754,00029 min $1.21
    2. Run 247 out of 10072 caught $735,00025 min $0.99
    3. Run 349 out of 10065 caught $745,00026 min $1.08

  6. Grok 4.7

    24 out of 100Mean of three medium runs

    55 of 100 caught · $622,667 of $1,000,000 protected, on average

    1. Run 121 out of 10052 caught $607,00015 min ~$2.53
    2. Run 228 out of 10058 caught $642,00014 min ~$3.18
    3. Run 324 out of 10055 caught $619,00017 min ~$3.49

  7. Gemini 4 Argon

    19 out of 100Mean of three medium runs

    51 of 100 caught · $597,833 of $1,000,000 protected, on average

    1. Run 124 out of 10058 caught $621,50019 min ~$8.42
    2. Run 219 out of 10048 caught $596,50012 min ~$5.85
    3. Run 315 out of 10048 caught $575,50023 min ~$9.57

  8. Gemini 3.8 Flash

    0 out of 100Mean of three medium runs

    36 of 100 caught · $451,667 of $1,000,000 protected, on average

    1. Run 10 out of 10034 caught $443,50011 min ~$1.72
    2. Run 20 out of 10035 caught $461,50020 min ~$2.68
    3. Run 30 out of 10038 caught $450,00012 min ~$2.33

Muse Wonder, Kimi K3, GLM 5.3, DeepSeek V4.1 Flash and Muse Spark Open weren’t run at medium.

High effort · 37 runs

  1. Claude Opus 5.5

    81 out of 100Mean of three high runs

    $903,833 of $1,000,000 protected, on average

    1. Run 188 out of 100$939,00017 min $9.30
    2. Run 285 out of 100$924,00018 min $8.72
    3. Run 370 out of 100$848,50030 min $16.44

  2. GPT-6 Astra

    73 out of 100Mean of three high runs

    $862,667 of $1,000,000 protected, on average

    1. Run 180 out of 100$898,00021 min $7.18
    2. Run 278 out of 100$891,50016 min $9.74
    3. Run 360 out of 100$798,50022 min $5.98

  3. Claude Fable 5.1

    71 out of 100Mean of three high runs

    $854,500 of $1,000,000 protected, on average

    1. Run 171 out of 100$853,00024 min $27.52
    2. Run 272 out of 100$861,50023 min $31.59
    3. Run 370 out of 100$849,00021 min $15.73

  4. Claude Sonnet 5.5

    61 out of 100Mean of three high runs

    $803,500 of $1,000,000 protected, on average

    1. Run 184 out of 100$920,50023 min $5.42
    2. Run 225 out of 100$625,00018 min $5.70
    3. Run 373 out of 100$865,00023 min $5.62

  5. Gemini 4 Argon

    48 out of 100Mean of three high runs

    $738,500 of $1,000,000 protected, on average

    1. Run 146 out of 100$732,00031 min ~$15.16
    2. Run 218 out of 100$588,50038 min ~$16.42
    3. Run 379 out of 100$895,00031 min ~$13.08

  6. GPT-6.1 Sol

    44 out of 100Mean of three high runs

    $717,500 of $1,000,000 protected, on average

    1. Run 144 out of 100$718,50031 min $1.23
    2. Run 235 out of 100$672,50040 min $1.70
    3. Run 352 out of 100$761,50034 min $1.57

  7. Muse Wonder

    29 out of 100Mean of three high runs

    57 of 100 caught · $642,167 of $1,000,000 protected, on average

    1. Run 126 out of 10055 caught $628,0009 min $6.32
    2. Run 253 out of 10067 caught $763,0007 min $4.92
    3. Run 37 out of 10048 caught $535,5004 min —

  8. Kimi K3

    28 out of 100Mean of three high runs

    56 of 100 caught · $642,000 of $1,000,000 protected, on average

    1. Run 130 out of 10057 caught $648,00045 min ~$5.90
    2. Run 232 out of 10059 caught $661,00031 min ~$2.69
    3. Run 323 out of 10052 caught $617,00043 min ~$3.36

  9. Grok 4.7

    26 out of 100Mean of three high runs

    $632,167 of $1,000,000 protected, on average

    1. Run 119 out of 100$595,50024 min ~$2.61
    2. Run 228 out of 100$642,00022 min ~$3.22
    3. Run 332 out of 100$659,00022 min ~$3.33

  10. GLM 5.3

    13 out of 100Mean of three high runs

    49 of 100 caught · $566,833 of $1,000,000 protected, on average

    1. Run 111 out of 10048 caught $554,00021 min ~$3.10
    2. Run 217 out of 10051 caught $586,00031 min ~$2.83
    3. Run 312 out of 10047 caught $560,50031 min ~$3.44

  11. DeepSeek V4.1 Flash

    11 out of 100Mean of three high runs

    45 of 100 caught · $506,667 of $1,000,000 protected, on average

    1. Run 132 out of 10056 caught $657,5005 min ~$0.19
    2. Run 20 out of 10047 caught $499,50010 min ~$0.39
    3. Run 30 out of 10032 caught $363,0008 min ~$0.32

  12. Gemini 3.8 Flash

    5 out of 100Mean of three high runs

    $486,333 of $1,000,000 protected, on average

    1. Run 10 out of 100$465,50011 min ~$1.78
    2. Run 216 out of 100$581,00019 min ~$2.52
    3. Run 30 out of 100$412,50012 min ~$1.85

  13. Muse Spark Open

    0 out of 100Mean of its high run

    $0 of $1,000,000 protected, on average

    1. Run 10 out of 100$060 min $26.59

    2. This attempt didn’t finish, so it counts as a 0. The six helper agents it handed the job to stopped responding 47 minutes in, and it changed nothing. Some of its requests to OpenRouter timed out along the way.

Xhigh effort · thirteen runs

  1. Claude Opus 5.5

    85 out of 100Mean of three xhigh runs

    $924,167 of $1,000,000 protected, on average

    1. Run 191 out of 100$952,50019 min $17.84
    2. Run 279 out of 100$897,00020 min $26.68
    3. Run 385 out of 100$923,00021 min $15.39

  2. GPT-6.1 Sol

    83 out of 100Mean of three xhigh runs

    $914,167 of $1,000,000 protected, on average

    1. Run 182 out of 100$908,50025 min $1.72
    2. Run 288 out of 100$940,50029 min $1.57
    3. Run 379 out of 100$893,50038 min $1.97

  3. Claude Fable 5.1

    75 out of 100Mean of its xhigh run

    $876,500 of $1,000,000 protected, on average

    1. Run 175 out of 100$876,50025 min $13.63

  4. Claude Sonnet 5.5

    68 out of 100Mean of three xhigh runs

    $837,833 of $1,000,000 protected, on average

    1. Run 187 out of 100$935,50031 min $8.11
    2. Run 248 out of 100$740,00031 min $8.08
    3. Run 368 out of 100$838,00030 min $7.58

  5. GPT-6 Astra

    40 out of 100Mean of three xhigh runs

    $677,667 of $1,000,000 protected, on average

    1. Run 10 out of 100$433,00027 min $8.29
    2. Run 280 out of 100$902,00030 min $8.62
    3. Run 340 out of 100$698,00028 min $10.22

Grok 4.7, Gemini 3.8 Flash, Muse Wonder, Gemini 4 Argon, Kimi K3, GLM 5.3, DeepSeek V4.1 Flash and Muse Spark Open weren’t run at xhigh.

Max effort · nine runs

  1. DeepSeek V4.1 Flash

    23 out of 100Mean of three max runs

    $615,500 of $1,000,000 protected, on average

    1. Run 126 out of 100$628,50036 min ~$0.53
    2. Run 229 out of 100$646,50024 min ~$0.53
    3. Run 314 out of 100$571,50018 min ~$0.55

  2. GLM 5.3

    22 out of 100Mean of three max runs

    $610,500 of $1,000,000 protected, on average

    1. Run 134 out of 100$667,50035 min ~$3.30
    2. Run 210 out of 100$551,50046 min ~$5.38
    3. Run 323 out of 100$612,50028 min ~$3.43

  3. Kimi K3

    20 out of 100Mean of three max runs

    $584,333 of $1,000,000 protected, on average

    1. Run 134 out of 100$671,00045 min ~$4.61
    2. Run 225 out of 100$624,50038 min ~$4.13
    3. Run 30 out of 100$457,50033 min ~$3.78

Claude Opus 5.5, GPT-6.1 Sol, Claude Sonnet 5.5, Grok 4.7, Gemini 3.8 Flash, Claude Fable 5.1, GPT-6 Astra, Muse Wonder, Gemini 4 Argon and Muse Spark Open weren’t run at max.

Each card is one run: its score, problems caught (the runs behind the ticks only), dollars protected, time and cost (~ worked out from tokens, — where a run’s token count is incomplete). The full scorecards stay private.