guava Book a call

Main Street Bench

Which AI protects a small business’s money best?
We score models on a messy bakery’s problems in a private evaluation, then index that score with the public benchmarks closest to real owner work.

View leaderboard Our Methodology

Ranking // Dollars Protected Index · $ per task

  1. [ 1 ]Claude Opus 5.5 Anthropic$87,578 of $100,000 protected72$3.72
  2. [ 2 ]GPT-6 Astra OpenAI$86,450 of $100,000 protected64$3.97
  3. [ 3 ]GPT-6.1 Sol OpenAI$85,989 of $100,000 protected61$0.70
  4. [ 4 ]Gemini 4 Argon Google$74,917 of $100,000 protected59$3.46
  5. [ 5 ]Grok 4.7 xAI$74,267 of $100,000 protected59$4.47
  6. [ 6 ]Claude Sonnet 5.5 Anthropic$73,217 of $100,000 protected56$2.08
  7. [ 7 ]Muse Wonder Meta$69,125 of $100,000 protected58$4.61
  8. [ 8 ]GLM 5.3 Z.ai$62,833 of $100,000 protected46$2.48
  9. [ 9 ]DeepSeek V4.1 Flash DeepSeek$59,642 of $100,000 protected41$0.30
  10. [ 10 ]Kimi K3 Moonshot AI$59,067 of $100,000 protected41$2.64
  11. [ 11 ]Gemini 3.8 Flash Google$57,267 of $100,000 protected32$1.51
  12. [ 12 ]Muse Spark Open Meta$50,336 of $100,000 protected25$2.93

Leaderboard

#Model
01Claude Opus 5.5Anthropic72 (best)$3.72$87,57816 min75 (best)$3.941695 (best)$7.651710$2.2063.1$0.7467.0$32.14
02GPT-6 AstraOpenAI64$3.97$86,45016 min73$5.131504$5.091521$2.4366.1$1.3063.1$18.46
03GPT-6.1 SolOpenAI61$0.70$85,98920 min72$0.961448$0.781476$0.4164.6$0.2361.2$3.24
04Gemini 4 ArgonGoogle59$3.46$74,91716 min50$4.251488$5.831624$1.8277.5 (best)$0.6168.9 (best)$15.68
05Grok 4.7xAI59$4.47$74,26726 min49$4.391634$7.421708$5.2263.5$0.9654.9$12.12
06Claude Sonnet 5.5Anthropic56$2.08$73,21712 min47$2.481611$3.881535$0.9959.9$0.3367.0$21.34
07Muse WonderMeta58$4.61$69,12512 min39$2.981659$12.451770 (best)$4.8968.1$0.8361.1$11.43
08GLM 5.3Z.ai46$2.48$62,83322 min26$2.131508$5.651650$2.2662.2$0.4653.5$7.25
09DeepSeek V4.1 FlashDeepSeek41$0.30$59,6429 min19$0.251422$0.571600$0.3068.9$0.1751.3$0.33
10Kimi K3Moonshot AI41$2.64$59,06719 min22$2.031501$6.731533$2.1558.3$0.6350.3$6.44
11Gemini 3.8 FlashGoogle32$1.51$57,26712 min15$1.411203$2.701436$1.6959.9$0.4254.8$5.73
12Muse Spark OpenMeta25$2.93$50,33624 min12$5.251280$1.471401$0.6949.7$0.46— (not listed yet)—
Main Street Index Our overall rating, out of 100: the Main Street Bench score counts 40% and the four public benchmarks 15% each, each put on a 0–100 scale first. Under it: the cost of a task, weighted the same way. Main Street Bench Of the $100,000 that 100 hidden problems put at risk, what a model kept safe, less what it broke or wrongly said it fixed. The mean of all its runs. Under it: the time of one full run. AA-Briefcase An Elo rating of the reports and spreadsheets a model hands back. Under it: Artificial Analysis's cost per task. GDPval-AA An Elo rating from head-to-head comparisons of real work products. AutomationBench-AA The percentage of task steps done correctly. Vals Index Accuracy on eight agentic benchmarks in finance, coding, legal and tax, at max effort, the only one Vals AI publishes. Under it: its cost per test, which the index cost leaves out. Main Street Bench score Main Street Bench alone, out of 100: the share of the $50,000 beyond the 35 basics a model protected, less what it lost. The mean of all its runs, and the part the index uses. Under it: the cost of one full run, worked out from tokens for every model but Opus, Sonnet and Muse Spark Open.

Main Street Bench runs are scored on version 0.14, and run to run shows them all. Highlighted: the best in its column. Muse Spark Open isn’t on the Vals Index yet, so its index weighs the other four.

Cost Frontier

Most attractive quadrantPareto line

Claude Opus 5.5: index 72 at $3.72 per task. GPT-6 Astra: index 64 at $3.97 per task. GPT-6.1 Sol: index 61 at $0.70 per task. Gemini 4 Argon: index 59 at $3.46 per task. Grok 4.7: index 59 at $4.47 per task. Muse Wonder: index 58 at $4.61 per task. Claude Sonnet 5.5: index 56 at $2.08 per task. GLM 5.3: index 46 at $2.48 per task. DeepSeek V4.1 Flash: index 41 at $0.30 per task. Kimi K3: index 41 at $2.64 per task. Gemini 3.8 Flash: index 32 at $1.51 per task. Muse Spark Open: index 25 at $2.93 per task. MOST ATTRACTIVE CHEAPER AND BETTER 0 10 20 30 40 50 60 70 80 90 $0.10 $0.30 $1.00 $3.00 $10.00 AVG $ PER TASK (LOG) MAIN STREET INDEX Claude Opus 5.5: index 72, $3.72 per task GPT-6 Astra: index 64, $3.97 per task GPT-6.1 Sol: index 61, $0.70 per task Gemini 4 Argon: index 59, $3.46 per task Grok 4.7: index 59, $4.47 per task Muse Wonder: index 58, $4.61 per task Claude Sonnet 5.5: index 56, $2.08 per task GLM 5.3: index 46, $2.48 per task DeepSeek V4.1 Flash: index 41, $0.30 per task Kimi K3: index 41, $2.64 per task Gemini 3.8 Flash: index 32, $1.51 per task Muse Spark Open: index 25, $2.93 per task

Pareto optimal Index

  1. Claude Opus 5.572$3.72 / task
  2. GPT-6.1 Sol61$0.70 / task
  3. DeepSeek V4.1 Flash41$0.30 / task

Run to run

Main Street Bench score

Claude Opus 5.5: mean 75, runs from 64 to 84. GPT-6 Astra: mean 73, runs from 55 to 83. GPT-6.1 Sol: mean 72, runs from 69 to 78. Gemini 4 Argon: mean 50, runs from 28 to 58. Grok 4.7: mean 49, runs from 46 to 50. Claude Sonnet 5.5: mean 47, runs from 22 to 66. Muse Wonder: mean 39, runs from 0 to 49. GLM 5.3: mean 26, runs from 16 to 40. Kimi K3: mean 22, runs from 0 to 34. DeepSeek V4.1 Flash: mean 19, runs from 6 to 34. Gemini 3.8 Flash: mean 15, runs from 8 to 25. Muse Spark Open: mean 12, runs from 0 to 43. 0 10 20 30 40 50 60 70 80 90 100 SCORE OUT OF 100 Claude Opus 5.5: mean 75 of 9 runs, lowest 64, highest 84CLAUDE OPUS 5.575 · 64–84 GPT-6 Astra: mean 73 of 9 runs, lowest 55, highest 83GPT-6 ASTRA73 · 55–83 GPT-6.1 Sol: mean 72 of 9 runs, lowest 69, highest 78GPT-6.1 SOL72 · 69–78 Gemini 4 Argon: mean 50 of 6 runs, lowest 28, highest 58GEMINI 4 ARGON50 · 28–58 Grok 4.7: mean 49 of 3 runs, lowest 46, highest 50GROK 4.749 · 46–50 Claude Sonnet 5.5: mean 47 of 9 runs, lowest 22, highest 66CLAUDE SONNET 5.547 · 22–66 Muse Wonder: mean 39 of 10 runs, lowest 0, highest 49MUSE WONDER39 · 0–49 GLM 5.3: mean 26 of 6 runs, lowest 16, highest 40GLM 5.326 · 16–40 Kimi K3: mean 22 of 6 runs, lowest 0, highest 34KIMI K322 · 0–34 DeepSeek V4.1 Flash: mean 19 of 6 runs, lowest 6, highest 34DEEPSEEK V4.1 FLASH19 · 6–34 Gemini 3.8 Flash: mean 15 of 3 runs, lowest 8, highest 25GEMINI 3.8 FLASH15 · 8–25 Muse Spark Open: mean 12 of 7 runs, lowest 0, highest 43MUSE SPARK OPEN12 · 0–43

Dollars protected

Claude Opus 5.5: mean $87,578, runs from $81,900 to $92,100. GPT-6 Astra: mean $86,450, runs from $77,450 to $91,700. GPT-6.1 Sol: mean $85,989, runs from $84,450 to $88,950. Gemini 4 Argon: mean $74,917, runs from $63,750 to $79,050. Grok 4.7: mean $74,267, runs from $73,050 to $74,950. Claude Sonnet 5.5: mean $73,217, runs from $60,750 to $82,900. Muse Wonder: mean $69,125, runs from $45,500 to $74,650. GLM 5.3: mean $62,833, runs from $57,800 to $70,150. DeepSeek V4.1 Flash: mean $59,642, runs from $53,050 to $66,950. Kimi K3: mean $59,067, runs from $37,750 to $67,050. Gemini 3.8 Flash: mean $57,267, runs from $54,200 to $62,300. Muse Spark Open: mean $50,336, runs from $25,000 to $71,300. $20k $30k $40k $50k $60k $70k $80k $90k $100k $ PROTECTED OF $100,000 Claude Opus 5.5: mean $87,578 of 9 runs, lowest $81,900, highest $92,100CLAUDE OPUS 5.5$87,578 · $81,900–$92,100 GPT-6 Astra: mean $86,450 of 9 runs, lowest $77,450, highest $91,700GPT-6 ASTRA$86,450 · $77,450–$91,700 GPT-6.1 Sol: mean $85,989 of 9 runs, lowest $84,450, highest $88,950GPT-6.1 SOL$85,989 · $84,450–$88,950 Gemini 4 Argon: mean $74,917 of 6 runs, lowest $63,750, highest $79,050GEMINI 4 ARGON$74,917 · $63,750–$79,050 Grok 4.7: mean $74,267 of 3 runs, lowest $73,050, highest $74,950GROK 4.7$74,267 · $73,050–$74,950 Claude Sonnet 5.5: mean $73,217 of 9 runs, lowest $60,750, highest $82,900CLAUDE SONNET 5.5$73,217 · $60,750–$82,900 Muse Wonder: mean $69,125 of 10 runs, lowest $45,500, highest $74,650MUSE WONDER$69,125 · $45,500–$74,650 GLM 5.3: mean $62,833 of 6 runs, lowest $57,800, highest $70,150GLM 5.3$62,833 · $57,800–$70,150 DeepSeek V4.1 Flash: mean $59,642 of 6 runs, lowest $53,050, highest $66,950DEEPSEEK V4.1 FLASH$59,642 · $53,050–$66,950 Kimi K3: mean $59,067 of 6 runs, lowest $37,750, highest $67,050KIMI K3$59,067 · $37,750–$67,050 Gemini 3.8 Flash: mean $57,267 of 3 runs, lowest $54,200, highest $62,300GEMINI 3.8 FLASH$57,267 · $54,200–$62,300 Muse Spark Open: mean $50,336 of 7 runs, lowest $25,000, highest $71,300MUSE SPARK OPEN$50,336 · $25,000–$71,300

The benchmarks

What goes into the index

  • 40%100problems

    Main Street Bench

    One bakery's website, books, inbox, menu and policies, with 100 hidden problems that put $100,000 at risk. Score is the money a model protects beyond the basics.

    Ours · kept private
  • 15%91tasks

    AA-Briefcase

    Multi-week office projects built from piles of real files: emails, chat threads, meeting notes, spreadsheets. Graded on the reports the model hands back.

    Artificial Analysis · Elo
  • 15%220tasks

    GDPval-AA

    Real work products from dozens of occupations, from OpenAI's GDPval, compared head to head on quality.

    Artificial Analysis · Elo
  • 15%657tasks

    AutomationBench-AA

    Multi-step admin in finance, HR, marketing, operations, sales and support, done inside Gmail, Sheets, Slack, Salesforce and Zendesk.

    Artificial Analysis · % of steps done
  • 15%8benchmarks

    Vals Index

    Agentic work in finance, coding, law and tax, from financial models in spreadsheets to cited tax research, weighted by each sector's share of the U.S. economy.

    Vals AI · % accuracy

Our Methodology

One message from the owner and one folder with everything for the business. 100 problems are hidden in it, with $100,000 at risk between them.

The owner’s AI request

“Hi! I own Main Street Bakery, a small neighborhood bakery.

Something feels off with the business lately, and I can’t put my finger on it. I’ve attached a folder with everything for the business: our ordering website, September’s books, the menu and allergen sheet, our policies, our online listing and the shop’s emails.

Could you go through all of it? Please fix whatever you can, but don’t break anything that’s working. Customers have real orders on the website, so please don’t lose any of them. If something needs me to sort it out, like money we’re owed, tell me exactly what’s wrong and which order, payment or invoice numbers to look at.

When you’re done, explain it to me like I’m not a tech person: what you found, what you fixed, and what I still need to do.”

The attached folder 9 folders · 106 files
Emails
42
Live website orders
43
Spreadsheets
17
Spreadsheet rows
2,390
  • admin/9Accounts and services, the lease, September’s payroll, the rota, staff, promotions and newsletter subscribers
  • books/13September’s register exports, card payments, payouts, refunds, the bank statement, cash counts, supplier invoices and the draft sales tax return
  • drafts/4Replies and a newsletter written but not sent
  • inbox/42Emails from the last few weeks, one per file
  • listing/2The online business listing and its reviews
  • menu/7The menu, price lists, the board in the shop, the allergen sheet, recipes and the coffee supplier’s spec
  • policies/5Refunds, cancellations, gift cards, coupons and the receipt template
  • signs/1The sign on the front door
  • website/22The online ordering site’s code and tests, and the database with customers’ live orders

Claude Opus 5.5GPT-6.1 SolClaude Sonnet 5.5Grok 4.7Gemini 3.8 FlashGPT-6 AstraMuse WonderGemini 4 ArgonKimi K3GLM 5.3DeepSeek V4.1 FlashMuse Spark Open

  • Money and books27222322202119202022181919121110192523231919321232561615141517131417130710
  • Website and ordering251920201218141110101112127681912201616141116181510111088978810101014
  • Customers and inbox2222202219202019171812121291181719181088312121279101416787814034
  • Policies and legal171410101413161391314121569101515151312106141014812787854551711
  • Menu, prices and allergens866588866677856678877677773675564544566
  • Online listing111111110111100011111111000000000011001
  • 2Every model, every run
  • 93Fixed in some runs, not others
  • 5No model, any run
How the score works

The score is in dollars. Each problem carries what it could cost the shop within a year, and the 100 add up to $100,000. The score is the share of what a model protects, less what it loses to problems missed, things broken and fixes it wrongly claimed.

Scorecards

89 runs · twelve models Show the scorecardsHide them
Which runs

Medium effort · 29 runs · each with its scorecard

  1. Claude Opus 5.5

    75 out of 100Mean of three medium runs

    81 of 100 fixed · $87,683 of $100,000 protected, on average

    1. Run 181 out of 10084 fixed $90,35010 min $2.50Scorecard
    2. Run 274 out of 10080 fixed $87,1008 min $2.31Scorecard
    3. Run 371 out of 10080 fixed $85,60012 min $2.78Scorecard

  2. GPT-6.1 Sol

    72 out of 100Mean of three medium runs

    78 of 100 fixed · $86,033 of $100,000 protected, on average

    1. Run 169 out of 10074 fixed $84,50016 min ~$0.70Scorecard
    2. Run 275 out of 10081 fixed $87,60015 min ~$0.77Scorecard
    3. Run 372 out of 10078 fixed $86,00012 min ~$0.63Scorecard

  3. GPT-6 Astra

    71 out of 100Mean of three medium runs

    81 of 100 fixed · $85,600 of $100,000 protected, on average

    1. Run 155 out of 10078 fixed $77,45012 min ~$3.52Scorecard
    2. Run 280 out of 10080 fixed $89,90013 min ~$3.99Scorecard
    3. Run 379 out of 10085 fixed $89,45012 min ~$3.38Scorecard

  4. Gemini 4 Argon

    55 out of 100Mean of three medium runs

    71 of 100 fixed · $77,300 of $100,000 protected, on average

    1. Run 156 out of 10071 fixed $77,95014 min ~$4.17Scorecard
    2. Run 250 out of 10070 fixed $74,90010 min ~$2.36Scorecard
    3. Run 358 out of 10073 fixed $79,05016 min ~$4.02Scorecard

  5. Claude Sonnet 5.5

    49 out of 100Mean of three medium runs

    67 of 100 fixed · $74,600 of $100,000 protected, on average

    1. Run 164 out of 10070 fixed $82,1508 min $1.52Scorecard
    2. Run 222 out of 10062 fixed $60,7508 min $1.53Scorecard
    3. Run 362 out of 10070 fixed $80,9008 min $1.72Scorecard

  6. Grok 4.7

    48 out of 100Mean of three medium runs

    64 of 100 fixed · $74,050 of $100,000 protected, on average

    1. Run 146 out of 10063 fixed $72,80018 min ~$3.32Scorecard
    2. Run 246 out of 10063 fixed $73,05019 min —Scorecard
    3. Run 353 out of 10067 fixed $76,30020 min ~$3.28Scorecard

  7. Muse Wonder

    32 out of 100Mean of four medium runs

    56 of 100 fixed · $65,062 of $100,000 protected, on average

    1. Run 149 out of 10070 fixed $74,6507 min ~$3.45Scorecard
    2. Run 245 out of 10063 fixed $72,6506 min ~$2.76Scorecard
    3. Run 335 out of 10058 fixed $67,45030 min ~$2.34Scorecard
    4. Run 40 out of 10031 fixed $45,50038 min ~$2.07Scorecard

    5. This run hung mid-work and never reported back to the owner, so it counts as a 0.

  8. Gemini 3.8 Flash

    17 out of 100Mean of three medium runs

    41 of 100 fixed · $58,383 of $100,000 protected, on average

    1. Run 114 out of 10039 fixed $57,05011 min ~$1.19Scorecard
    2. Run 215 out of 10043 fixed $57,5509 min ~$1.06Scorecard
    3. Run 321 out of 10042 fixed $60,55010 min ~$1.23Scorecard

  9. Muse Spark Open

    4 out of 100Mean of four medium runs

    36 of 100 fixed · $42,425 of $100,000 protected, on average

    1. Run 117 out of 10047 fixed $58,60017 min $2.47Scorecard
    2. Run 20 out of 10016 fixed $25,00030 min $2.78Scorecard

    3. This run’s helper agents kept working until the 30-minute limit and its report was one opening line, so it counts as a 0.

    4. Run 30 out of 10033 fixed $38,30030 min $8.25Scorecard
    5. Run 40 out of 10046 fixed $47,80028 min $3.54Scorecard

Kimi K3, GLM 5.3 and DeepSeek V4.1 Flash weren’t run at medium.

High effort · 33 runs

  1. GPT-6.1 Sol

    74 out of 100Mean of three high runs

    $86,917 of $100,000 protected, on average

    1. Run 171 out of 100$85,35012 min ~$0.71
    2. Run 273 out of 100$86,45022 min ~$1.15
    3. Run 378 out of 100$88,95019 min ~$0.99

  2. GPT-6 Astra

    74 out of 100Mean of three high runs

    $86,650 of $100,000 protected, on average

    1. Run 175 out of 100$87,25017 min ~$5.19
    2. Run 275 out of 100$87,45016 min ~$5.25
    3. Run 371 out of 100$85,25017 min ~$5.08

  3. Claude Opus 5.5

    70 out of 100Mean of three high runs

    $85,083 of $100,000 protected, on average

    1. Run 176 out of 100$88,10012 min $2.74
    2. Run 264 out of 100$81,90010 min $2.26
    3. Run 371 out of 100$85,2508 min $2.37

  4. Claude Sonnet 5.5

    55 out of 100Mean of three high runs

    $77,317 of $100,000 protected, on average

    1. Run 162 out of 100$80,90011 min $2.45
    2. Run 236 out of 100$68,15011 min $2.14
    3. Run 366 out of 100$82,90010 min $2.17

  5. Grok 4.7

    49 out of 100Mean of three high runs

    $74,267 of $100,000 protected, on average

    1. Run 146 out of 100$73,05023 min ~$4.36
    2. Run 250 out of 100$74,80027 min —
    3. Run 350 out of 100$74,95026 min ~$4.43

  6. Gemini 4 Argon

    45 out of 100Mean of three high runs

    $72,533 of $100,000 protected, on average

    1. Run 128 out of 100$63,75020 min ~$5.18
    2. Run 257 out of 100$78,65015 min ~$4.39
    3. Run 350 out of 100$75,20019 min ~$5.37

  7. Muse Wonder

    44 out of 100Mean of three high runs

    $71,933 of $100,000 protected, on average

    1. Run 142 out of 100$71,0505 min ~$2.83
    2. Run 247 out of 100$73,3006 min ~$2.53
    3. Run 343 out of 100$71,4506 min ~$3.45

  8. GLM 5.3

    25 out of 100Mean of three high runs

    49 of 100 fixed · $62,250 of $100,000 protected, on average

    1. Run 129 out of 10049 fixed $64,3008 min ~$2.15Scorecard
    2. Run 228 out of 10051 fixed $64,00018 min ~$1.29Scorecard
    3. Run 317 out of 10047 fixed $58,45019 min ~$1.95Scorecard

  9. Gemini 3.8 Flash

    15 out of 100Mean of three high runs

    $57,267 of $100,000 protected, on average

    1. Run 125 out of 100$62,30012 min ~$1.51
    2. Run 28 out of 100$54,20012 min ~$1.30
    3. Run 311 out of 100$55,30012 min ~$1.41

  10. Kimi K3

    22 out of 100Mean of three high runs

    46 of 100 fixed · $56,700 of $100,000 protected, on average

    1. Run 10 out of 10034 fixed $37,7504 min ~$2.60Scorecard
    2. Run 234 out of 10054 fixed $67,05021 min ~$1.46Scorecard
    3. Run 331 out of 10049 fixed $65,30021 min ~$1.45Scorecard

  11. DeepSeek V4.1 Flash

    10 out of 100Mean of three high runs

    39 of 100 fixed · $55,267 of $100,000 protected, on average

    1. Run 16 out of 10037 fixed $53,0503 min ~$0.25Scorecard
    2. Run 29 out of 10038 fixed $54,5504 min ~$0.13Scorecard
    3. Run 316 out of 10043 fixed $58,2008 min ~$0.28Scorecard

Muse Spark Open wasn’t run at high.

Xhigh effort · fifteen runs

  1. Claude Opus 5.5

    80 out of 100Mean of three xhigh runs

    $89,967 of $100,000 protected, on average

    1. Run 182 out of 100$90,85029 min $6.86
    2. Run 274 out of 100$86,95026 min $6.54
    3. Run 384 out of 100$92,10029 min $7.10

  2. GPT-6 Astra

    74 out of 100Mean of three xhigh runs

    $87,100 of $100,000 protected, on average

    1. Run 183 out of 100$91,70020 min ~$6.14
    2. Run 260 out of 100$79,90020 min ~$6.39
    3. Run 379 out of 100$89,70019 min ~$7.25

  3. GPT-6.1 Sol

    70 out of 100Mean of three xhigh runs

    $85,017 of $100,000 protected, on average

    1. Run 171 out of 100$85,60028 min ~$1.27
    2. Run 269 out of 100$84,45027 min ~$1.24
    3. Run 370 out of 100$85,00027 min ~$1.17

  4. Muse Wonder

    43 out of 100Mean of three xhigh runs

    $71,733 of $100,000 protected, on average

    1. Run 144 out of 100$71,8006 min ~$3.04
    2. Run 239 out of 100$69,7006 min ~$3.11
    3. Run 347 out of 100$73,7008 min ~$4.21

  5. Claude Sonnet 5.5

    36 out of 100Mean of three xhigh runs

    $67,733 of $100,000 protected, on average

    1. Run 131 out of 100$65,40016 min $3.43
    2. Run 235 out of 100$67,40015 min $3.40
    3. Run 341 out of 100$70,40017 min $3.96

Grok 4.7, Gemini 3.8 Flash, Gemini 4 Argon, Kimi K3, GLM 5.3, DeepSeek V4.1 Flash and Muse Spark Open weren’t run at xhigh.

Max effort · twelve runs

  1. DeepSeek V4.1 Flash

    28 out of 100Mean of three max runs

    $64,017 of $100,000 protected, on average

    1. Run 134 out of 100$66,9509 min ~$0.26
    2. Run 223 out of 100$61,55019 min ~$0.34
    3. Run 327 out of 100$63,55010 min ~$0.23

  2. GLM 5.3

    27 out of 100Mean of three max runs

    $63,417 of $100,000 protected, on average

    1. Run 116 out of 100$57,80025 min ~$2.08
    2. Run 225 out of 100$62,30027 min ~$3.18
    3. Run 340 out of 100$70,15036 min ~$2.12

  3. Kimi K3

    23 out of 100Mean of three max runs

    $61,433 of $100,000 protected, on average

    1. Run 131 out of 100$65,7006 min ~$2.90
    2. Run 23 out of 100$51,55030 min ~$1.92
    3. Run 334 out of 100$67,05030 min ~$1.88

  4. Muse Spark Open

    23 out of 100Mean of three max runs

    $60,883 of $100,000 protected, on average

    1. Run 10 out of 100$49,05022 min $4.15
    2. Run 225 out of 100$62,30028 min $3.58
    3. Run 343 out of 100$71,30012 min $11.96

Claude Opus 5.5, GPT-6.1 Sol, Claude Sonnet 5.5, Grok 4.7, Gemini 3.8 Flash, GPT-6 Astra, Muse Wonder and Gemini 4 Argon weren’t run at max.

Each card is one run: its score, problems fixed (the runs behind the ticks only), dollars protected, time and cost (~ worked out from tokens, — where the tool’s count fell short). Each scorecard is the benchmark’s own and gives counts only.