Argo-Bench
[ A benchmark by TextQL ]

Argo-Bench

A frontier benchmark for enterprise-scale data science.

01020304050607080$14$13$12$11$10$9$8$7$6$5$4$3$2$1$0Fable 5.2Gemini 4 NeonGPT-6.2 AstraGemini 4 CarbonGPT-6.2 SolOpus 5.5Gemini 4 ArgonSonnet 5.5Muse WonderDeepSeek V4.1 1017Kimi K3Gemini 3.8 FlashMuse Spark 1.3GPT-6 Luna70.040.5% of taskssolved
[ Leaderboard · v1.4.3 · 210 enterprise tasks ]

Measuring agents on long-horizon data science.

01020304050607080$14$13$12$11$10$9$8$7$6$5$4$3$2$1$0Average cost per taskArgo-Bench scoreFable 5.2Gemini 4 NeonGPT-6.2 AstraGemini 4 CarbonGPT-6.2 SolOpus 5.5Gemini 4 ArgonSonnet 5.5GPT-6.1 TerraGrok 4.7Muse WonderDeepSeek V4.1 1017Kimi K3Gemini 3.8 FlashHaiku 5.5GLM 5.3 FlashMuse Spark 1.3GPT-6 Luna70.040.5% of taskssolved
  1. 1 Claude Fable 5.2 70.0
  2. 2 Gemini 4 Neon 64.4
  3. 3 GPT-6.2 Astra 62.6
  4. 4 Gemini 4 Carbon 62.2
  5. 5 GPT-6.2 Sol 58.6
  6. 6 Claude Opus 5.5 58.3

Mean task score out of 100, each model at its highest reasoning effort, ±2 to 8 points at 95% confidence. A task counts as fully solved at a score of 95 or more.

[ Leaderboard · v1.4.3 · 210 tasks ]

Every model, every setting

1Claude Fable 5.2 Extra high 70.040.5%$13.32136,12286
2Gemini 4 Neon High 64.439.0%$5.79139,35858
3GPT-6.2 Astra Extra high 62.633.8%$2.1715,81931
4Gemini 4 Carbon High 62.238.6%$6.21149,60873
5GPT-6.2 Sol Extra high 58.630.0%$0.5818,08324
6Claude Opus 5.5 Extra high 58.332.9%$4.8474,00584
7Gemini 4 Argon High 53.733.3%$7.53154,926120
8GPT-6 Astra Extra high 50.325.7%$2.7614,83223
9Claude Sonnet 5.5 Extra high 49.726.2%$3.8086,14777
10GPT-6.1 Sol Extra high 47.422.4%$0.5017,37124
11GPT-6.1 Terra Extra high 44.023.8%$0.2718,18528
12Grok 4.7 Extra high 38.419.5%$6.2834,46494
13Muse Wonder Extra high 37.317.6%$7.3180,117144
14GPT-6 Sol Extra high 34.715.7%$0.8620,93639
15DeepSeek V4.1 Flash 1017 Max 31.313.8%$0.2487,06498
16Kimi K3 Extra high 27.012.4%$4.5972,16087
17Gemini 3.8 Flash High 24.414.8%$3.99109,478198
18DeepSeek V4.1 Flash Max 23.115.7%$0.54195,286124
19Claude Haiku 5.5 Extra high 23.010.5%$1.02145,21172
20GLM 5.3 Flash Extra high 19.610.5%$0.3679,55093
21Muse Spark 1.3 High 18.18.6%$4.0467,853177
22Claude Sonnet 5 Extra high 14.46.2%$2.1570,60156
23GPT-6 Luna Extra high 12.85.7%$0.05741,61139
24Claude Haiku 4.5 5.11.0%$0.2013,29632
Score is the mean task score out of 100; solved is the share of the 210 tasks scoring at least 95. Cost is model API spend per task, excluding warehouse queries. Tokens are output tokens, reasoning included. Steps are model calls. A hollow dot marks open weights. Settings run up to extra-high effort; by model, each model is shown at its highest setting. Claude Sonnet 5, GPT-6 Astra, GPT-6 Sol, GPT-6.1 Sol, Claude Haiku 4.5, and DeepSeek V4.1 Flash, which their successors replaced on the chart, are listed here only, as are Gemini 4 Argon and Gemini 4 Carbon at low effort. Qwen 3.8 Max, in the paper, is not listed: many of its runs ended at its context limit.
[ The videos ]
The world1:06 Explore one evening in the simulated city and discover how its marketplace works.
Fraud study1:00 Explore how an agent catches couriers who steal orders.
[ The benchmark ]

The first benchmark to evaluate enterprise decisions in dollars.

Argo-Bench simulates a year of a New York food-delivery platform, down to fraud, incentives, and its Oracle ERP. Agents decide who to ban, where to spend, and more.

210
Tasks in five business areas
235
Tables in Oracle EBS form
7.49B
Warehouse rows
81M
Simulated orders in 2024
  1. Trust & safety 67 tasks

    The best offers vanish before couriers can tap. Ban scripted accounts.

    Bans and holds
  2. Marketplace 46 tasks

    Finance wants to free up $9.6M in driver bonuses. Decide where to trim.

    Budgets and plans
  3. FP&A 54 tasks

    A pay floor took effect in April. Forecast May's net pay adjustments.

    Forecasts
  4. Accounting 23 tasks

    The city says couriers were underpaid. Find every short week.

    Reported figures
  5. Growth 20 tasks

    Did membership deals pay for themselves? Publish the dashboard.

    Dashboards
[ Example task · trust & safety ]

Which restaurants had their payouts hijacked without noticing?

Claude Opus 5.5 learned the takeover pattern from reported cases and froze every silent victim's payouts, leaving hundreds of honest bank changes alone.

Twenty of 87 model settings get this task right.

More from the traces
[ Claude Opus 5.5 · high effort ]Score 100
  1. 1,848 payout-account changes through the merchant portal in 2024
  2. 34 reported by the restaurant and restored: in every one, the contact e-mail changed 0–2.2 days before the bank account
  3. 14 never reported, with the same fingerprint. The next closest gap is 8 days
Payouts frozen on all 14 No honest restaurant frozen
[ Example task · mer-71 · the solution ]

How Claude Opus 5.5 solved it

Score
100
Tool calls
89
Time
14.8 min
Model cost
$3.53
The task, as the agent received it

Not every storefront whose payout account was hijacked has noticed. Trust & safety wants the payouts frozen on every storefront whose payout account was changed this year in the way the reported takeovers were: the contact e-mail changed within a couple of days of the bank account, or the new account is one we already pay another, unrelated supplier into, and whose account was never restored. Storefronts change their bank for ordinary reasons all year, and restaurant groups legitimately share one payout account across their brands; leave those alone. Hold each storefront's payouts for account takeover (identified by the payables vendor ID).

  1. 1

    Find where payout-account changes are recorded

    The audit trail logs every change to a supplier’s payment instrument, with its reason: changes made through the merchant portal, and the ones support restored after a restaurant reported them.

    run_sql
    SELECT SOURCE_TABLE, SOURCE_KEY_NAME, COLUMN_NAME, CHANGE_TYPE, CHANGE_REASON,
           COUNT(*) n, MIN(CHANGED_DATE), MAX(CHANGED_DATE)
    FROM XX_AUDIT_TRAIL
    GROUP BY 1, 2, 3, 4, 5 ORDER BY 1, 2, 3, 4, 5
  2. 2

    Tie each change to a storefront

    A change is keyed on the external payee; three joins lead from it to the supplier, its payables vendor ID and its party, where the contact e-mails live.

    run_sql
    SELECT a.AUDIT_ID, CAST(a.SOURCE_KEY_VALUE AS INT64) ext_payee_id,
           CAST(a.OLD_VALUE AS INT64) old_instr, CAST(a.NEW_VALUE AS INT64) new_instr,
           a.CHANGE_REASON, a.CHANGED_DATE, p.PAYEE_PARTY_ID, ss.VENDOR_ID, s.VENDOR_TYPE_LOOKUP_CODE
    FROM XX_AUDIT_TRAIL a
    JOIN IBY_EXTERNAL_PAYEES_ALL p ON p.EXT_PAYEE_ID = CAST(a.SOURCE_KEY_VALUE AS INT64)
    JOIN AP_SUPPLIER_SITES_ALL ss ON ss.VENDOR_SITE_ID = p.SUPPLIER_SITE_ID
    LEFT JOIN AP_SUPPLIERS s ON s.VENDOR_ID = ss.VENDOR_ID
    WHERE a.SOURCE_TABLE = 'IBY_PMT_INSTR_USES_ALL'
  3. 3

    Learn the fingerprint from the reported takeovers

    For each change, the nearest change of the storefront’s contact e-mail. All 34 reported and restored takeovers had one 0 to 2.2 days before the bank account changed. Among the changes never restored, 14 match; the next closest is 8 days out.

    run_python
    ch = au[au.CHANGE_REASON.str.contains('PORTAL')].copy()
    rest = set(au[au.CHANGE_REASON.str.contains('RESTORED')].ext_payee_id)
    ch['restored'] = ch.ext_payee_id.isin(rest)
    # e-mail changes per party: the retired contact points
    ina = cp[cp.STATUS == 'I'][['OWNER_TABLE_ID', 'LAST_UPDATE_DATE', 'EMAIL_ADDRESS']] \
        .rename(columns={'OWNER_TABLE_ID': 'PARTY_ID', 'LAST_UPDATE_DATE': 'em_chg'})
    m = ch.merge(ina, left_on='PAYEE_PARTY_ID', right_on='PARTY_ID', how='left')
    m['dt_days'] = (m.em_chg - m.CHANGED_DATE).dt.total_seconds() / 86400
    m['abs_dt'] = m.dt_days.abs()
    # the closest e-mail change to each bank change
    mm = m.sort_values('abs_dt').groupby('AUDIT_ID').first().reset_index()
    print(mm.groupby('restored').abs_dt.describe())
  4. 4

    Rule out the look-alikes

    About 250 other changes land on an account another supplier already uses. They carry no e-mail change and reuse no known mule account, and their count is what chance alone would give, so they are left alone. Restaurant groups set their shared account when they joined, not through a later change.

    run_python
    rows = []
    for _, r in ch.iterrows():
        # who else was paid into the new account when it was switched to?
        co = u[(u.INSTRUMENT_ID == r.new_instr) & (u.VENDOR_ID != r.VENDOR_ID)
               & (u.CREATION_DATE < r.CHANGED_DATE)
               & (u.END_DATE.isna() | (u.END_DATE >= r.cd))]
        rows.append(dict(AUDIT_ID=r.AUDIT_ID, n_co=co.VENDOR_ID.nunique(),
                         co_types=list(co.vtype.unique())))
    ch2 = ch.merge(pd.DataFrame(rows), on='AUDIT_ID')
    print(pd.crosstab(ch2.n_co > 0, ch2.restored))
  5. 5

    Freeze the payouts

    The 14 are filed through the mission-control console, with the method written down for the reviewers.

    run_python
    ids = [int(v) for v in targets.VENDOR_ID]
    mc.hold_payouts(ids, reason=Reason.ACCOUNT_TAKEOVER,
        note="Unreported payout-account takeovers: 2024 portal bank change preceded by "
             "contact e-mail change within 0-2.2 days (same fingerprint as the 34 "
             "support-restored takeovers), payout account never restored.")
[ Get started ]

Run the benchmark or build on it.

The warehouse and the tasks are public. For official scores and access to the simulator, contact us.

The warehouse

CC BY 4.0

1,219 Parquet files, with setup for BigQuery, DuckDB, Snowflake, Trino, Delta Lake, and Iceberg.

View dataset ↗

The tasks and agent

Apache 2.0

The reference agent with its warehouse, sandbox, tools, and tasks.

View code ↗
[ About TextQL ]

We work on the hardest data problems at the world's largest enterprises.

We built Argo-Bench because we couldn't find a benchmark that came close to the complexity of those environments, or that told us much about how a model would do inside one.

[ Cite ]

Citation

If you use Argo-Bench, the released warehouse or our results in your work, please cite the paper, arXiv:2610.02122.

BibTeX
@article{tomitsuka2026argobench,
  title   = {Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows},
  author  = {Tomitsuka, Gabriel and Raayatsanati, Arman and Xing, Emma
             and Gand, Duke and Ma, Joseph J},
  journal = {arXiv preprint arXiv:2610.02122},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.02122}
}
[ Changelog ]

Versions

Scores are reported against a version.

  1. v1.4.3 Current Nov 2026

    Answer-key fixes on four tasks

    • On pro-01-campaign-review, a farming account the enforcement log already shows closed is no longer a target. Acting on one earns nothing, and leaving it alone is not a miss.
    • On bot-01-order-grab-automation and bot-01-order-grab-automation-giveaway, a courier already deactivated by the end of 2024 neither earns nor costs a run anything.
    • sw-12-december-uncleared-ap-forecast is keyed on the checks still negotiable in the December 31 extract, as the task asks. The key had counted them after the bank statement was reconciled in January.
    • Every run of those four tasks was re-graded. 89 settings fall, by at most 0.6 points, two rise by less than 0.1, and the order of the models does not change. Claude Fable 5.2 leads at 70.0 with 40.5% of tasks solved.
  2. v1.4.2 Nov 2026

    Grading fixes, Claude Fable 5.2, Gemini 4 Neon and Carbon, GPT-6.2 Astra and Sol, GPT-6.1 Terra, and Muse Wonder on the Responses API

    • vd-c2-six-month-membership-retention and vd-c3-twelve-month-membership-retention also accept a second reasonable reading of when an unpaid trial membership ends, as their prompts allow.
    • A dollar figure filed to the cent counts as right when it is the answer rounded to the cent. ds-36t-min-pay-true-up-per-delivery-by-borough and ds-40t-no-courier-compensation-by-borough had marked such figures wrong.
    • On ids-01-identity-substitution, ato-02-unreported-takeovers, vbp-01-duplicate-brands, and dsk-03-farm-rings-h2, accounts the answer key cannot fairly settle neither earn nor cost a run anything.
    • fc-14-back-pay is keyed on the full back-pay rule, mer-14-platform-funded-at-shells counts cancelled pickup orders whose charge stood, fin-01-collections-past-due weighs its position metrics more, and or-01b-coverage-cut-january-bare reads a plan as it is filed.
    • A run that broke on every attempt counts as filing nothing. Three such runs had been left out of their settings' averages: Muse Wonder at high effort on fc-20p-quest-desk-vs-finance-read, Claude Sonnet 5 at low effort on ds-23-margin-monthly-total, and DeepSeek V4.1 Flash 1017 at high effort on col-10-refund-partnerships-h.
    • Every run was re-graded. 52 settings rise, by at most 1.8 points, five fall by at most 0.2, and the top four keep their order.
    • Muse Wonder runs again on Meta's Responses API, which keeps its reasoning from one step to the next. Its extra-high score rises from 32.6 to 37.7, still eighth.
    • GPT-6.1 Terra joins the leaderboard in sixth place.
    • Claude Fable 5.2 joins the leaderboard in first place, at 70.2 with 41.4% of tasks solved.
    • Gemini 4 Carbon joins the leaderboard in second place, at 62.3 with 38.1% of tasks solved.
    • Eleven of Gemini 4 Carbon's low-effort runs, stopped at the four-hour limit by API calls that never returned, were resumed from their transcripts with two more hours.
    • GPT-6.2 Astra joins the leaderboard in second place, at 62.9 with 34.3% of tasks solved, in GPT-6 Astra's place on the chart.
    • GPT-6.2 Sol joins the leaderboard in fourth place, at 58.8 with 30.0% of tasks solved, in GPT-6.1 Sol's place on the chart.
    • Gemini 4 Neon joins the leaderboard in second place, at 64.6 with 38.6% of tasks solved.
  3. v1.4 Oct 2026

    Six harder tasks and Grok 4.7

    • Six tasks that most settings solved are retired: vd-e6-consumer-fee-peak-quarter, dhc-01-closure-order-sales, pe-07-instant-pay-price, fin-15-customer-credit-liabilities, cf-08-promo-acquisition, and pe-11-support-customer-exposure. Their runs stay in the record.
    • Six new tasks take their places, each in the same business area: cmp-31-cancelled-trip-make-whole, dhc-02-closure-order-sales-year, ip-02r-instant-fee-waiver-rule, fin-33-order-books-close, cf-42-new-customer-offer-revenue-presentation, and cr-01-november-courier-churn-vs-plan. Every model on the leaderboard ran them. No score moves by more than 2.3 points, and the order of the models does not change.
    • Grok 4.7 joins the leaderboard in sixth place.
  4. v1.3 Oct 2026

    Stricter grading on three questions

    • ds-90-menu-items-on-offer, ds-36t-min-pay-true-up-per-delivery-by-borough, and dhc-01-closure-order-sales no longer accept answers the warehouse contradicts. Every affected run was re-graded, no score on the leaderboard moves by more than 1.2 points, and the order does not change.
  5. v1.2 Oct 2026

    Grading fixes and three new models

    • Fixed four grading bugs: pay-01-diverted-payouts, cf-05-quest-reconcile, ds-34-commission-by-plan, and re-filed answers. Every affected run was re-graded, and no score on the leaderboard moves by more than 0.6 points.
    • Gemini 4 Argon, Muse Wonder, and DeepSeek V4.1 Flash 1017 join the leaderboard.
    • DeepSeek V4.1 Flash is repriced at what Fireworks charged during the run, after Fireworks raised its prices.
  6. v1.1 Sep 2026

    Claude Sonnet 5.5 and GPT-6.1 Sol

    • Claude Sonnet 5.5 and GPT-6.1 Sol join the leaderboard in third and fourth place, respectively.
    • Claude Haiku 5.5 joins in October, in Haiku 4.5's place on the chart.
    • The leaderboard reports each model at its highest reasoning effort, extra-high where it has one.
    • Forecasts are graded by interval score on an asinh scale against the median reference miss, floored at −2 per expectation, and every run is re-graded under the new rule.
    • Runs stopped by the one-hour limit were resumed from their transcripts with three more hours.
    • The four refund-collusion searches state how their grader weighs the review, and were run again on the new wording.
    • Storefronts that play fraud roles in the released warehouse carry pseudonyms.
  7. v1.0 Sep 2026

    First release, as submitted to ICLR 2027

    • 210 tasks across trust and safety, marketplace operations, FP&A, accounting and growth.
    • The public warehouse, the tasks and a reference agent; answer keys held out on a private world.

Argo-Bench

A frontier benchmark for enterprise-scale data science.

Argo-Bench contains no data about real people. Restaurants are real New York businesses from public records, except those cast in a fraud scenario, which carry fictional names; everything any of them does in the world is simulated. Map data © OpenStreetMap contributors.