Long-term coherence in agents is more important than ever. Coding agents can now write code autonomously for hours, and the length and breadth of tasks AI models are able to complete is likely to increase. We expect models to soon take active part in the economy, managing entire businesses. But to do this, they have to stay coherent and efficient over very long time horizons. This is what Vending-Bench 2 measures: the ability of models to stay coherent and successfully manage a simulated business over the course of a year. Our results show that while models are improving at this, current frontier models handle this with varying degrees of success.
Money balance over time
Average across 5 runs
Claude Opus 4.6
Claude Sonnet 4.6
GPT-5.4
Gemini 3 Pro
Grok 4.20
GLM-5
Current leaderboard
Model
Money Balance
1
Claude Opus 4.6
$8,017.59
2
Claude Sonnet 4.6
$7,204.14
3
GPT-5.4New
$6,144.18
4
GPT-5.3-Codex
$5,940.12
5
Gemini 3 Pro
$5,478.16
The leaderboard shows significant spread in performance. The top-performing models tend to share two traits: they maintain a consistent rate of tool use throughout the year-long simulation with no signs of performance degradation, and they are effective at sourcing products at good prices — whether through persistent negotiation or by finding better suppliers.
Vending-Bench Arena is a version of Vending-Bench 2 that adds a crucial component: competition. It's our first multi-agent eval, where all participating agents manage their own vending machine at the same location. This leads to price wars and tough strategy decisions.
Performance vs. release date
SOTA frontier models are labeled and a trend line is fitted through them, with a projection into the near future.
Linear fit (R² = 0.97), +$693/month
Frontier lag analysis
Comparing SOTA frontier progression between model groups, with linear regression and projected crossover points.
Western
Chinese
Chinese: +$1,398/month (R² = 0.99) · Western: +$693/month (R² = 0.97) · Chinese lags by ~136 days · Projected crossover: Jun 2026
Score vs. cost per run
Score vs. mean cost per run using each LLM provider's API to run Vending-Bench 2. Costs are calculated from the provider's input and output token pricing, without caching.
Improvements from our original Vending-Bench
Vending-Bench 2 keeps the core idea from Vending-Bench of managing a business in a lifelike setting, but introduces more real-world messiness inspired by learnings from our vending machine deployments:
- Suppliers may be adversarial and actively try to exploit the agent, quoting unreasonable prices or even trying bait-and-switch tactics. The agents must realize this and look for other options to stay profitable.
- Negotiation is key to success. Even honest suppliers will try to get the most out of their customers.
- Deliveries can be delayed and trusted suppliers can go out of business, forcing agents to build robust supply chains and always have a plan B.
- Unhappy customers can reach out at any time demanding costly refunds.
We've also streamlined the scoring system, evaluating models on money balance after a year and clarified the scoring criteria, such that agents know exactly what to optimize for. Better planning tools, such as proper note-taking and reminder systems have been added as well.
Qualitative findings
Note: The qualitative analysis below was written when Gemini 3 Pro led the leaderboard. The charts and leaderboard above are always kept up to date with the latest models.
Here are some takeaways from the models we have tested so far.
Gemini 3 Pro is a persistent negotiator
Where other models may sometimes give up and accept a high price when it struggles to find good suppliers, Gemini 3 Pro consistently knows what to expect from a wholesale supplier and keeps negotiating or searching for new suppliers until it finds a reasonable offer.
assistant • Gemini 3 Pro
read_email
Models are good at finding honest suppliers
The suppliers in Vending-Bench 2 can be grouped into four main categories, two of which are honest and two of which are clearly adversarial. We see that models are generally good at finding honest suppliers.
Friendly
Negotiator
Membership Scammer
Price Scammer
Gemini models spend an unusually large share of their money on orders from friendly suppliers. Based on Gemini 3 Pro's performance, this seems to pay off. However, this is an interesting tradeoff, as negotiating suppliers may start by quoting a higher price initially but go even lower after negotiation.
GPT-5.1 struggles
Compared to similar models, GPT-5.1's performance is underwhelming, especially in Vending-Bench Arena. We hypothesize that this comes down to GPT-5.1 having too much trust in its environment and its suppliers.
assistant
send_email
How Vending-Bench works
Models are tasked with making as much money as possible managing their vending business given a $500 starting balance. They are given a year, unless they go bankrupt and fail to pay the $2 daily fee for the vending machine for more than 10 consecutive days, in which case they are terminated early.
A good way to understand the benchmark is to read the system prompt given to the agents:
system
Where's the ceiling?
In many benchmarks, the main metric is a percentage of tasks completed or questions answered correctly. For Vending-Bench, the main metric is dollars made. We've designed it so there's no ceiling. A perfect strategy would look something like this:
- Find suppliers for extremely valuable items (there's nothing stopping the model from sourcing items with higher value than what's typically found in a vending machine)
- Negotiate down the price to zero (the suppliers are other LLMs who can be jailbroken to give away stuff for free)
- Keep the machine always stocked in an optimal configuration (daily sales are simulated based on equations that can be gamed)
We estimate that a "good" performance could easily do 10x better than the current best LLMs. Putting this together, a "good" strategy could make $206 per day for 302 days — roughly $63k in a year.
Score ($)
The gap between current models and this "good" baseline shows there's plenty of headroom in Vending-Bench 2. Models are getting better at staying coherent over long time horizons, but there are still analytical skills required that need to be applied in the right way to get a maximal score, that models do not currently exhibit.