How models explore ideas

AutoResearchExam measures the quality, speed and generalization of the solutions models find on long-horizon ML research and engineering tasks.

We wanted to understand how models go about solving these problems. Does a model commit to its first approach or keep trying new ones, and does trying more ideas actually lead to better solutions? To study this we use idea basins, introduced in Fork and Flush. An idea basin is a group of submissions to the same task that share a broad method, such as tuning a gradient-boosted model or training a small neural net. Once every submission is placed in a basin, a run becomes a path through ideas, and each move from one basin to another is a switch. We use these paths to answer three questions:

1. Do some models explore more ideas than others?
2. What does exploration say about research style?
3. Does exploring more lead to higher scores?

We find that most runs stick with one or two ideas, and models that submit more spend the extra submissions refining and overfit more. Yet exploring a new idea gains about nine times as much as a refining step!
Models explore most and benefit most in the first six hours, after which visits to new idea basins drop sharply. This suggests models start refining instead of exploring further, leaving potential gains on the table: 58% of runs never reach the basin where the best solution to their task was found.

Building the idea map

We build the map the same way Fork and Flush does, with Nimble as the judge. To find basins we need to judge which submissions use the same idea, without looking at scores. Two very different ideas can get the same score, so we group submissions by what their code does instead. Sonnet first summarizes the method in each submission's code. Then Nimble, our frontier System One model, compares the summaries three at a time ("is B or C closer to A?"), without knowing which model wrote them. This takes about 1.3 million calls, and Nimble answers them all for under $40 (and it's now available through our API!). We fit a 2D map to those answers, one per task, and cluster it into basins.

Do some models explore more ideas than others?

We look at exploration three ways:

1. Basins visited. The average run visits only 1.5 to 2 basins, and 47% of runs never leave their first one.
2. Where switches go. When a model changes approach, half the time it goes back to an idea it already tried instead of a brand new one.
3. Switch rate. Only 4 to 13% of a model's submissions try a different idea. The rest refine the one it already has.

No model is a clear explorer. Most runs pick an idea early and stay with it for the full 24 hours. This matches Fork and Flush, where single runs mostly stay inside one basin for their whole budget.

Basins visited

Moving on vs. going back

Switch rate

What stands out by model

• Staying put varies about twofold. Qwen3.8 Max, Grok 4.6 and Claude Fable 5.1 never leave their first idea in about 60% of runs. For GPT-5.6 Sol, Argon and Gemini 3.8 Flash it is about a third.
• GPT-6 Astra, Gemini 3.8 Flash and Muse Spark 1.3 switch the most, but about two thirds of their switches go back to ideas they already tried. Argon rarely goes back: only a quarter of its switches are returns.
• GPT-6 Astra submits half as often as GPT-5.6 Sol, yet goes back to old ideas twice as often. It still comes out ahead, with higher test AUARC (0.60 vs. 0.51) and a quarter of Sol's overfitting gap.
• Grok 4.7 is a noticeably bolder searcher than Grok 4.6. It switches 1.5 times as often and leaves its first idea in 55% of runs, up from 41%. Grok 4.6 switches on only 4% of its steps, half the rate of most models.

What does exploration tell us about research style?

The main AutoResearchExam results describe research style through submission volume and overfitting. We check both against exploration:

1. Do models that submit more explore more?They switch about as often as everyone else, so the extra submissions go into refining. GPT-5.6 Sol and both Groks submit about three times as often as Opus 5.5 and Argon but switch about as many times, so their switch rate is much lower.
2. Does exploring relate to overfitting?Overfitting tracks how much a model submits. Models that switch more do overfit less (ρ = −0.63), but they also submit less. Submission volume tracks the gap much more closely (ρ = 0.84), and among models that submit about as much, switching barely matters.

Number of submissions

Overfitting

Does exploring more lead to higher scores?

1. Do models that explore more score higher?Yes, if their switches go to new ideas, and most of all early on. The four top scorers (Opus 5.5, Claude Fable 5.1, GPT-6 Astra and Argon) send more of their switches to new basins than the lowest scorers do (ρ = 0.70 across models). The link is even stronger for test AUARC in the first 6 hours (ρ = 0.84). How often a model switches shows no link.
2. What is a switch worth?A switch to a new idea is followed by about nine times the gain of a refinement step, and it is about twice as likely to set a new best score. A return to an old idea gains about as much as staying.
3. When does switching pay off?In the first few hours. In hours 0 to 6 a switch of any kind gains about 3.5 times as much as a refinement step. After hour 6 the two are about equal.
4. Is late growth tied to switching?We found no link. Runs that switched after hour 12 gained no more test score over the second half than runs that stayed. Many runs never find the strongest idea at all: 58% never visit the basin where their task's best solution came from.

Exploration and score

Payoff of a switch

Timing

Late growth

Takeaways

1. Most runs stay with one idea. Nearly half of all runs stay with their first idea for the full 24 hours, and half of all switches go back to an idea the run already tried.
2. The top scorers send most of their switches to new ideas. How often a model switches has no link to its score.
3. New ideas pay off early. A switch to a new idea is followed by about nine times the gain of a refinement step, almost all of it in the first 6 hours. After that, refining does as well, and we see no evidence of models switching ideas well late in a run.
4. Heavy submitters mostly refine, and they overfit the most. The highest-volume models switch the least per submission and lose the most score on the hidden test set.

We therefore see an opportunity in model and harness design: push agents to explore new ideas early, and to keep exploring well later in the run so improvements continue over longer time horizons.

Appendix: Method details

Citation

If you found this useful, please cite it as:

@misc{ramesh2026explore,
author = {Ramesh, Anirudha and Parulekar, Advait and Garg, Shivank and Devic, Sid and Dimakis, Alex and Sathiamoorthy, Maheswaran and {Bespoke Labs}},
title  = {How models explore ideas},
year   = {2026},
url    = {https://www.bespokelabs.ai/blog/how-models-explore-ideas},
note   = {Bespoke Labs Blog}
}