Learning to use tools is critical for building agents that can interact with the external world. For example, a deep research agent for private enterprise data needs to master a wide range of tools, such as search, SQL, and Pandas, to generate a report. It must not only be able to invoke a single tool in isolation, but also combine the output of multiple tools as the input to another — in essence, the agent should be able to orchestrate many tools.
Manual prompt engineering for building agents is easy to start with, but doesn’t scale well. To begin with, the list of tools may be long and diverse. Moreover, a prompt engineer designing the agent has to spend increasing amounts of time to cover more and more edge cases with handcrafted rules through trial-and-error. Such a manual approach is reminiscent of what Rich Sutton warned against in his essay, The Bitter Lesson:
"1) AI researchers have often tried to build knowledge into their agents, 2) this always helps in the short term, and is personally satisfying to the researcher, but 3) in the long run it plateaus and even inhibits further progress, and 4) breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning"
An alternative method is to use supervised finetuning, where the model learns from demonstrations of responses to user queries. We can collect these demonstrations from humans, which can be expensive. We can also collect these demonstrations from a teacher model, but this only works when we have existing teacher models that are already very good at the task.
We need a more scalable method that can teach agents to use tools.
Reinforcement learning (RL) algorithms can take on many forms, such as PPO or GRPO, but at their core, they allow models to learn by trial-and-error. A model explores a wide range of actions and receives feedback from the environment in the form of rewards. Actions that result in high reward are then reinforced and actions that result in low reward are discouraged. In this way, the model learns to take actions that maximize its reward.
RL is a scalable learning algorithm because it allows models to self-discover solutions by learning from its own experience. Unlike prompt engineering or supervised finetuning, which are both ultimately bottlenecked by human-generated data, RL holds the promise of scaling arbitrarily with compute, a key property for successful algorithms outlined in The Bitter Lesson.
Recently, OpenAI has demonstrated that RL can be used to train a research agent that uses tools to carry out complex, multi-step workflows. However, they did not disclose a lot of details about their training recipe. Meanwhile, experiments in the open-source and research community are often single-turn, lacking back-and-forth interaction with an environment.
We are excited to share some recent findings which show that RL can improve multi-step tool use capabilities, a first milestone. Using GRPO, the core algorithm behind DeepSeek-R1, we improved Qwen2.5-7B-Instruct’s tool use performance by 23% on a subset of the BFCL benchmark using only 100 training samples. The tasks in this benchmark require an agent to orchestrate multiple tools to complete multi-step tasks in a simulated environment, such as booking air travel using credit cards and placing orders in a trading system.
We found that our agent exhibited qualitatively better tool use capabilities after undergoing training with RL. As an example, consider the scenario below, where the agent is tasked with booking a flight from Los Angeles to New York using the user's credit card:
Before training, the agent attempts a single tool call, book_flight, using a non-existent airport for New York (NYC) and the wrong cost ($0).
After training, the agent is able to solve the task successfully. It is able to:
Concretely, the agent:
This is an encouraging result: our agent learned to orchestrate tools without seeing a single demonstration from a human or a teacher model!
You can browse more examples that show such improvements in our agent evaluation app.
We used 200 questions from the BFCL benchmark’s v3 multi-turn-base category and performed a 50-50 train-test split, picking only the first query of each example to speed up training.
We designed a simple reward function that gives the agent a reward of 1 when it passes BFCL’s evaluation check, which consists of a state-based check and a response-based check (more details here), and 0 otherwise.
We then trained Qwen-2.5-7B-Instruct for 1600 steps (100 epochs) using GRPO with the following hyperparameters:
More details can be found in our training code, which we adapted from Will Brown's verifiers library.
We found that the agent improved by 23% on the eval set (from 55% to 78%) with the recipe above while maintaining training stability.
In this section, we detail the many findings that led us to our final training recipe.
A major issue that occurred in many of our experiments is completion length blowup, where the agent starts outputting longer and longer gibberish responses with random characters, see below.
This blowup in response length also leads to a complete collapse in training and eval accuracy.
We tried many treatments to mitigate this training instability, which we describe below. It is worth noting that in all of the below settings, we performed GRPO with μ = 2 (one on-policy step and one off-policy step taken for every batch) to ensure fast convergence.
❌ Dr.GRPO: We found that re-scaling the reward and replacing sequence-level normalization with a constant, following the Dr.GRPO paper, still resulted in completion length blowup.
❌ Gibberish Penalty: We used gpt-4o-mini as a judge to detect gibberish in agent trajectories and assign -1 reward when gibberish is detected. We found that the gpt-4o-mini judge was able to detect such gibberish responses with very high accuracy. Despite this and to our surprise, length completion blowup occurred even earlier with the gibberish penalty added. Our hypothesis here is that negative rewards encourage exploration which in turn further encourages gibberish.
❌ Overlong Filtering + Removing KL Divergence Penalty: We tried using overlong filtering and removing the KL divergence term, as suggested by the DAPO paper. Removing the KL divergence term was very appealing to us since this significantly simplifies the loss calculation and speeds up training by allowing us to avoid keeping an extra reference model in memory and skip the KL penalty calculation. However, while we did see faster training, completion length blowup, and accuracy collapse occurred very quickly, at around 300 steps. Our result corroborates a similar finding in a discussion thread from the HuggingFace team.
✅ Overlong Filtering + KL Weight = 0.001: This was the treatment included in our final recipe. After experimenting with both high KL weight (0.04) and no KL weight (0) and seeing collapse on both settings, we decide to pick a small KL weight of 0.001 (inspired by this comment from the discussion thread above), which resulted in our best-performing and most stable run.
Our initial reward design consisted of three components:
We found that this complex reward design leads to worse training stability, presumably due to reward hacking by the agent. In the graph below, green lines are results from an experiment containing reward components 1, 2, and 3 above, red lines are results from an experiment containing components 2 and 3, and pink lines are results from an experiment containing only component 3. The most straightforward reward design, containing only the correctness component, has the best stability and performance.
We found that updating the reference model every 100 steps can boost performance compared to using the same reference model throughout the entire training process. In the graph below, pink lines are results from an experiment with reference model update, whereas blue lines are results from an experiment without reference model update.
Intuitively, since the KL term will pull the current policy model towards the reference model, a stronger policy model requires a stronger reference model as a regulator. Regularly updating the reference model ensures that the KL term does not pull the model completely towards the initial performance while still adding some stability to the training process.
True to our name, we want to make it very easy to build bespoke agents. We believe this is best done using scalable methods such as RL. This work represents a first step towards this mission.
We believe there is a gap in tooling to deeply understand, explore, and evaluate agent behaviors. Our agent evaluation app is a demo of what we think the tooling should look like. Please reach out to us at company@bespokelabs.ai if you're interested in collaborating to build bespoke agents for your use case.
@misc{bespoke_improving_multi_turn_tool_use,
author = {Bespoke Labs},
title = {Improving Multi-Turn Tool Use with Reinforcement Learning},
howpublished = {https://www.bespokelabs.ai/blog/improving-multi-turn-tool-use-with-reinforcement-learning},
note = {Accessed: 2025-04-17},
year = {2025}
}
