Synthetic data generation often involves generating large volumes of data. For these large data generation jobs, a number of API providers offer discounts (from 50%-70%) through batch mode, which asynchronously processes data in large batches with a more relaxed horizon for returning responses (often up to 24 hours). However, many API users stick to online processing since managing batch jobs requires more steps for uploading, polling, downloading, and mapping requests back to responses.
Curator makes using batch APIs dead simple. All you have to do is add batch=True and our library handles the rest, giving you the same easy experience as online processing. You can now focus on generating the best data at the lowest price possible without the hassle of implementing your own batch API client logic.
We are launching Curator batch mode today with support for OpenAI, Anthropic, and other batch APIs compatible with OpenAI. kluster.ai provides open source models like DeepSeek-R1 via the compatible API and is easily accessed with Curator. We are partnering with kluster.ai so that new users of kluster.ai that use Curator can get a $25 credit (see instructions below) (valid till 16th February 2025).
Many inference providers offer up to 50% discounts through batch APIs. However, batch APIs are much tricker to use than online APIs, making it hard for users to take advantage of this discount. For example, instead of simply sending HTTP requests for inference, users have to write their requests into a batch file, upload this batch file, send a request for batch inference on the batch file, continuously poll for the batch’s completion status, and then download the batch once completed. Large datasets also don’t fit into a single batch due to batch size limits, forcing you to split the data into batches and manage many small batches in your code.
Furthermore, batch APIs are not standardized between providers, which means that many implementation details might differ between providers. This adds even more complexity if you want to experiment with different providers.
Curator batch mode abstracts away the complexity of batch APIs. With Curator, you only need to toggle a single flag in order to switch from online to batch mode and take advantage of cost savings. We also support a wide variety of providers, including OpenAI, Anthropic, and kluster.ai.
Note that DeepSeek-R1, a state-of-the-art reasoning model from DeepSeek, is available at up to a 70% discount when run in batch inference mode through kluster.ai. Moreover, new users using kluster.ai’s batch mode with Curator will get a $25 credit (limits apply). Please fill out this form to claim your credit before 16th February 2025.
To see Curator batch mode in action, please look at this example that uses DeepSeek-R1 through the kluster.ai batch API to annotate GSM8K.
Happy data generation!
