Personalizing Inkling for your Code Repository with Post-Training

Abstract

Can an open coding model be post-trained to work better on a specific repository? How do we curate the RL environments and how do we use SFT and RL to improve performance? To answer these questions, we curate coding tasks specialized for a repository and post-train Inkling using SFT and RL. Our post-trained model achieves a significant lift in accuracy on held-out fontTools evaluation coding tasks and also demonstrates good general performance on other benchmarks we did not post-train on.

We use supervised fine-tuning (SFT) with trajectories coming from a strong teacher model, and reinforcement learning (GRPO) on the fontTools specialized environments that we curate. SFT gives a 52% improvement in performance on the held-out fontTools evaluation set. Further RL training lifts the total improvement to 57% compared to the base Inkling model. In addition to the in-distribution evaluation set on fontTools coding tasks, we test our models on Terminal-Bench 2.1 and SWE-Bench Lite. Our experiments show that the post-trained models roughly match the performance of the base Inkling, but become up to 40% more token efficient from post-training.

Introduction

There is growing interest in building specialized and personalized models from proprietary data. Specifically, we focus on improving performance and token efficiency of coding agents on a custom repository. In this blog post, we create training data from a repository (fontTools) using SWE-Smith techniques, and post-train Inkling to show significant performance and token efficiency gains in the target codebase, while still retaining generally good performance.

Methodology and experimental setup

Repository environments and tasks

For supervised fine-tuning, we create a fontTools task set with 58 tasks for supervised training and 10 held-out tasks for in-repository evaluation. Each task contains a pinned repository state, an injected defect, a task instruction, and a grader that checks whether the intended repair holds under the repository test suite. As one would hope, the held-out set never enters either training stage.

Our tasks are designed to be especially challenging for the model: we present a bug to the model - but also add a second soft bug whose location isn’t explicitly mentioned to the agent, but can be discovered in the environment. In order to pass the task, the model, like a good software engineer, has to solve both bugs successfully.

We also create 10 SQLGlot tasks in the same SWE-smith-style format. SQLGlot is an unseen repository: its tasks, trajectories, patches, and rubrics are excluded from both SFT and RL. We use these tasks to answer the question: does any behavior from fontTools transfer to a new, unseen repository?

Supervised fine-tuning

For SFT, we use a strong teacher model (Claude Opus 4.8) to solve the set of 58 tasks we engineer for supervised fine-tuning. Across the 58 tasks, we generate 146 successful trajectories - with some tasks having 1, 2, or 3 trajectories from the teacher model.

Reinforcement learning with GRPO

Our RL training begins from the SFT checkpoint, not from the base model. Our goal, through RL, is to improve performance and token efficiency. To achieve this, we use rubric-guided Group Relative Policy Optimization. We engineer a set of 14 rubrics contributing to the RL reward, each designed to improve the model's efficiency through targeted reasoning and planning. We layer these rubrics on top of a set of 92 new tasks on fontTools, which we curate in a similar style to the previous ones and use exclusively for RL. Through RL, we seek two improvements: improvement in performance, coupled with improved token efficiency.

Evaluation protocol

For our evaluation, we run 10 rollouts per task, across 10 tasks each derived from fontTools(in-distribution) and SQLGlot(out-of-distribution). The model has never been exposed to any of these tasks in its prior training. We report pass rate and mean output tokens per scheduled rollout. Terminal-Bench 2.1 and SWE-bench Lite provide a broader check on whether the specialized model preserves general coding performance while maintaining its token efficiency capabilities.

To test the model’s performance as it is tested in a day-to-day use case, we use the model’s default (0.9) effort setting. We benchmark the model across Terminal-Bench 2.1 and SWE-bench Lite, and see scores comparable to those reported with the same setup by Artificial Analysis, using the Terminus2 harness.

Results

Mean pass rate ± one standard error.

In-distribution Repository

On the 10 fontTools evaluation tasks, base Inkling passes 0 of 100 fontTools attempts. SFT passes 52 of 100. SFT + RL passes 57 of 100, a five percent improvement over SFT and a 57-point improvement over the base model.

Held-out fontTools under the ordinary fix-it prompt: 10 tasks × 10 graded attempts.

Out-of-distribution Benchmarks

Apart from the 10 evaluation tasks derived from fontTools, we also evaluate the base, SFT, and SFT + RL versions of Inkling on 3 other evaluations. Our first out-of-distribution evaluation is through 10 SWE-Smith style tasks, similar to the 10 fontTools tasks, but derived from the SQLGlot repository - a repository the model has not seen through our post-training. Prior to SFT, the base Inkling scores 0% on these 10 tasks, with 10 rollouts per task. After SFT, the model jumps to 49% in performance, reaching 54% after RL. From SFT to RL, the model uses, on average, 13.7% fewer tokens. We then run the model on Terminal-Bench 2.1. Here, SFT makes the model 1.1% better in performance, and RL makes the model 2.7% better versus base Inkling. However, the significant shift we notice is in token efficiency. After SFT, the model uses 44.1% fewer tokens - a reduction RL matches as well. Our third out-of-distribution evaluation is on SWE-Bench Lite. After SFT, we see a 0.5% decrease in performance (which falls under the standard error). However, similar to our observations from Terminal-Bench 2.1, the model after SFT uses 26.9% fewer tokens. After RL, the model uses 34.9% fewer tokens than the base model, and 10.9% fewer than the SFT model. In addition, the RL model's performance jumps by 3.1% versus the base, and 3.6% versus the SFT model.

Evaluation

Mean output tokens per scheduled rollout. Each SWESmith setting contains 100 ordinary-prompt attempts.

Benchmark tables

The four tables below showcase the key findings from all three versions of Inkling we evaluate. In addition to tracking pass rate for performance, and output tokens for token efficiency, we add other metrics to view model efficiency.

Terminal-Bench 2.1

fixed 86-task cohort · three samples for Inkling and SFT · Typhoon · 8,000-token limit

SWE-bench Lite

fixed 300-task cohort · three samples for Inkling and SFT · Typhoon · 8,000-token limit

Output tokens are the primary efficiency measure in this cross-repository comparison; shell, turn, and pytest counts remain supporting behavioral diagnostics.

SWESmith · fontTools

ordinary fix-it prompt · 10 held-out tasks × 10 graded attempts

SWESmith · SQLGlot

ordinary fix-it prompt · 10 held-out tasks × 10 graded attempts

Analysis and discussion

We train the model on fontTools coding tasks. After SFT, the pass rate on held-out fontTools tasks goes from 0% to 52%, and RL raises it to 57%. These improvements show that post-training achieves its objective of personalizing the model.

Then, we evaluate the same checkpoints on SQLGlot, a different coding repository. We emphasize that none of the SQLGlot tasks, trajectories, patches, or rubrics have been used in our post-training. As before, the base model scores 0% while the SFT model scores 49%.

So the interesting finding is that training on one repository also produces a significant lift in another.

One possible explanation is that the model learns general tricks for repairing software bugs. The teacher trajectories do not just contain facts about fontTools but show how to enter an unfamiliar repository, search for the relevant code, understand the local abstractions, make a change, run tests, and continue when the first attempt fails. Those are repository-independent skills that can be learned by SFTing on strong trajectories.

The alternative explanation is that both evaluation sets have the same structure as the coding tasks we curate. The post-trained model may become better at fixing the specific types of defects that our curation method constructs. This is a narrower benefit, and we plan to perform additional research ablations to study this more.

A second result is less dramatic but equally clear: SFT does most of the work. The 146 successful trajectories we use for SFT move performance on fontTools from 0% to 52% and SQLGlot from 0% to 49%. RL adds five percentage points on each. SFT teaches the model the main behavior, while RL helps refine it.

The token efficiency results tell a similar story. On Terminal-Bench, SFT cuts average output tokens by 44.1%, and RL retains essentially the same token use. On SWE-bench Lite, SFT cuts output tokens by 26.9%, while RL increases the reduction to 34.9%. On SQLGlot, RL uses 13.7% fewer tokens than SFT while solving more tasks.

But “fewer tokens” is not always the same as “more efficient.” The base model produces the shortest SQLGlot trajectories and solves nothing: giving up is extremely token-efficient. What matters is whether a model can use fewer tokens while preserving or improving its probability of success. By that definition, RL still succeeds across all benchmarks - improving either performance, token efficiency, or both, without any regression.

We start with a repository-specialization question: can post-training make an open model great at working on one codebase? The answer is a clear "yes." But the experiment leaves us with a more interesting question: why does training on one codebase make the model better on another?

The next version of this experiment should use more repositories and, critically, more diverse tasks. If the transfer benefits survive that test, then repository-specific post-training may be less repository-specific than its name suggests.

Limitations and conclusion

The post-training we do is not representative of the results of post-training in general. We target improvement in only one repository - and manage to see improvement across other environments as well. We do not claim that our Inkling + SFT + RL model is efficient in every use case; however, it proves more efficient versus the base in evaluations that are critical for coding agents and relate to this post's original question.

Within those limits, the answer to the original practical question is yes. Creating executable, repository-specific repair tasks; using strong teacher trajectories for SFT; and applying targeted RL produces a model that is substantially more successful on held-out fontTools tasks. The same model retains similar observed performance on broader benchmarks while using fewer tokens, and RL adds an efficiency-and-outcome improvement on unseen SQLGlot. The next question is how far this recipe transfers across languages, test frameworks, and repository sizes.

Citation

If you use this work, please cite it as:

@misc{bespoke_personalizing_inkling_for_your_code,
    author = {Patel, Romir* and Dimakis, Alex and Sathiamoorthy, Maheswaran},
    title = {Personalizing Inkling For Your Code Repository with Post-Training},
    howpublished = {https://www.bespokelabs.ai/blog/personalizing-inkling-for-your-code-repository-with-post-training},
    note = {Published: 2026-09-02},
    year = {2026}
}

@misc{bespoke_personalizing_inkling_for_your_code,
    author = {Patel, Romir* and Dimakis, Alex and Sathiamoorthy, Maheswaran},
    title = {Personalizing Inkling For Your Code Repository with Post-Training},
    howpublished = {https://www.bespokelabs.ai/blog/personalizing-inkling-for-your-code-repository-with-post-training},
    note = {Published: 2026-09-02},
    year = {2026}
}

@misc{bespoke_personalizing_inkling_for_your_code,
    author = {Patel, Romir* and Dimakis, Alex and Sathiamoorthy, Maheswaran},
    title = {Personalizing Inkling For Your Code Repository with Post-Training},
    howpublished = {https://www.bespokelabs.ai/blog/personalizing-inkling-for-your-code-repository-with-post-training},
    note = {Published: 2026-09-02},
    year = {2026}
}

References

1. Yang, J. et al. SWE-smith: Scaling Data for Software Engineering Agents. 2025.

2. Jimenez, C. E. et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. ICLR, 2024.

3. Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024. Introduces GRPO.