# How to go from your agent's traces to a fine-tuned model in one workflow

- Channel: [LangChain](https://www.youtube.com/channel/UCC-lyoTfSrcJzA1ab3APAgw)
- Published: 2026-09-24
- Source video: [Watch on YouTube](https://youtube.com/watch?v=pb-AAvvQZ-U)
- Read on ReadTube: https://www.read.tube/p/videos/pb-AAvvQZ-U
- Language: Original (source language)

## Article

### From Agent Traces to a Fine-Tuned Model

Hi, it’s Jake from Langchain, and today we’re launching Langmith fine-tuning in public beta with Smith Tune, a CLI that allows you to post-train models from your Langmith traces in one workflow.

Supervised fine-tuning, or SFT, teaches a model desired behavior through golden examples. For agents, that might mean:

- Choosing the right tools.
- Following a particular workflow.
- Recovering from errors.
- Knowing when to ask for clarification.

You might use SFT to specialize a large model for your application, distill a frontier model’s behavior into a smaller model, or establish a starting policy for reinforcement learning. Depending on your goal and training data, it can improve task accuracy and tool reliability, or reduce cost and latency by teaching more efficient behavior.

Getting there from your agent traces usually means manually stitching together several steps:

1. Selecting good traces.
2. Preserving their messages and tool context.
3. Formatting training data.
4. Running fine-tuning.
5. Evaluating whether the resulting model actually improves.
6. Deploying that model if you like the result.

In this video, we’ll walk through how Smith Tune makes this process seamless in one end-to-end workflow.

### Representing the Agent’s Full Context

Firstly, it’s extremely important that we have a strong representation of the context behind each action that our agent took.

In Langsmith, this is represented in the trajectory format, where we can see a detailed reconstruction of the agent’s trajectory as it went out to complete a particular task.

We want to capture what the agent saw at each turn. It’s important to have a representation where we can see everything that the agent referenced, including things like the tools available to it at that turn.

This is particularly important because, in some cases for long-running agents, the tools available to the agent might change at each turn. We need to have a strong representation of that. Fortunately, Langmith’s trajectory format allows us to reconstruct this context in the exact way that we need for our post-training jobs.

### Pulling and Reviewing Sample Trajectories

Smith Tune starts with your Langsmith tracing project, where we have sample trajectories. From there, we’ll pull down those trajectories, optionally applying any filters, to store them in a local directory.

Next, you and your coding agent that’s using the Smith Tune CLI will work to define a task-specific rubric to help define what’s a good trajectory. This will help us keep the trajectories that we want to use as our golden examples.

That rubric will then be passed to a model council with three independent reviewers, who will each decide whether to keep or to drop a trajectory based on the rubric that you’ve worked to define.

We then upload those traces back to Langsmith to have a created data set that we’ll use in our supervised fine-tuning jobs.

### Checking the Smith Tune CLI

Let’s dive into using Smith Tune. First, I’m going to run the help command just to get a sense of what commands are available to me and to get a general layout of the CLI.

I can see that I need to run:

```bash
smith tune doctor
```

We can see that all my API keys and everything are set here. Another important part that I saw was that I need to confirm the data rights. I’ve gone ahead and done that already, and so we can see that saved version that I can reference for the remainder of using the CLI.

### Pulling Candidate Trajectories

Now I’m going to start working with my coding agent that’s going to help me in this data set pull process.

I’ve got this command copied here that asks Smith Tune to pull trajectories from my Open Suite Reviewer tracing project. This is an open-source coding agent that we use a lot at Langchain.

I want 50 target trajectories, with a maximum candidate size for it to go and look through of 500. I’ve given it the project workspace IDs as well as a filter so that it can help identify the correct traces.

We can see that my coding agent is reasoning about the way that it needs to pull these trajectories. We can see that it’s downloading a candidate pool, and then we’ll separately do a model council review after we define a rubric of what good looks like for us.

The pull is complete, and we’re ready for our council triage.

### Defining a Rubric and Judging the Traces

I’m going to ask it to help me define a rubric for what good looks like for these trajectories. We can see that it’s asking me what the reviewer training set should prioritize.

This first suggestion looks good. We can see that it’s helped me design a rubric. This looks great.

Now, I’m going to kick off the judging. I’m going to use Deepseek 41 Flash and GLM 5.3 Flash. I’ll limit it to two judges. Both of them need to agree on keeping an example for us to continue to have it in the data set.

After the judging stage is finished, we can see that we’ve got 50 trajectories that have been approved by both judges. I’m going to now go and look at this report and just confirm that I’m happy with the structure of what’s been accepted.

It looks great.

### Uploading the Golden Trajectories

Now I’m going to tell Smith Tune, in my coding agent, to please use the data set push command to upload this to Langsmith.

I know that it contains some PII after looking through it. I’ll take care and remove that once it’s in Langsmith.

The data set upload has completed, and I’ll go and check that out now. We’re ready for the next step in our process.

### Preparing the Data Set for Training

Now that we have our data set of golden trajectories, we’re going to prepare it so that it’s ready for training.

We’ll start from our created trajectories and then go and choose a provider, either base 10 or Fireworks, as well as a base model. From there, we’ll create the training, validation, and test splits that we’ll use when we train our model and then when we evaluate it to see how it performs.

Now we’ll run the prepare stage in Smith Tune. We’re going to specify our training provider. We’ll be using base 10. We’ll pass the workspace ID and the data set ID for the data set that we just created. We’ll also pass the model that we want to use.

In this case, we’re going to be using Quen 3.827B, which is a great model for supervised fine-tuning. It balances speed and accuracy well and has a large sequence length so that we can train on longer trajectories.

Once the prepare step is finished, we can see some high-level information that will be useful for us before we kick off the next actions. That includes:

- The model that we’re going to be training.
- The sequence length.
- Who our provider is.
- How many examples were prepared.
- How many examples were rejected.
- More information about the train and test split.

In this case, we’re going to train on 41 examples, test on five, and keep four for validation. This helps us with our data separation and makes sure that none of the examples that we train on leak into our validation or test sets.

For demonstration purposes, my data set size is quite small. If you were running an actual job, you’d want to have a much broader range of test, validation, and train examples.

### Planning the Training Job

Before starting training, we’ll run `Smith Tune plan`. This shows the selected model, the number of examples, and settings such as learning rate, batch size, and epochs. Review these, then pass your chosen settings to the training command.

Now that I’m nearly ready to train, I’m first going to run `Smith Tune plan` just to make sure that everything looks in order before I kick off a paid training job.

This is going to tell us some key information, as well as some other key configurations for the training job that we’ll run. This all looks good to me, and now I’m ready to kick off my training job.

### Fine-Tuning and Selecting the Best Checkpoint

Now run `smith tune train` to start fine-tuning with base 10 or Fireworks. In our case, we’ll be using base 10.

During training, Smith Tune checks performance on the validation set and selects the checkpoint with the lowest validation loss.

The plan that we generated earlier looks good, and now I’m ready to start training. The CLI will select the checkpoint with the lowest validation loss, compare it against our base model on our held-out test data, and publish the results to Langsmith.

The `confirm` flag authorizes the paid training and evaluation. Once our training job is finished, and because we provided the evaluate command, we’ll be provided with a link to the Langsmith experiment comparison view to see the results of our fine-tuning job.

### Comparing the Base and Fine-Tuned Models

The base Quen model is on the left, and the fine-tuned model is on the right. Each row represents a held-out trajectory, with individual replay predictions and judge explanations underneath.

The feedback chart summarizes agreement with the recorded reference actions. We can expand a row to inspect what each model predicted and why it passed or failed.

This wasn’t a very large job, and again, I wouldn’t expect to see much of a benefit in the tuned model that we have here. But we can see that our average agreement with the judge, or golden trajectory, is slightly higher on our fine-tuned Quen compared with our base Quen.

Promisingly, we can see the value that fine-tuning this model has provided for this particular task.

### Deploying the Model and Iterating

Finally, we have the deploy step. If the results meet your quality target, use `Smith Tune Deploy` to serve the tuned model and connect it to your application.

If the results need work, use the evaluation to refine your examples or training settings. Then run another iteration.

That’s the workflow:

1. Create.
2. Prepare.
3. Plan.
4. Train.
5. Evaluate.
6. Deploy.

Try Smith Tune on GitHub. The link is in the description, and we’d love to hear what you build.
