Evals with Jev
How I run evals on a real feature: Jev, a cheap classification model, filtering alerts in Sputnik Intelligence. Build a golden dataset with a tiny labeling UI, work out what good looks like with a smart model, then score Jev against it.
Evals with Jev
How do you know a cheap model is good enough? Build a golden dataset and test it.
- All 2 videos in this series
- Prompts, templates and resources from each episode
- Watch at your own pace, on any device
- Just the videos.
Free. No credit card required.
What you'll learn
- A practical walkthrough of how I do evals
- How to discuss evals with your smartest model
- What a golden dataset looks like
- What a good example UI for labeling looks like
- How Claude can improve the task iteratively by itself once you have a golden dataset
Learn from Peter
Peter Van Dijck
Peter has been building AI products since 2023 and teaching teams how to build them since 2024. He has spent that time in the trenches with evals, observability, synthetic data and context engineering, and turns what actually works into practical, no-fluff lessons.
Every episode is taught by Peter himself, in plain language, for product managers, designers, researchers and strategists who need to understand how AI systems are really built, without needing to be an engineer.
Peter on LinkedInWhat's included
- All 2 videos in this series
- Prompts, templates and resources from each episode
- Watch at your own pace, on any device
- Just the videos.
What participants say
“This is opening up all sorts of new neural pathways for me to see under the hood more of how the sausage is made! 🙏”
“Very timely at my enterprise software company as evaluation of AI features scales.”
“Everything I know about evals is from Peter's talk, which is why I'm back to find out more!”
The videos
1. Evals with Jev
Building a golden dataset
Jev is a cheap model that sorts things into categories. In the alerts feature of Sputnik Intelligence, it reads a podcast excerpt and an alert filter ("AI in the context of politics") and answers yes or no. To test it, we need a golden dataset, so I had Claude build a small labeling UI with keyboard shortcuts and I label the rows myself.
Running the eval
After labeling a batch, I talk the labels through with Claude Opus to pin down what good looks like, then let it label the rest. Then we run Jev against the golden dataset: 88% correct. Why that's fine for this task, why 99% on the first try is a warning sign, and how this makes switching models easy.