Production case study / LLM data engineering / 2026
An engineering write-up of a layered, automated validation pipeline for a production supervised fine-tuning dataset. What actually broke past a few thousand examples, and the defense that held.
You can have three thousand training examples that each look perfect, and the whole set can still be quietly broken. Catching that is the job this system does.
The work was on a fine-tuning dataset, which is the set of example conversations you train a model on so it learns to answer the way you want. The models here are large language models, the kind behind tools like ChatGPT. This one was meant to give an assistant a specific personality and a strict set of behaviors: roughly three thousand conversations across seven topic areas, growing one batch at a time. The task was not just to label data. It was to make the whole set trustworthy enough to train on, and to keep it trustworthy as it grew. Two things make that hard, and you cannot see either one by reading examples one at a time.
The first hard thing is that an example being correct on its own is not enough. A single conversation can read perfectly and still be wrong for training, because it repeats an answer shape that is already too common, or uses a banned phrase, or refuses in a way that breaks the rules, or states a fact that contradicts the subject.
The second hard thing is the dangerous one, and it only shows up across the whole batch. If every casual answer is casual in the same way, the model copies that one habit, and you cannot spot it by reading answers one at a time. You would have to measure the pattern across all three thousand. That is the failure this system is built to find.
I did not promise to remove the pattern completely. I promised to keep these patterns inside the range a real human writer would naturally produce, and to measure that range so we could prove it. The goal is manageable, not perfectly solvable, and saying that plainly on day one is what made the rest of the work credible.
What I delivered was a pipeline of automatic quality checks, not a one-time cleanup. Anyone on the team can run it on every new batch of data and get the same answers without me in the room. It is a defense built in three layers, and all three run on every batch.
A defense built in three layers, run on every batch
The first layer runs on each new batch in seconds, and it needs no model. It checks the required shape of each record, the writing voice, whether claims stick to allowed facts, and whether answers repeat each other. A tool sorts every answer by its shape, and if any one shape shows up too often inside the batch, the check flags it. blockwarnpass
The second layer catches the slow leak that the first one cannot see. A single answer shape can sit at twelve percent, then fourteen, then eleven across three weeks, never tripping any one batch, while the combined set is now overloaded with it. This layer is an audit that looks across all the batches together, not just the newest one, so a problem that no single batch reveals still surfaces.
The third layer is a watchlist that tracks whether known problems are getting better or worse. Specific failures the team has seen before get a measured starting point and a tracked verdict, so a known weakness stops being a gut feeling and becomes a number you can follow from one milestone to the next.
Underneath those three layers sits the full set of checks. There are sixteen numbered checks, plus the audit that looks across all the batches together, which makes seventeen in all. Each one watches for a single kind of failure, gives a clear pass or fail on every record, and writes its findings to a file so any flag can be traced back and audited. Here are six of them, as examples.
The over-used-phrase check blocks any single phrase that takes more than a quarter of the answers it applies to. It also has a safeguard so a small batch cannot trip a false alarm: under about a dozen such answers, it warns instead of blocking.
The biggest lesson was that a check can pass and still be lying to you. My first version felt finished and was not, and fixing that is the real story here.
The tool that sorts every answer by its shape started with a small set of shapes and a respectable hit rate. On a real, growing dataset that meant a chunk of the repetition slipped past it unsorted, so the limits were only policing part of the answers while the dashboard showed all green. Its catch rate was about sixteen percent. I widened it to the shapes that actually turned up in the data, and that, together with one small fix for curly quotation marks worth about eight points on its own, lifted the catch rate to about sixty percent. The check did not get smarter. It got honest about how much it was actually watching, and that is the part that matters, because the check that passes is the one you end up trusting.
A problem can build up across weeks of batches that each look fine on their own. That is the whole reason the audit that looks across all the batches together exists. Without it, the dataset passes every single-batch check and still teaches the model a bad habit. This is the most important design decision in the pipeline, and it is invisible if you only ever look at one batch at a time.
In a single batch, two nearly identical casual phrases took up far too much of the same group of answers: one at about twenty-nine percent, the other at twenty-five, together more than half of them. Every one of those answers read fine on its own. The problem only existed as a pattern across the group, which is exactly what a person checking one answer at a time cannot see and the over-used-phrase check can.
At real production scale, the quality bar is not a person reading samples. It is a system anyone can run and trust, where every check that passes is honest about how much it watches, and where a pattern that builds up slowly is measured, not guessed at.
These figures are from the live work. The client and the subject are not named here.
The pipeline runs on every new batch, folds it into the audit that spans all the batches, and produces two kinds of report: one a person can read, and one a machine can read for every record. It does not need its author present to run again, which was the whole point.
Most teaching material on fine-tuning stops at the file format and the command you type to start training. The part engineers actually struggle with in production is everything around the data: how you decide a dataset is trustworthy, how you keep its quality from drifting as it grows, and how you build a quality bar that holds up when more than one person is looking at it. That is a teachable, hands-on story, and it has a real working system behind it, which is the kind of thing I would walk through in a session.
This is one of several production systems I can teach from. The others include an inference server written in Rust for small language models, the software that runs a model and answers requests, which is open source, and a set of AI agents I run as part of my own daily engineering workflow.