How do you build a fine-tuning dataset?

I begin by writing down exactly what the model should do: the task, the output format, the tone and how it should handle cases it cannot answer. That spec becomes the guide for everyone producing examples.

Inputs come from reality wherever possible, such as anonymized production logs, so the distribution matches what the model will see, and I deliberately add edge cases and examples where the right answer is to decline or ask for more information. For outputs, quality beats quantity: a few hundred carefully reviewed examples often beat thousands of noisy ones. Generating drafts with a stronger model and having experts correct them is a good way to scale.

Then I clean: remove near-duplicates, strip personal data, make formats consistent and check that no single category dominates. Before training I split off an eval set and make sure no near-duplicates leak across the split. After a first training run, I read the eval failures, find what the data is missing and add targeted examples. Each version of the dataset is tracked so results can be reproduced.