Supervised fine-tuning
By Gareth HoyleLast updated How we research
Supervised fine-tuning is further training of a pre-trained model on example inputs paired with target outputs, so it learns to produce the desired responses.
Supervised fine-tuning (SFT) is the stage in which a pre-trained model is trained further on examples of inputs paired with the outputs it should produce. The model's weights are updated to make the target outputs more likely, usually by minimizing cross-entropy loss on the response tokens.
Where does supervised fine-tuning fit in training?
Supervised fine-tuning comes after pre-training and usually before preference tuning. In OpenAI's 2022 InstructGPT paper by Long Ouyang and colleagues, human labelers wrote demonstrations of the desired behavior, the model was fine-tuned on them and RLHF followed.
How much data does supervised fine-tuning need?
Supervised fine-tuning needs far less data than pre-training, and quality matters more than volume. The InstructGPT supervised dataset held about 13,000 training prompts.
Company records that suit SFT include expert answers to real questions, resolved support tickets and reviewed code changes. See expert domain data and support ticket data.