InstructGPT: How RLHF Became the OpenAI API Default

What happened on this date, and what did not

On January 27, 2022 OpenAI announced that instruction-tuned models were the default language models on its API. The models were not new that day. An earlier version had been deployed in beta in January 2021, and the post says they had been on the API for more than a year. The paper describing the method followed on March 4. The post also notes that the deployed models are updated versions trained with a similar but slightly different method than the one written up. [1][2]

Atlas interpretation: The date is chosen for the default switch rather than the paper because that is when the technique stopped being research and became the behavior every API customer got unless they opted out. OpenAI framed it that way at the time, saying it was the first time its alignment research had been applied to its product, and Jan Leike repeated the point to a reporter the same day. Nothing was invented on this date. A default changed. [1][3]

Three stages, and what the humans did in each

Around 40 contractors, screened on a test, produced the training data in two different shapes. In the first stage they wrote answers themselves, demonstrating the desired response to prompts drawn from the API playground and from prompts they invented, and GPT-3 was fine-tuned by ordinary supervised learning to imitate them. In the second stage they wrote nothing: they ranked several model-generated candidate answers to the same prompt, and those rankings trained a separate reward model to score any output. [2]

The third stage has no humans in it at all. The reward model stands in for them as a reward function, and the fine-tuned model is optimized against it with Proximal Policy Optimization. A variant mixes in ordinary pretraining updates alongside. OpenAI notes the whole procedure uses under two percent of the compute and data of pretraining, which is why it describes the result as unlocking capabilities the base model already had rather than adding new ones. [2][1]

The result everyone quotes, stated precisely

Labelers preferred the outputs of a 1.3 billion parameter instruction-tuned model to those of the 175 billion parameter GPT-3, a model over a hundred times larger. The paper's framing is that the two share the GPT-3 architecture and differ by this fine-tuning, setting the size gap aside. The published win rates, however, are for the largest model: the 175B instruction-tuned model was preferred to GPT-3 85 percent of the time, plus or minus 3, and to few-shot GPT-3 71 percent of the time, plus or minus 4. [2]

The conditions matter. The judges are the same hired labelers who produced the training data, on prompts submitted by held-out API customers, and the headline metric is how often they preferred one model's output to another's rather than an absolute score. A separate one-to-seven quality rating is reported alongside it. Held-out labelers who contributed no training data preferred the outputs at about the same rate. Other reported results are mixed rather than uniform: closed-domain hallucination fell from 41 to 21 percent and toxic outputs by about a quarter when prompted to be respectful, while on the bias benchmarks the paper reports no significant improvement over GPT-3 at all. [2]

Atlas interpretation: Secondary accounts routinely attach the 85 percent figure to the 1.3B-beats-175B claim. They are different comparisons in different tables, and the smaller model's advantage is read off a quality curve rather than a head-to-head win rate. [2]

The alignment tax, and whose preferences these are

The paper names a cost it calls an alignment tax: performance regressions against GPT-3 on public benchmarks including SQuAD, DROP, HellaSwag and French-to-English translation. Mixing pretraining updates into the reinforcement learning stage reduces these without costing labeler preference, though the paper says this does not completely mitigate them. The framing is strategic rather than incidental: a technique with a high tax would not be adopted, so a low tax is presented as the good news. [2]

A section titled "Who are we aligning to?" answers the question more bluntly than most later work does. The paper states it has aligned to a set of labelers' preferences, shaped by the instructions they were given and the fact that this was a paid job, and to the researchers who wrote those instructions, and to the prompts OpenAI's customers happened to send, and it notes that the customer base itself was seeded from a waitlist that began with OpenAI employees. Inter-labeler agreement was about 73 percent. The paper says plainly that it is not claiming any of these groups are the right source of preferences. [2]

It also records a failure mode that instruction-following creates rather than solves: the models follow the user's instruction even where doing so could cause harm in the real world, and OpenAI's own post observes that training a model to follow instructions may make it more susceptible to misuse. [2][1]

The line to ChatGPT

Ten months later OpenAI introduced ChatGPT as a sibling model to the instruction-following work, trained with the same reinforcement learning method and differing mainly in how the data was collected. The launch post states that the dialogue dataset was mixed with the earlier instruction dataset, reformatted as conversation. The connection is not an inference from shared authorship; it is OpenAI's own description in the second paragraph of the announcement. [4]