Deliberative alignment: reasoning enables safer language models

We introduce deliberative alignment, a training paradigm that directly teaches reasoning LLMs the text of human-written and interpretable safety specifications, and trains them to reason explicitly about these specifications before answering. We used deliberative alignment to align OpenAI’s o-series models, enabling them to use chain-of-thought (CoT) reasoning to reflect on user prompts, identify relevant text from OpenAI’s internal policies, and draft safer responses. Our approach achieves highly precise adherence to OpenAI’s safety policies, and without requiring human-labeled CoTs or answers. We find that o1 dramatically outperforms GPT-4o and other state-of-the art LLMs across a range of internal and external safety benchmarks, and saturates performance on many challenging datasets. We believe this presents an exciting new path to improve safety, and we find this to be an encouraging example of how improvements in capabilities can be leveraged to improve safety as well.

Despite extensive safety training, modern LLMs still comply with malicious prompts, overrefuse benign queries, and fall victim to jailbreak attacks. One cause of these failures is that models must respond instantly, without being given sufficient time to reason through complex and borderline safety scenarios. Another issue is that LLMs must infer desired behavior indirectly from large sets of labeled examples, rather than directly learning the underlying safety standards in natural language. This forces models to have to reverse engineer the ideal behavior from examples and leads to poor data efficiency and decision boundaries. 

Deliberative alignment overcomes both of these issues. It is the first approach to directly teach a model the text of its safety specifications and train the model to deliberate over these specifications at inference time. This results in safer responses that are appropriately calibrated to a given context.

In comparison, prior alignment approaches, including Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning through AI Feedback, e.g. Constitutional AI (CAI), use safety specifications only to generate training labels. The specifications themselves are not provided to the model. Deliberative alignment is also unique in its ability to do complex reasoning over safety specifications at inference time. Other strategies that refine responses at inference time, like Self-REFINE, restrict the model to predefined reasoning paths and do not involve direct reasoning over learned safety specifications (since these were not taught).

Figure 2: Comparison of deliberative alignment and representative methods of existing alignment approaches. a) Training data generation: Even though RLAIF methods like CAI use safety specifications to generate training labels, only the labels themselves are used in training. Knowledge of the specifications themselves is thereby lost to the model. Whereas in deliberative alignment, the chain-of-thought, which contains both the content of the specifications and how to reason over them, is supervised in addition to other model output during SFT.  The trained model can thereby retrieve relevant policies at inference time and apply them to generate aligned responses. b) Inference time behavior: In RLHF and CAI, there is no reasoning during inference time. In Self-REFINE, reasoning occurs through structured few-shot prompting. In deliberative alignment, reasoning occurs automatically via chain-of-thought, including reasoning over learned safety specifications.

Deliberate alignment training uses a combination of process- and outcome-based supervision:

• We first train an o-style model for helpfulness, without any safety-relevant data.  

• We then build a dataset of (prompt, completion) pairs where the CoTs in the completions reference the specifications. We do this by inserting the relevant safety specification text for each conversation in the system prompt, generating model completions, and then removing the system prompts from the data.

• We perform incremental supervised fine-tuning (SFT) on this dataset, providing the model with a strong prior for safe reasoning. Through SFT, the model learns both the content of our safety specifications and how to reason over them to generate aligned responses.

• We then use reinforcement learning (RL) to train the model to use its CoT more effectively. To do so, we employ a reward model with access to our safety policies to provide additional reward signal.

In our training procedure, we automatically generate training data from safety specifications and safety-categorized prompts, without requiring human-labeled completions. Deliberative alignment’s synthetic data generation pipeline thus offers a scalable approach to alignment, addressing a major challenge of standard LLM safety training—its heavy dependence on human-labeled data.

Figure 3: Illustration of overall methodology. Key processes are shown along the left side of the figure. During SFT data generation, we construct a dataset of {prompt, CoT, output} tuples where the CoTs refer to relevant policies. We collect these by prompting a reasoning model G_base with safety prompts along with safety specifications (spec) that are tailored to safety categories (cat). After filtering with a policy-aware reward model G_RM, this data is then used for SFT training to teach the model to reason about the spec in its CoT. In the RL training stage, we provide reward signal using that same reward model G_RM with access to the spec. Our resulting model G_spec is aligned with the safety specifications.

We compare the safety of o1 to GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro across a range of internal and external safety benchmarks (e.g., jailbreaks, content policy refusals). The o1 model saturates many of our hardest safety evaluations and achieves a Pareto improvement on both under- and overrefusals. This means that we are simultaneously better at avoiding harmful outputs while being more permissive with benign prompts. We also find that safety training with deliberative alignment enables strong generalization to out-of-distribution safety scenarios.

Figure 4: Main safety results. The o1 models advance the Pareto frontier of refusing to answer malicious jailbreak prompts (from StrongREJECT) and not over-refusing benign prompts (from XSTest), compared to GPT-4o and other state-of-the-art LLMs. The error bars represent standard deviation estimated over 1,000 bootstrap trials.

Advances in LLM capabilities, such as those demonstrated by o1 and o3, come with substantial risks. As models gain more intelligence and autonomy, the scale of potential harm that could be caused by AIs through misalignment or misuse increases dramatically. This underscores the urgent need for ongoing research in AI safety. We are actively investing in this space, particularly in areas like monitoring chain-of-thoughts for deception, to ensure that as AI systems become more capable, they remain aligned with human values.

Deliberative alignment represents the latest advancement in our efforts, and we are highly encouraged by its results. The approach is effective at improving adherence to specifications and robustness to jailbreaks, and allows us to specify the boundary between compliance, refusal, and safe completion in finer detail than was possible before. With its application to o-series models, we are encouraged by how advances in model capabilities can be harnessed to improve AI safety.