Improving Model Safety Behavior with Rule-Based Rewards

Our research shows that Rule-Based Rewards (RBRs) significantly enhance the safety of our AI systems, making them safer and more reliable for people and developers to use every day. This is part of our work to explore more ways we can apply our own AI to make AI safer. 

Traditionally, fine-tuning language models using reinforcement learning from human feedback (RLHF) has been the go-to method for ensuring they follow instructions accurately. OpenAI has been at the forefront of developing these alignment methods to create smarter and safer AI models.

To ensure AI systems behave safely and align with human values, we define desired behaviors and collect human feedback to train a "reward model." This model guides the AI by signaling desirable actions. However, collecting this human feedback for routine and repetitive tasks is often inefficient. Additionally, if our safety policies change, the feedback we've already collected might become outdated, requiring new data.

Thus, we introduce Rule-Based Rewards (RBRs) as a key component of OpenAI’s safety stack to align model behavior with desired safe behavior. Unlike human feedback, RBRs uses clear, simple, and step-by-step rules to evaluate if the model's outputs meet safety standards. When plugged into the standard RLHF pipeline, it helps maintain a good balance between being helpful while preventing harm, to ensure the model behaves safely and effectively without the inefficiencies of recurrent human inputs. We have used RBRs as part of our safety stack since our GPT-4 launch, including GPT-4o mini, and we plan to implement it in our models moving forward.

The process of implementing RBRs involves defining a set of propositions—simple statements about the desired or undesired aspects of the model’s responses, such as “being judgmental”, “containing disallowed content”, “referring to safety policies”, “disclaimer” and more. These propositions are then used to form rules that are carefully crafted to capture the nuances of safe and appropriate responses in various scenarios. For instance, a refusal (e.g. “Sorry I can’t help you with that.”) is a desired model response when facing unsafe requests –  the associated rules would state that the refusal “should contain a brief apology" and that it "should state an inability to comply". 

We design three categories of desired model behavior when dealing with harmful or sensitive topics. Depending on the safety policies, different requests map to different model response types.

The grader, a fixed language model, scores the responses based on how well they adhere to these rules, allowing the RBR approach to flexibly adapt to new rules and safety policies. The RBR uses these scores to fit a linear model with weight parameters learned from a small dataset of prompts with known ideal response type, as well as corresponding desired and undesired completions. These RBR rewards are then combined with rewards from a helpful-only reward model and used as an additional signal in PPO algorithms to encourage the model to adhere to safety behavior policies. The method allows us to provide fine-grained control over the model’s behavior, ensuring that it not only avoids harmful content but does so in a manner that is both respectful and helpful.

In our experiments, RBR-trained models demonstrated safety performance comparable to those trained with human feedback. They also reduced instances of incorrectly refusing safe requests (“overrefuse”) without affecting evaluation metrics on common capability benchmarks. RBRs also significantly reduce the need for extensive human data, making the training process faster and more cost-effective. In addition, as model capabilities and safety guidelines evolve, RBRs can be quickly updated by modifying or adding new rules, without the need for extensive retraining.

We are evaluating our model safety behavior in a framework where we can easily track the trade-off between helpfulness and harmfulness. On one hand, it's easy to be safe if the model refuses everything, but the utility of the model is zero. On the other hand, we don't want to build a model that optimizes for maximum utility, but is unsafe or harmful. An optimally aligned model should thread this needle between helpfulness and harmfulness.

While RBRs work well for tasks with clear, straightforward rules, they can be tricky to apply to more subjective tasks like writing a high-quality essay. However, RBRs can be combined with human feedback to balance these challenges. For instance, RBRs can enforce specific guidelines (like "Don't use slang" or rules in the Model Spec), while human feedback can help with more nuanced aspects (like overall coherence). The strength of the RBR is optimized to correctly enforce safety preferences but not impact the final reward score more than needed - in this way the RLHF reward model can still provide strong signal on e.g. writing style.

Ethical Considerations: Shifting safety checks from humans to AI can reduce human oversight of AI safety and might amplify potential biases in the models if biased models are used to provide RBR rewards. To address this, researchers should carefully design RBRs to ensure fairness and accuracy, and consider using a combination of RBRs and human feedback to minimize risks.

Here we introduced a novel preference modeling approach using Rule-Based Rewards (RBRs) for safety training of language models. Our method is cost- and time-efficient, requiring minimal human data, and is easy to update if the desired model behavior changes, while maintaining a balance between safety and usefulness.

RBRs are not limited to safety training. They can be adapted for various tasks where explicit rules can define desired behaviors, such as tailoring the personality or format of model responses for a specific application. Looking ahead, we plan to run more extensive ablation studies for more comprehensive understanding of different RBR components, the use of synthetic data for rule development, and human evaluations to validate the effectiveness of RBRs in diverse applications including other domains beyond safety.

We invite researchers and practitioners to explore the potential of RBRs in their own work. By sharing insights and collaborating on best practices, we can collectively advance the field of safe and aligned AI, ensuring that these powerful tools better serve people.