Direct Preference Optimization is one of the most significant advances in AI over the last six months. It provides a simpler and more efficient way to align a model's preferences. You can try out in packages like TRL. Direct Preference Optimization (DPO) - A Simplified Explanation: https://medium.com/@joaolages/direct-preference-optimization-dpo-622fc1f18707 Direct Preference Optimization: Your Language Model is Secretly a Reward Model - https://arxiv.org/pdf/2305.18290.pdf DPO Trainer: https://