Andrii Bidochko

• Updated: August 13, 2026
• 3 min read

LEMUR: Learning to Align with Multi‑Objective Reinforcement Learning from Preference Feedback

LEMUR: Learning to Align with Multi‑Objective Reinforcement Learning from Preference Feedback

Reinforcement Learning (RL) has traditionally relied on a single, well‑defined scalar reward function. Real‑world decision‑making, however, often involves competing objectives—such as performance versus energy efficiency—where a single scalar reward is insufficient or impossible to specify. While Multi‑Objective Reinforcement Learning (MORL) addresses these trade‑offs by modeling rewards as vectors, existing MORL methods still assume access to precise reward functions for each objective, inheriting the same limitations as single‑objective RL.

Preference‑based RL (PbRL) has demonstrated that agents can learn effective policies from human feedback without any pre‑defined reward function, but its research has been limited to single‑objective settings. LEMUR (Learning to Align with Multi‑Objective Reinforcement Learning from Preference Feedback) bridges this gap. It enables agents to learn optimal multi‑objective policies by jointly learning objective‑specific reward models from the preferences of multiple human annotators.

Key Contributions

• Introduces a novel framework that combines MORL with preference‑based learning, eliminating the need for hand‑crafted reward functions.
• Jointly learns a set of reward models—one per objective—directly from human preference data.
• Demonstrates superior performance over state‑of‑the‑art baselines on a suite of benchmark multi‑objective tasks.
• Provides a scalable pipeline for real‑world applications where objectives are ambiguous or costly to quantify.

Method Overview

LEMUR operates in an iterative loop:

1. The agent collects trajectories under its current policy.
2. Human annotators compare pairs of trajectories, expressing preferences that reflect trade‑offs between objectives.
3. A set of reward models—one per objective—is updated using the preference data via a Bayesian inference process.
4. The updated reward models guide a multi‑objective policy optimization step (e.g., Pareto‑frontier learning or scalarization techniques).
5. The loop repeats until convergence or a predefined budget of human feedback is exhausted.

This approach ensures that the learned policies align with human values across all relevant objectives.

Experimental Results

We evaluated LEMUR on classic MORL benchmarks (e.g., Multi‑Objective CartPole, Robot Navigation with Energy Constraints) and on newly curated tasks that require nuanced trade‑offs. Across all domains, LEMUR consistently outperformed:

• Standard MORL with handcrafted reward vectors.
• Preference‑based RL extended naively to multi‑objective settings.
• Weighted‑sum baselines using static weightings.

Metrics such as hypervolume improvement and user‑satisfaction scores confirmed the advantage of learning reward models directly from human preferences.

Implications and Future Work

LEMUR opens new avenues for deploying RL in safety‑critical and ethically sensitive domains—autonomous driving, energy‑aware robotics, and personalized recommendation systems—where multiple objectives must be balanced without explicit reward engineering. Future research will explore:

• Scalable preference collection via crowdsourcing platforms.
• Integration with large‑language‑model based reward modeling.
• Robustness to noisy or contradictory human feedback.

For a deeper dive into the methodology, code, and full experimental suite, visit the arXiv paper and explore related resources on ubos.tech.

Author: UBOS Team

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.