OccuReward: LLM-Guided Occupant-Centric Reward Shaping for Demographic Equity in Grid-Interactive Buildings

Direct Answer

OccuReward introduces a novel framework that uses large language models (LLMs) to iteratively shape reward functions for deep‑reinforcement‑learning (DRL) controllers in grid‑interactive buildings, explicitly targeting demographic equity in occupant comfort. By embedding a Comfort Equity Index (CEI) into the reward loop, the approach demonstrates measurable improvements in satisfaction for historically underserved groups while also cutting energy costs.

Background: Why This Problem Is Hard

Smart building control systems have increasingly turned to DRL for optimizing HVAC operation, lighting, and load shifting. These agents excel at minimizing energy consumption, but they typically treat occupants as a monolithic “comfort” signal derived from temperature set‑points or generic satisfaction scores. In reality, comfort perception varies dramatically across age, gender, health status, and cultural background. Existing DRL pipelines lack a principled way to encode such heterogeneity, leading to hidden biases where certain demographic groups—often the elderly or those with health sensitivities—receive sub‑optimal thermal conditions.

Traditional mitigation strategies rely on manually crafted reward terms or post‑hoc fairness constraints. Manual reward engineering is labor‑intensive, brittle, and prone to overlooking nuanced comfort drivers. Moreover, fairness constraints added after training can conflict with the agent’s primary objective, causing instability or degraded performance. The core challenge, therefore, is to integrate equity considerations directly into the reward design process in a scalable, data‑driven manner.

What the Researchers Propose

The authors present OccuReward, a three‑stage framework that couples an LLM (accessed via the Gemini API) with a DRL controller to produce equity‑aware reward functions. The key ingredients are:

• Comfort Equity Index (CEI): A composite feedback signal that quantifies the disparity in comfort satisfaction across predefined occupant cohorts.
• Iterative LLM‑guided reward shaping: The LLM generates logical reward components and weight suggestions based on CEI trends, then a human‑in‑the‑loop validates and refines them across multiple rounds.
• DRL agent integration: A Soft Actor‑Critic (SAC) agent consumes the refined reward function while interacting with the CityLearn v2 simulation environment.

By treating the LLM as a “reward architect” rather than a per‑step inference engine, the system can embed high‑level policy insights without incurring runtime latency.

How It Works in Practice

The operational workflow of OccuReward can be broken down into four logical steps, illustrated in the diagram below ({{IMAGE}}):

1. Data Collection & Cohort Definition: Occupant feedback from the ASHRAE Global Thermal Comfort Database II is clustered into four representative profiles—Young Males, Mid‑aged Females, Health‑Sensitive occupants, and Elderly Females.
2. Initial Reward Generation: The LLM receives a prompt describing the DRL objective (energy cost minimization) and the four cohorts. It outputs a baseline reward formula that includes terms for energy usage, temperature deviation, and a generic comfort penalty.
3. Equity Evaluation & CEI Computation: After a training episode, the system calculates CEI by measuring the variance in satisfaction scores across the cohorts. High CEI values flag inequitable outcomes.
4. Iterative Refinement: The CEI and episode statistics are fed back to the LLM, which suggests new reward components (e.g., cohort‑specific temperature weighting) and adjusts existing weights. Human reviewers approve the changes before the next training round.

What sets this approach apart is the closed‑loop feedback between quantitative equity metrics and natural‑language reward synthesis. The LLM can reason about “why” a particular group is underserved and propose targeted corrective terms, while the DRL agent continuously adapts its policy to the evolving objective.

Evaluation & Results

To validate the framework, the researchers conducted 15 independent runs of a SAC agent in the CityLearn v2 environment, each spanning three refinement rounds. The key evaluation dimensions were:

• Comfort Satisfaction per Cohort: Measured as the proportion of time a occupant’s thermal condition fell within the ASHRAE‑defined comfort zone.
• Overall Energy Cost: Calculated from the simulated electricity bill under a time‑varying tariff.
• Comfort Equity Index (CEI): The standard deviation of satisfaction scores across cohorts.

Findings revealed a stark baseline disparity: elderly female occupants consistently reported the lowest satisfaction, lagging behind other groups by more than 30 %. After three rounds of LLM‑guided shaping, the system achieved:

• +17.6 % satisfaction for Young Males
• +28.2 % for Mid‑aged Females
• +53.8 % for Health‑Sensitive occupants
• +567 % for Elderly Females (the most dramatic uplift)
• A 3.2 % reduction in total energy cost, demonstrating that equity improvements did not come at the expense of efficiency.

Crucially, the CEI dropped by over 40 % across rounds, indicating a more balanced comfort distribution. These results underscore that reward‑level interventions, when informed by LLM reasoning, can simultaneously advance fairness and sustainability goals.

Why This Matters for AI Systems and Agents

For practitioners building AI‑driven building management systems, OccuReward offers a concrete pathway to embed fairness directly into the learning objective, rather than treating it as an afterthought. The framework demonstrates that:

• Reward design can be automated: Leveraging LLMs reduces the manual effort required to craft nuanced, demographic‑aware reward terms.
• Equity metrics become actionable: The CEI provides a clear, quantitative signal that can trigger adaptive policy updates.
• Energy efficiency and fairness are not mutually exclusive: The observed cost savings show that equitable control can coexist with, and even enhance, operational performance.

These insights are directly applicable to the development of UBOS platform overview, where modular reward components can be swapped in via a low‑code interface, and to the deployment of AI marketing agents that must respect diverse user preferences while optimizing campaign spend.

What Comes Next

While the study marks a significant step forward, several open challenges remain:

• Scalability to Real‑World Deployments: Translating simulation‑based gains to live building fleets will require robust sensor integration and privacy‑preserving data pipelines.
• Dynamic Demographic Shifts: Occupant populations evolve over time; future work should explore continual learning mechanisms that update reward structures without retraining from scratch.
• Broader Fairness Dimensions: Beyond age and gender, factors such as cultural background, disability, and occupancy patterns merit inclusion in the CEI.
• Explainability for Facility Operators: Operators need transparent rationales for why a controller is prioritizing certain comfort adjustments; integrating LLM‑generated explanations could bridge this gap.

Addressing these avenues will likely involve tighter coupling between LLMs, reinforcement learners, and building‑automation standards (e.g., BACnet, OpenADR). Researchers are also encouraged to open‑source the reward‑shaping pipeline, enabling community‑driven extensions and cross‑domain validation.

For readers interested in the full technical details, the original manuscript is available on arXiv. The authors encourage collaboration with industry partners to pilot the framework in commercial and residential settings.