Andrii Bidochko

• Updated: June 13, 2026
• 9 min read

PIRS: Physics-Informed Reward Shaping for SAC-Based Building Energy Management

Direct Answer

PIRS (Physics‑Informed Reward Shaping) replaces ad‑hoc comfort proxies in deep‑reinforcement‑learning (DRL) controllers for building energy management with the ISO 7730 Predicted Mean Vote (PMV) model, delivering a standards‑aligned, interpretable reward signal for Soft Actor‑Critic (SAC) agents. By grounding the comfort term in a physically‑based metric, PIRS enables more reliable trade‑offs between occupant comfort and grid‑aware energy efficiency without redesigning the learning pipeline.

Background: Why This Problem Is Hard

Modern commercial and residential complexes must simultaneously satisfy two competing imperatives: keeping occupants comfortable and minimizing energy consumption (often with carbon‑intensity constraints). Achieving this balance is non‑trivial for three reasons.

• Multi‑objective tension. Comfort is typically expressed as a temperature range, while energy efficiency depends on HVAC set‑points, equipment scheduling, and real‑time electricity prices. Small changes that improve one metric can dramatically degrade the other.
• Reward design is heuristic. In DRL‑based building controllers, the reward function is the sole driver of policy learning. Most existing works inject comfort via simple temperature‑deviation penalties or manually tuned heuristics, which lack a physical basis and are difficult to calibrate across climates, building envelopes, and occupant populations.
• Simulation fidelity vs. compute budget. High‑resolution building physics simulators (e.g., EnergyPlus) are computationally expensive, limiting the number of training steps. When training budgets are tight, poorly shaped rewards can lead to sub‑optimal policies that never discover the true Pareto frontier.

Consequently, practitioners often resort to rule‑based controllers (RBC) that are safe but inefficient, or they spend weeks fine‑tuning reward weights—a process that does not scale to large portfolios of buildings.

What the Researchers Propose

The authors introduce PIRS (Physics‑Informed Reward Shaping), a framework that swaps the conventional temperature‑deviation comfort term with the ISO 7730 PMV formulation. PMV predicts the mean thermal sensation of a large group of occupants based on six environmental variables (air temperature, mean radiant temperature, humidity, air velocity, clothing insulation, and metabolic rate). By embedding this model directly into the reward, PIRS achieves three objectives:

1. Standards alignment. PMV is an internationally recognized metric, ensuring that the learned policy respects established comfort criteria.
2. Interpretability. The comfort component of the reward can be traced back to a physically meaningful quantity, making it easier for engineers to audit and adjust.
3. Minimal pipeline disruption. PIRS does not alter the SAC algorithm, network architecture, or simulation environment; it merely reshapes the scalar reward signal.

Key components of the PIRS framework include:

• Physics module. A lightweight implementation of the PMV equation that consumes real‑time HVAC and indoor‑environment observations.
• Weighted multi‑objective reward. A linear combination of the PMV‑derived comfort term and an energy‑cost term (e.g., electricity price × power consumption).
• SAC agent. The underlying Soft Actor‑Critic learner that optimizes the combined reward through stochastic policy updates.

How It Works in Practice

Conceptual Workflow

The end‑to‑end loop can be described in four stages, each occurring at the simulation’s control timestep (typically 15 minutes).

1. State observation. Sensors (or the simulator) report indoor temperature, humidity, airflow, and external weather conditions.
2. Physics‑informed comfort evaluation. The PMV module ingests the observations along with assumed clothing and activity levels to compute a comfort score ranging from -3 (cold) to +3 (hot). This score is then transformed into a reward component (e.g., negative absolute deviation from neutral PMV = 0).
3. Energy cost calculation. The current electricity price (potentially time‑varying) is multiplied by the HVAC power draw, yielding an economic penalty.
4. Reward synthesis & policy update. The weighted sum of the comfort and cost components forms the scalar reward fed to the SAC algorithm, which updates its actor‑critic networks and selects the next HVAC set‑point.

Interaction Between Components

Because the PMV module is deterministic and inexpensive, it can be evaluated at every timestep without adding noticeable overhead. The SAC agent treats the combined reward exactly as it would a handcrafted one, preserving the algorithm’s exploration‑exploitation balance. Importantly, the weighting between comfort and cost can be tuned once (e.g., via a grid search) and then left unchanged across training runs, eliminating the need for per‑seed manual adjustments.

What Sets PIRS Apart

• It grounds the comfort signal in a physics‑based, industry‑standard model rather than a proxy.
• It retains full compatibility with existing DRL pipelines, meaning organizations can adopt PIRS without rewriting code.
• It provides a transparent, auditable metric that aligns with building‑code compliance checks.

Evaluation & Results

Experimental Setup

The authors benchmarked PIRS using the PIRS paper on arXiv’s CityLearn v2.1.2 environment, a widely‑used open‑source platform that simulates a district of five heterogeneous buildings. The evaluation protocol mirrored the 2022 CityLearn Challenge Phase 1:

• Training horizon: 50 k interaction steps per seed.
• Random seeds: five independent runs to capture variance.
• Baselines:
  • Rule‑Based Controller (RBC) – a deterministic, industry‑standard baseline.
  • Manually engineered reward (E2) – a hand‑tuned temperature‑deviation reward.
  • Energy‑only reward (E3) – optimizes cost without any comfort term.
  • Naïve temperature‑deviation reward (E4) – simple absolute temperature error.

Key Findings

Performance was reported as ratios relative to the RBC across three district‑level KPIs: total cost, carbon emissions, and electricity consumption. The most salient observations include:

• Cost & carbon parity. PIRS matched the manually engineered baseline (E2) on cost and carbon, demonstrating that a physics‑informed comfort term does not sacrifice economic efficiency.
• Load ramping improvement. PIRS reduced peak‑to‑valley load ramping by a factor of 1.78× compared to RBC, outperforming all non‑physics baselines (which hovered around 2.4×). This indicates smoother demand profiles, beneficial for grid stability.
• Peak demand reduction. Daily peak electricity demand under PIRS was consistently lower than under E4 and E3, highlighting the advantage of a balanced comfort‑energy objective.
• Training efficiency. Even with the modest 50 k step budget, all DRL policies—including PIRS—exceeded RBC, suggesting that the physics‑informed reward accelerates convergence toward useful policies.

Importantly, the authors present the results with an honest caveat: the DRL agents still lag behind RBC in absolute terms when training budgets are severely limited, positioning PIRS as a foundation for future, higher‑budget experiments rather than a silver‑bullet replacement.

Why This Matters for AI Systems and Agents

For practitioners building AI‑driven energy management solutions, PIRS offers a concrete pathway to embed domain expertise directly into the reward signal—a practice that has traditionally been left to trial‑and‑error. The implications are threefold.

• Accelerated development cycles. By using a standards‑based comfort metric, engineers can skip lengthy calibration phases, freeing resources for scaling to larger building portfolios.
• Regulatory compliance made automatic. Since PMV is referenced in many building codes, a PIRS‑trained policy inherently respects legal comfort thresholds, reducing the risk of non‑compliance penalties.
• Interoperability with existing AI platforms. The modular nature of the physics module means it can be plugged into any SAC‑compatible framework, including those offered by the UBOS platform overview. This aligns with the growing trend of composable AI services where reward shaping is a reusable micro‑service.

Moreover, the transparent reward formulation simplifies post‑deployment monitoring. Facility managers can trace a policy’s decision back to a PMV value, making it easier to explain actions to occupants or auditors—a key requirement for trust in AI‑enabled building automation.

What Comes Next

While PIRS marks a significant step forward, several avenues remain open for research and productization.

Current Limitations

• Static occupant assumptions. The PMV calculation requires inputs for clothing insulation and metabolic rate, which were held constant in the experiments. Real‑world deployments would need adaptive models that infer these parameters from occupancy sensors.
• Training budget constraints. The study used 50 k steps; larger budgets could reveal whether PIRS scales to outperform RBC in absolute terms.
• Single‑objective weighting. The linear combination of comfort and cost may not capture more nuanced preferences (e.g., time‑of‑day comfort weighting).

Future Research Directions

• Integrate Workflow automation studio to dynamically adjust PMV inputs based on real‑time occupancy analytics.
• Explore multi‑agent extensions where each building in a district runs its own SAC agent but shares a global comfort‑energy budget, leveraging the Enterprise AI platform by UBOS for coordinated learning.
• Combine PIRS with model‑based RL techniques to reduce sample complexity, potentially achieving RBC‑level performance with fewer interactions.
• Validate the framework on hardware‑in‑the‑loop testbeds, bridging the gap between simulation and field deployment.

Potential Applications

Beyond traditional HVAC control, the physics‑informed reward concept can be transplanted to other domains where comfort or safety is governed by well‑established physical models:

• Thermal management of data centers, using server temperature and airflow physics.
• Smart lighting systems that respect visual comfort standards (e.g., CIE glare metrics).
• Electric vehicle charging stations that balance grid load with battery health models.

Organizations looking to prototype such solutions can leverage the AI marketing agents as a template for building reusable, physics‑aware reward services across verticals.

Conclusion

PIRS demonstrates that embedding a rigorously validated comfort model into the reward function of a SAC‑based DRL controller yields interpretable, standards‑compliant policies without sacrificing energy efficiency. The framework’s simplicity—replacing a heuristic term with a PMV calculation—makes it attractive for both academic researchers and industry practitioners seeking to accelerate AI‑driven building automation. While further work is needed to address occupant variability and larger training budgets, PIRS establishes a solid foundation for physics‑informed reward design across a spectrum of AI‑controlled physical systems.

References

• Zaregarizi, S., & Yavari, K. (2026). PIRS: Physics-Informed Reward Shaping for SAC-Based Building Energy Management. arXiv preprint arXiv:2605.28232.
• ISO 7730:2005. Ergonomics of the thermal environment – Analytical determination and interpretation of thermal comfort using calculation of the PMV and PPD indices.
• CityLearn v2.1.2 – Open-source platform for district‑level energy management research.

Illustration

The diagram below visualizes the PIRS workflow, highlighting the flow from sensor observations through the PMV module to the combined reward that guides the SAC agent.

Andrii Bidochko

CTO UBOS

Andrii Bidochko is an AI entrepreneur and researcher focused on AI agents, reinforcement learning, and autonomous systems. He writes about the technologies shaping the future of machine intelligence, from frontier models and agent architectures to real-world AI applications.