> For the complete documentation index, see [llms.txt](https://docs.buildings.ability.abb/collection/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.buildings.ability.abb/collection/english-v14/getting-started/kunstliche-intelligenz-in-abb-buildingpro/reinforcement-learning.md). # Reinforcement Learning ## What is Reinforcement Learning? **Reinforcement Learning (RL)** is a learning method in which a **agent** through *trial and error* learns to make optimal decisions. The agent tries out various actions in its environment and receives a **reward (Reward)** and adjusts its strategy until it achieves the highest total reward in the long term. In the context of building automation, this means: an RL system independently tests control commands for HVAC, lighting, storage, and more, continuously evaluates their effects on energy consumption and comfort - and improves its control step by step with no manual intervention at all. *** ## Typical fields of application
DomainGoalsExamples
Building automationReduce energy & CO₂ costs, maintain comfort rangeHVAC schedules, peak load smoothing
Energy & smart gridShift load, control storagePV storage dispatch, tariff adjustment
RoboticsDexterous movementsGrasping, navigation, drone flight
Industry 4.0Increase throughput, reduce scrapDynamic pacing of production lines
Games & simulationStrategy developmentAlphaGo, complex 3-D games
*** ## Current challenges of Reinforcement Learning Already in the much-cited blog post from 2018 [“**Deep Reinforcement Learning Doesn't Work Yet**” (February 2018)](https://www.alexirpan.com/2018/02/14/rl-hard.html) it shows where RL fails in practice - and despite all progress, these exact hurdles are still largely unresolved to this day: 1. **High data demand**\ Many RL algorithms need millions of interactions - difficult to implement in real systems. 2. **Sensitive hyperparameter tuning**\ Learning rate, network architecture, and so on are often experimental - small changes determine success or failure. 3. **Reward design & "reward hacking"**\ Misleading rewards can lead to completely undesirable behavior. 4. **Exploration vs. exploitation & local optima**\ Too little exploration ends in suboptimal but easily reachable solutions. 5. **Instability and reproducibility**\ Same code, different random seed -> sometimes completely different results. 6. **Weak generalization**\ Models are often trained on *a* environment; even small changes cause performance to collapse. 7. **Safety and compliance issues**\ Autonomous agents must be limited and auditable in order to avoid risks in critical infrastructure. *** ### Data scarcity - the biggest hurdle in smart buildings As if hyperparameter tuning, reward design, and instability weren't challenge enough, reinforcement learning in building operations faces an additional core problem: **too little raw data**.\ Typical BMS or heat pump systems deliver measurement points **at 5-minute intervals**. With one step per 5 minutes, that adds up to only about **100,000 timestamps** - orders of magnitude away from the **millions of interactions** that classic RL algorithms require for robust policies. *** ### **Approaches to fill the data gap**
ApproachIdeaPros and cons
High-fidelity digital twinA complete simulation model of the building (e.g. EnergyPlus) that delivers exact thermal responses down to wall assemblies, window glazing, occupancy, and weather. RL collects its millions of steps in simulation.+ Physically grounded
+ No live risks
- BIM creation & calibration are time-consuming
- Computing load for long rollouts
Model-based RL / world modelsInstead of heavy physics, one uses a learning-based world model, which extracts a differentiable mini-world from the 100,000 log samples. In it, the agent can experience millions of "dreamed" steps per GPU minute.

+ Extremely data-efficient (often < 10,000 real steps until benefit)
+ Continuous online fine-tuning

+ Immediately deployable if pre-trained on data
- The model only learns areas of the data it has already experienced

Offline RL & imitation learningRL learns exclusively from the existing logs (BCQ, CQL ...), so it first learns to imitate them and improve from there, but never overwrites the policy with actions that do not appear in the data store.+ No twin needed
+ Immediately deployable
- Quality depends directly on log diversity
*** ## How Eliona masters all RL hurdles: world model + simulation steps At Eliona, we follow a **world-based RL approach**: A neural world model learns the dynamics of the building from historical and live data as needed - including weather, occupancy, or window openings. An RL agent then trains its control strategy on it via simulation ("dream steps"), tests thousands of actions per real step, and thus learns in a data-efficient, robust, and safe way.
### **High data demand** **Problem:** Classic RL requires millions of real interactions - unattainable with 5-minute intervals and \~100,000 timestamps per year.\ **Solution:** * A pre-trained world model absorbs the 100,000 historical samples and generates a mini-world from them in which the agent can simulate without limit. * Even after ≈ 2,000 real steps, tests show significant savings. * Offline pre-training + millions of simulation steps make it possible to start with a directly deployable model - completely without a live training phase. ### **Sensitive hyperparameter tuning** **Problem:** Otherwise, learning rates, network architectures, and regularizers require tedious grid search and expert knowledge.\ **Solution:** * Our world-based system has been calibrated on dozens of RL problems. * A robust default parameter set delivers immediate reproducible performance - without any additional tuning. ### **Reward design & "reward hacking"** **Problem:** Incorrectly defined rewards lead to undesirable strategies or exploit behavior.\ **Solution:** * Users define only target ranges via GUI (e.g. 21-23 °C) and metrics that should be minimized or maximized (costs, CO₂, peak load). * Eliona generates a proven reward function adapted to the system structure in the background. * This keeps the reward understandable, safe, and free of perverse incentives. ### **Exploration vs. exploitation & local optima** **Problem:** Exploiting known actions too early blocks the discovery of global optima.\ **Solution:** * Broad exploration of all strategies takes place risk-free in the mini-world. * In reality, only the currently best strategy is applied. * Long-term tests consistently show convergence to optimal operating modes rather than suboptimal plateaus. ### **Instability and reproducibility** **Problem:** Models that vary greatly depending on random seed or training run are unreliable in real operation.\ **Solution:** * HVAC benchmarks document a narrowly limited learning horizon of **1,000-2,000 real steps** - independent of seed. * Results are predictable and ROI estimates reliable. ### **Weak generalization** **Problem:** Models trained only on summer or test data fail when real operating conditions change.\ **Solution:** * Continuous online fine-tuning: Newly arriving 5-minute data updates the world model and thus the control strategy. * The system adapts within **a few days** to new seasons, renovations, or tariff changes. ### **Safety and compliance issues** **Problem:** Autonomous actions without control mechanisms can exceed comfort or safety limits.\ **Solution:** * Through the integrated rule-chain engine, **hard comfort and safety thresholds** can be defined. * If limit values are violated or unexpected actions occur, Eliona automatically switches to a proven fallback controller and triggers alarm escalation. Thanks to this approach, Eliona achieves with RL **double-digit energy and cost savings**, while comfort and safety meet the highest standards - and all without years of data collection or expert fine-tuning. *** ## **Simple commissioning via the RL app** The configuration in Eliona is intentionally designed for minimal effort - no code required: 1. **Select assets**\ In the RL app interface, choose the building parts, systems, or sensor groups that should provide feedback (e.g. room HVAC units, heat pump, window contacts). 2. **Define controllable attributes**\ Mark which actuating variables the agent may control (e.g. supply temperature, fan speed, damper). 3. **Goal definition made easy**\ For each attribute, specify whether it should be kept within a range (e.g. 21-23 °C), minimized (costs, CO₂) or maximized (COP, self-consumption rate) - or whether a dynamic or exact setpoint should apply. 4. **Start & observe**\ The system automatically pulls the latest historical data, builds the world model, and immediately begins offline training. Afterwards, the agent can be switched live at any time - with a click. From that point on, it learns fully automatically: first offline from history, then on the fly with every incoming data point. *** ## Three proven RL applications in building operations Recent studies and field trials show that reinforcement learning already enables double-digit energy savings and comfort gains today - without years of data collection. Below are three solid examples in which RL systems were applied to real buildings, including configuration overview, achieved results, and source references. ### 1. Office building HVAC: 12% energy & 28% comfort improvement **Scenario & goal:**\ A multi-story office building uses Deep RL to optimize heating, ventilation, and air conditioning operation. The goal is to reduce energy consumption by double-digit percentages and minimize comfort violations (temperature fluctuations outside the target range). **RL app configuration (example):** * **Data inputs:** Indoor temperature, outdoor temperature, occupancy rates, window contacts, historical HVAC actuator setpoints * **Controllable actuators:** Supply temperature, fan stages * **Reward definition:** Temperature in \[21 °C, 23 °C] *keep*, energy consumption *minimize.* **Result:**\ In the simulation and the subsequent field test, the RL system achieved **12% less energy consumption** and **28% fewer comfort violations** compared with PID and schedule-based control [ScienceDirect](https://www.sciencedirect.com/science/article/abs/pii/S0360544224001154?utm_source=chatgpt.com). ### 2. Manhattan high-rise: 15.8% less HVAC energy **Scenario & goal:**\ A 32-story office tower in New York City uses a commercial AI-powered RL system to reduce heating and cooling costs. **RL app configuration (example):** * **Data inputs:** Building and zone temperatures, outdoor temperature, occupancy data, real-time tariffs * **Controllable actuators:** Heating/cooling circuits, fan control * **Reward definition:** Minimization of total energy consumption, compliance with comfort bands **Result:**\ The AI reduced HVAC energy consumption by **15,8 %**, saving around 37 t CO₂ and $42,000 in costs per year - all fully automated and without intervention from building management [TIME](https://time.com/7201501/ai-buildings-energy-efficiency/?utm_source=chatgpt.com). ### 3. DFAB House (Empa): Up to 30% energy savings **Scenario & goal:**\ In the research building DFAB House (Empa, Switzerland), an RL agent was trained that jointly optimizes room temperature and bidirectional EV charging. **RL app configuration (example):** * **Data inputs:** Room temperature, outdoor weather data, PV generation, EV SoC, electricity tariff * **Controllable actuators:** Radiator setpoint, charging station power * **Reward definition:** Maximization of comfort score + PV self-consumption, minimization of grid import costs **Result:**\ In the real three-week field test during the heating season, the RL system achieved **up to 30% energy savings** compared with conventional controller strategies, at the same comfort level [arXiv](https://arxiv.org/abs/2103.01886?utm_source=chatgpt.com). ### **Conclusion:** These scenarios show that RL-based applications in a wide variety of building types and operating modes are ready for deployment today. With simple configuration steps in the Eliona RL app, similar results can be achieved with just a few clicks - from office complexes to high-rises to intelligent research buildings. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.buildings.ability.abb/collection/english-v14/getting-started/kunstliche-intelligenz-in-abb-buildingpro/reinforcement-learning.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.