Code from my M.Sc. thesis in Mathematical Engineering (Politecnico di Torino, 2022): optimizing the energy consumption of a hybrid vehicle using offline model-based reinforcement learning against a vehicle simulator.
Hybrid-vehicle energy management means deciding, at every moment, how to split power demand between the combustion engine and the electric motor/battery. Training a controller online against the real system is impractical, so the policy must be learned offline from a fixed dataset of previously collected driving data — the setting where naive RL methods fail due to distribution shift and value overestimation on out-of-distribution actions.
The main algorithm studied is COMBO (Conservative Offline Model-Based policy Optimization), which combines:
- a learned dynamics model of the environment, used to generate synthetic rollouts that augment the offline dataset, and
- conservative value estimation (in the spirit of CQL) that penalizes Q-values on model-generated state-action pairs, keeping the policy within the support of the data.
| Directory | Contents |
|---|---|
combo/ |
COMBO implementation — the main algorithm of the thesis |
cql/ |
Conservative Q-Learning (model-free offline RL baseline) |
dqn/ |
DQN baseline |
rssm/ |
Recurrent state-space model components (world-model / latent dynamics) |
collector/ |
Utilities for collecting and assembling the offline dataset from the simulator |
- Yu et al., COMBO: Conservative Offline Model-Based Policy Optimization (NeurIPS 2021)
- Kumar et al., Conservative Q-Learning for Offline Reinforcement Learning (NeurIPS 2020)
- Hafner et al., Learning Latent Dynamics for Planning from Pixels (PlaNet/RSSM)