Adaptive Portfolio Optimization via PPO-HER: A Reinforcement Learning Framework for Non-Stationary Markets

Abstract

We propose PPO-HER, a novel reinforcement learning framework for adaptive portfolio optimization in non-stationary markets, which integrates Proximal Policy Optimization (PPO) with Hindsight Experience Replay (HER) to address sparse rewards and dynamic market conditions. The proposed method reformulates the portfolio optimization problem as a goal-conditioned Markov Decision Process, where the agent learns to reallocate assets by processing spatiotemporal market data through a Transformer-based actor network. The reward function combines logarithmic returns, risk penalties, and sparse bonuses, while HER relabels suboptimal trajectories to improve sample efficiency. Moreover, the architecture employs a TimeSformer for cross-asset attention and a GRU-based critic with spectral normalization to stabilize training. Experimental results demonstrate that PPO-HER outperforms conventional methods in terms of risk-adjusted returns, particularly during regime shifts detected by an auxiliary Changepoint-LSTM module. The framework is implemented using cuDNN-accelerated PyTorch, enabling efficient high-frequency trading with liquidity constraints. Our approach achieves state-of-the-art performance by explicitly modeling non-stationary dependencies and dynamically adjusting reward shaping based on realized volatility.