Generalized Deep Reinforcement Learning for Trading
DOI:
https://doi.org/10.47611/jsrhs.v12i1.4316Keywords:
Artificial Intelligence, Deep Reinforcement Learning, TradingAbstract
This paper proposes generalized deep reinforcement learning with multivariate state space, discrete rewards, and adaptive synchronization for trading any stock held in the S&P 500. Specifically, the proposed trading model observes the daily historical data of a stock held in the S&P 500 and multiple market-indicating securities (SPY, IEF, EUR=X, GSG), selects a trading action, and observes a discrete reward that is based on the correctness of the selected action and independent of the volatility of stocks. The proposed trading model’s reward-maximizing behavior is optimized by using a standard deep q-network (DQN) with adaptive synchronization that stabilizes and enables to track learning performance on generalizing new experiences from each stock. The proposed trading model was trained on the top 50 holdings of the S&P 500 and tested on the top 100 holdings of the S&P 500 starting from 2006 to 2022. Experimental results suggest that the proposed trading model significantly outperforms the 100% long-strategy benchmark in terms of annualized return, Sharpe ratio, and maximum drawdown.
Downloads
References or Bibliography
A. W. Li and G. S. Bastos, “Stock Market Forecasting Using Deep Learning and Technical Analysis: A Systematic Review,” IEEE Access, vol. 8, pp. 185232–185242, 2020, doi: 10.1109/ACCESS.2020.3030226.
B. Lim, S. Zohren, and S. Roberts, “Enhancing Time-Series Momentum Strategies Using Deep Neural Networks,” The Journal of Financial Data Science, 2019, doi: 10.3905/jfds.2019.1.015.
S. Siami-Namini and A. S. Namin, “Forecasting Economics and Financial Time Series: ARIMA vs. LSTM.” arXiv, 2018. doi: 10.48550/ARXIV.1803.06386.
Q. Guo, S. Lei, Q. Ye, and Z. Fang, “MRC-LSTM: A Hybrid Approach of Multi-scale Residual CNN and LSTM to Predict Bitcoin Price,” in 2021 International Joint Conference on Neural Networks (IJCNN), 2021, pp. 1–8. doi: 10.1109/IJCNN52387.2021.9534453.
V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, Feb. 2015, doi: 10.1038/nature14236.
V. Mnih et al., “Playing Atari with Deep Reinforcement Learning.” arXiv, 2013. doi: 10.48550/ARXIV.1312.5602.
A. Millea, “Deep Reinforcement Learning for Trading—A Critical Survey,” Data, vol. 6, no. 11, 2021, doi: 10.3390/data6110119.
Z. Zhang, S. Zohren, and S. Roberts, “Deep Reinforcement Learning for Trading,” The Journal of Financial Data Science, vol. 2, no. 2, pp. 25–40, 2020, doi: 10.3905/jfds.2020.1.030.
Y. Li, W. Zheng, and Z. Zheng, “Deep Robust Reinforcement Learning for Practical Algorithmic Trading,” IEEE Access, vol. 7, pp. 108014–108022, 2019, doi: 10.1109/ACCESS.2019.2932789.
Y. Deng, F. Bao, Y. Kong, Z. Ren, and Q. Dai, “Deep Direct Reinforcement Learning for Financial Signal Representation and Trading,” IEEE Transactions on Neural Networks and Learning Systems, vol. 28, no. 3, pp. 653–664, 2017, doi: 10.1109/TNNLS.2016.2522401.
S. Carta, A. Corriga, A. Ferreira, A. S. Podda, and D. R. Recupero, “A multi-layer and multi-ensemble stock trader using deep learning and deep reinforcement learning,” Applied Intelligence, vol. 51, no. 2, pp. 889–905, Feb. 2021, doi: 10.1007/s10489-020-01839-5.
F. Rundo, “Deep LSTM with Reinforcement Learning Layer for Financial Trend Prediction in FX High Frequency Trading Systems,” Applied Sciences, vol. 9, no. 20, 2019, doi: 10.3390/app9204460.
X.-Y. Liu, Z. Xiong, S. Zhong, H. Yang, and A. Walid, “Practical Deep Reinforcement Learning Approach for Stock Trading.” arXiv, 2018. doi: 10.48550/ARXIV.1811.07522.
Y. Li, “Deep Reinforcement Learning: An Overview.” arXiv, 2017. doi: 10.48550/ARXIV.1701.07274.
L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: A survey,” Journal of artificial intelligence research, vol. 4, pp. 237–285, 1996, doi: https://doi.org/10.1613/jair.301.
V. Mnih et al., “Asynchronous Methods for Deep Reinforcement Learning,” 2016, doi: 10.48550/ARXIV.1602.01783.
L. Menkhoff, “The use of technical analysis by fund managers: International evidence,” Journal of Banking & Finance, vol. 34, no. 11, pp. 2573–2586, 2010, doi: https://doi.org/10.1016/j.jbankfin.2010.04.014.
J. W. Lee, J. Park, J. O, J. Lee, and E. Hong, “A Multiagent Approach to $Q$-Learning for Daily Stock Trading,” IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, vol. 37, no. 6, pp. 864–877, 2007, doi: 10.1109/TSMCA.2007.904825.
S. A. Badran and M. Rezghi, “An adaptive synchronization approach for weights of deep reinforcement learning.” arXiv, 2020. doi: 10.48550/ARXIV.2008.06973.
Published
How to Cite
Issue
Section
Copyright (c) 2023 Junyoung Sim; Benjamin Kirk
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright holder(s) granted JSR a perpetual, non-exclusive license to distriute & display this article.
