PGSGAN: Policy Gradient Stock GAN
Abstract
We introduce a novel generative adversarial network (GAN) designed to generate realistic trading orders for financial markets. Past models of GANs for creating synthesized trading orders have always been focused on continuous spaces because their architecture has constraints coming from the learning algorithm. Contrary to this, actual orders are placed on discontinuous space, which comes from the actual trading rules such as a minimum unit price for orders. In this study, therefore, we adopt a different approach supporting the actual trading rules and placing generated orders to discontinuous state space. The modification led to the inability to apply the standard GAN's learning algorithm, requiring us to use a policy gradient, a solution commonly used in the realm of reinforcement learning, as our learning strategy. Our experiments handle massive amounts of order data over half a year. Our experimental results indicated that the proposed model surpassed previous models in terms of the distribution pattern of the generated orders. A noteworthy advantage of integrating the policy gradient into our model is that it provides the ability to monitor the GAN's learning progress by analyzing the entropy of the generated policy.References
J. Li, X.Wang, Y. Lin, A. Sinha, and M.Wellman, “Generating Realistic Stock Market Order Streams,” AAAI Conference on Artificial Intelligence, vol. 34, no. 01, pp. 727–734, 2020.
Y. Naritomi and T. Adachi, “Data Augmentation of High Frequency Financial Data Using Generative Adversarial Network,” in 2020 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT). IEEE, 2020, pp. 641–648.
M. HIRANO, H. SAKAJI, and K. IZUMI, “Policy Gradient Stock GAN for Realistic Discrete Order Data Generation in Financial Markets,” in 14th IIAI International Congress on Advanced Applied Informatics. IEEE, 2023, pp. 361–368.
I. Maeda, D. deGraw, M. Kitano, H. Matsushima, H. Sakaji, K. Izumi, and A. Kato, “Deep reinforcement learning in agent based financial market simulation,” Journal of Risk and Financial Management, vol. 13, no. 4, p. 71, 2020.
S. M. Edmonds and Bruce, “Towards Good Social Science,” Journal of Artificial Societies and Social Simulation, vol. 8, no. 4, 2005.
J. D. Farmer and D. Foley, “The economy needs agent-based modelling,” Nature, vol. 460, no. 7256, pp. 685–686, 2009.
S. Battiston, J. D. Farmer, A. Flache, D. Garlaschelli, A. G. Haldane, H. Heesterbeek, C. Hommes, C. Jaeger, R. May, and M. Scheffer, “Complexity theory and financial regulation: Economic policy needs interdisciplinary network analysis and behavioral modeling,” Science, vol. 351, no. 6275, pp. 818–819, 2016.
T. Mizuta, “An Agent-based Model for Designing a Financial Market that Works Well,” 2019, http://arxiv.org/abs/1906.06000.
T. Mizuta, S. Kosugi, T. Kusumoto, W. Matsumoto, K. Izumi, I. Yagi, and S. Yoshimura, “Effects of Price Regulations and Dark Pools on Financial Market Stability: An Investigation by Multiagent Simulations,” Intelligent Systems in Accounting, Finance and Management, vol. 23, no. 1-2, pp. 97–120, 2016.
M. Hirano, K. Izumi, T. Shimada, H. Matsushima, and H. Sakaji, “Impact Analysis of Financial Regulation on Multi-Asset Markets Using Artificial Market Simulations,” Journal of Risk and Financial Management, vol. 13, no. 4, p. 75, 2020.
X. Zhou, Z. Pan, G. Hu, S. Tang, and C. Zhao, “Stock Market Prediction on High-Frequency Data Using Generative Adversarial Nets,” Mathematical Problems in Engineering, vol. 2018, 2018.
K. Zhang, G. Zhong, J. Dong, S.Wang, and Y.Wang, “Stock Market Prediction Based on Generative Adversarial Network,” vol. 147, pp. 400–406, 2019.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Nets,” Advances in Neural Information Processing Systems, pp. 2672–2680, 2014.
M. Mirza and S. Osindero, “Conditional Generative Adversarial Nets,” arXiv, vol. 1411, no. 1784, pp. 1–7, 2014, http://arxiv.org/abs/1411.1784.
A. Radford, L. Metz, and S. Chintala, “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks,” 2015, http://arxiv.org/abs/1511.06434.
X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in IEEE international conference on computer vision, 2017, pp. 2794–2802.
S. Nowozin, B. Cseke, and R. Tomioka, “f-gan: Training generative neural samplers using variational divergence minimization,” in 30th International Conference on Neural Information Processing Systems, 2016, pp. 271–279.
E. Denton, S. Chintala, A. Szlam, and R. Fergus, “Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks,” Advances in Neural Information Processing Systems, vol. 28, pp. 1486–1494, 2015.
A. B. L. Larsen, S. K. Sønderby, H. Larochelle, and O. Winther, “Autoencoding beyond pixels using a learned similarity metric,” in International conference on machine learning, 2016, pp. 1558–1566.
P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in IEEE conference on computer vision and pattern recognition, 2017, pp. 1125–1134.
C. Chu, A. Zhmoginov, and M. Sandler, “CycleGAN, a Master of Steganography,” 2017, http://arxiv.org/abs/1712.02950.
T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4401–4410.
T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of GANs for improved quality, stability, and variation,” in 6th International Conference on Learning Representations, 2018.
L. Yu, W. Zhang, J. Wang, and Y. Yu, “SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient,” Thirty-First AAAI Conference on Artificial Intelligence, 2017.
J. Donahue, P. Kr¨ahenb¨uhl, and T. Darrell, “Adversarial Feature Learning,” 2016, https://arxiv.org/abs/1605.09782.
T. Schlegl, P. Seeb¨ock, S. M. Waldstein, U. Schmidt-Erfurth, and G. Langs, “Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery,” Lecture Notes in Computer Science, vol. 10265, pp. 146–147, 2017.
H. Zenati, C. S. Foo, B. Lecouat, G. Manek, and V. R. Chandrasekhar, “Efficient GAN-Based Anomaly Detection,” 2018, http://arxiv.org/abs/1802.06222.
D. Li, D. Chen, J. Goh, and S.-k. Ng, “Anomaly detection with generative adversarial networks for multivariate time series,” 2018, https://arxiv.org/abs/1809.04758.
M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein GAN,” 2017, https://arxiv.org/abs/1701.07875.
M. Arjovsky and L. Bottou, “Towards Principled Methods for Training Generative Adversarial Networks,” 2017, http://arxiv.org/abs/1701.04862.
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, “Improved Training ofWasserstein GANs Montreal Institute for Learning Algorithms,” Advances in Neural Information Processing Systems, vol. 30, pp. 5767–5777, 2017.
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida, “Spectral normalization for generative adversarial networks,” in 6th International Conference on Learning Representations, 2018.
I. H.Witten, “An adaptive optimal controller for discrete-time markov environments,” Information and control, vol. 34, no. 4, pp. 286–295, 1977.
A. G. Barto, R. S. Sutton, and C. W. Anderson, “Neuronlike adaptive elements that can solve difficult learning control problems,” IEEE transactions on systems, man, and cybernetics, vol. SMC-13, no. 5, pp. 834–846, 1983.
R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” Advances in neural information processing systems, pp. 1057–1063, 2000.
R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning, vol. 8, no. 3, pp. 229–256, 1992.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in The 32nd International Conference on Machine Learning, vol. 1, 2015, pp. 448–456.
J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer Normalization,” 2016, http://arxiv.org/abs/1607.06450.
F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” 2015, https://arxiv.org/abs/1511.07122.
J. H. Lim and J. C. Ye, “Geometric GAN,” 2017, http://arxiv.org/abs/1705.02894.
S. Kullback and R. A. Leibler, “On information and sufficiency,” The annals of mathematical statistics, vol. 22, no. 1, pp. 79–86, 1951.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” Advances in neural information processing systems, vol. 30, 2017.
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved Techniques for Training GANs,” Tech. Rep., 2016.
