Implementation of Autoregressive Language Models for Generation of Seasonal Fixed-form Haiku in Japanese
DOI:
https://doi.org/10.52731/liir.v003.075Keywords:
haiku generation, natural language generation, human evaluation, language modelAbstract
This paper describes the implementation of an artificial intelligence haiku generator. We trained language models using existing haiku and literary studies, evaluated model performance using automatically computable evaluation indices such as perplexity, and subjectively evaluated the generated haiku by using a questionnaire.
The main contributions of this paper are as follows. First, the effectiveness of a series of model evaluation processes, including automatically calculable evaluation indices and the results of subjective evaluations using questionnaires, is investigated. These processes are effective in the development of haiku generation models. Second, high-quality haiku generation is achieved using high-performance language models such as GPT-2 and BART.
The results of the questionnaire survey revealed that it is possible to generate sensible haiku comparable to those written by humans.
The insight gained from this study is applicable to other generative tasks.
References
Xianchao Wu, Momo Klyen, Kazushige Ito, and Zhan Chen. Haiku generation using
deep neural networks. In The Association for Natural Language Processing, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural
Computation, 9(8):1735–1780, 11 1997.
A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. Language
models are unsupervised multitask learners. In OpenAI Blog, 2019.
Mika Hämäläinen, Khalid Alnajjar, and Thierry Poibeau. Modern French Poetry Generation
with RoBERTa and GPT-2. In 13th International Conference on Computational Creativity
(ICCC) 2022, Bolzano, Italy, June 2022. ICCC.
Xingxing Zhang and Mirella Lapata. Chinese poetry generation with recurrent neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language
Processing (EMNLP), pages 670–680, Doha, Qatar, October 2014. Association for Computational Linguistics.
Vaswani et al. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach,
R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Naoko Tosa, Hideto Obara, and Michihiko Minoh. Hitch haiku: An interactive supporting
system for composing haiku poem. In Scott M. Stevens and Shirley J. Saldamarco, editors,
Entertainment Computing - ICEC 2008, pages 209–216, Berlin, Heidelberg, 2009. Springer
Berlin Heidelberg.
Rafal Rzepka and Kenji Araki. Haiku generator that reads blogs and illustrates them with
sounds and images. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI’15, page 2496–2502. AAAI Press, 2015.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, page 2852–2858. AAAI Press, 2017.
Konishi et al. Generation of haiku preferable for ordinary people by seqgan. In Information
Processing Society of Japan, Kansai Branch Section Convention, 09 2017.
Mika Hämäläinen and Khalid Alnajjar. Human evaluation of creative nlg systems: An interdisciplinary survey on recent papers. In Proceedings of the 1st Workshop on Natural
Language Generation, Evaluation, and Metrics (GEM 2021), page 84–95, United States,
Workshop on Natural Language Generation, Evaluation, and Metrics, GEM Workshop.
Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. Applying conditional random fields to
Japanese morphological analysis. In Proceedings of the 2004 Conference on Empirical
Methods in Natural Language Processing, pages 230–237, Barcelona, Spain, July 2004.
Association for Computational Linguistics.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing
LSTM language models. 6th International Conference on Learning Representations, ICLR
- Conference Track Proceedings, 2018.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed,
Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational
Linguistics, pages 7871–7880, Online, July 2020. Association for Computational Linguistics.
Wolf et al. Transformers: State-of-the-art natural language processing. In Proceedings of the
Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October 2020. Association for Computational Linguistics.
