Interpretable Document-Level Polarity Classification with Inter-sentence Attention
Abstract
Large language models (LLMs), such as BERT and GPT-3, significantly impact our daily lives. They can derive accurate outputs for many tasks about natural languages, but they do not explain the reason for the outputs. In this sense, LLMs work as a black box and have the problem of interpretability. This paper is devoted to considering polarity classification of documents. Compared to simple sentence-level polarity classification, document-level one makes it more difficult to create a high-performance model because we have to consider relationships between sentences. To tackle this change, we use inter-sentence attention, which can capture the relationship between sentences: the higher an inter-sentence attention score is, the more relevant the corresponding sentences are to each other. We use intersentence attention scores to capture the context of sentences and propose a model whose classification is more similar to human judgment. To validate the proposed model, we conduct three types of experiments: one is to compare classification performance with prior models; the second one is to compare interpretability with prior models; and the last one is to show the ability of inter-sentence attention whether it could capture the relationship between sentences. From the first two types, we found that our model is more accurate on two real datasets. In the second type of experiment to assess interpretability, we examined the overlap between sentences that contribute to the model’s predictions and those annotated by humans for the same document and found that our model has a larger overlap and is more likely to extract interpretive sentences that humans intuitively consider important. In addition, our result partially captures the polarity of “implicit” sentences that do not contain direct expressions, which could not be captured by prior models, suggesting that our model may lead to a more natural interpretation. From the third type of experiment, we show that our model can capture the contexts of sentences.
References
R. Feldman, S. Govindaraj, J. Livnat, and B. Segal. Management’s tone change, post earnings announcement drift and accruals. Review of Accounting Studies, 15(4):915–953, 2010.
J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186, 2019.
A. B. Arrieta, N. D. Rodr´ıguez, J. D. Ser, A. Bennetot, S. Tabik, A. Barbado, S. Garc´ıa, S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila, and F. Herrera. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion, 58:82–115, 2020.
X.-Q. Chen, C.-Q. Ma, Y.-S. Ren, Y.-T. Lei, N. Q. A. Huynh, and S. Narayan. Explainable artificial intelligence in finance: A bibliometric review. Finance Research Letters, 56:104145, 2023.
S. M. Lundberg and S.-I. Lee. A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems, 30, 2017.
M. T. Ribeiro, S. Singh, and C. Guestrin. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144, 2016.
J. Lu, M. Henchion, I. Bacher, and B. M. Namee. A Sentence-Level Hierarchical BERT Model for Document Classification with Limited Labelled Data. In Proceedings of 24th International Conference on Discovery Science, pages 231–241, 2021.
E. Mosca, D. Demirt¨urk, L. M¨ulln, F. Raffagnato, and G. Groh. GrammarSHAP: An Efficient Model-Agnostic and Structure-Aware NLP Explainer. In Proceedings of the First Workshop on Learning with Natural Language Supervision, pages 10–16, 2022.
H. Yan, L. Gui, and Y. He. Hierarchical interpretation of neural text classification. Comput. Linguistics, 48(4):987–1020, 2022.
L. Luo, X. Ao, F. Pan, J. Wang, T. Zhao, N. Yu, and Q. He. Beyond Polarity: Interpretable Financial Sentiment Analysis with Hierarchical Query-driven Attention. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, pages 4244–4250, 2018.
S. Kato and D. Ikeda. Improving Interpretability in Document-Level Polarity Classification by Applying Attention. In Proceedings of 16th IIAI International Congress on Advanced Applied Informatics (IIAI-AAI), pages 21–26, 2024.
T. Ito, K. Tsubouchi, H. Sakaji, T. Yamashita, and K. Izumi. Contextual Sentiment Neural Network for Document Sentiment Analysis. Data Sci. Eng., 5(2):180–192, 2020.
S. Yang, L. Xing, Y. Li, and Z. Chang. Implicit sentiment analysis based on graph attention neural network. Engineering Reports, 4(1):e12452, 2022.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems 30, pages 5998–6008, 2017.
S. Jain and B. C. Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 3543–3556. Association for Computational Linguistics, 2019.
S. Vashishth, S. Upadhyay, G. S. Tomar, and M. Faruqui. Attention interpretability across NLP tasks. arXiv:1909.11218, 2019.
L. Bacco, A. Cimino, F. Dell’Orletta, and M. Merone. Explainable Sentiment Analysis: A Hierarchical Transformer-Based Extractive Summarization Approach. Electronics, 10:2195, 2021.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692, 2019.
S. Xu, X. Zhang, Y. Wu, F. Wei, and M. Zhou. Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1784–1795, 2020.
O. Vinyals, M. Fortunato, and N. Jaitly. Pointer Networks. In Advances in Neural Information Processing Systems 28, pages 2692–2700, 2015.
A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning Word Vectors for Sentiment Analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, 2011.
B. Pang and L. Lee. A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, pages 271–278, 2004.
R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, 2013.
M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of NAACLHLT 2019, 2019.
M. Jia, C. Xie, and L. Jing. Debiasing Multimodal Sarcasm Detection with Contrastive Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18354–18362, 2024.
L. Yang and D. Ikeda. The Influence of Linguistic Attribute Differences in Multilingual Datasets on Sarcasm Detection. International Journal of Service and Knowledge Management, 8(2), 2024.
