A Two-Step Approach for Syllabus Development and Evaluation using Machine Learning such as Doc2Vec based on Eduinformatics
Advancements and Challenges in Adopting Rubrics for Grad-ing Criteria in Japanese Higher Education
DOI:
https://doi.org/10.52731/lir.v003.128Keywords:
Two-Step Approach, Grading Criteria, Reference Rubrics (R2), Machine Learning, EduinformaticsAbstract
This study examines the adoption of rubrics in grading criteria at Japanese universities. Between 2016 and 2020, the percentage of universities explicitly stating grading criteria in syllabi increased to 98.1%. However, the use of rubrics for all subjects at the undergraduate level only rose from 2.9% to 6.1%, indicating challenges in rubric adoption. A two-step approach was proposed to address faculty members' difficulties in creating rubrics: 1) introducing a reference rubric (R2) based on common learning outcomes in diploma policies, and 2) having faculty members create rubrics for their individual learning outcomes. Successful implementation of this approach was reported at a university where rubrics were introduced for all subjects. Furthermore, the study evaluated the comparison between R2 and syllabi rubrics using doc2vec, pre-trained by Wikipedia text data, and cosine similarity to assess the degree of optimization for educational goals of each departments.
References
MEXT, “Status of reform of educational content and other aspects of education at universities (Japanese),” 2022. [Online]. Available: https://www.mext.go.jp/a_menu/koutou/daigaku/04052801/1417336_00009.htm. [Accessed: 15-Apr-2022].
D. D. Stevens and A. J. Levi, Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Stylus Publishing, LLC, 2013.
H. G. Andrade, “Using rubrics to promote thinking and learning,” Educ. Leadersh., vol. 57, no. 5, pp. 13–19, 2000.
Y. M. Reddy and H. Andrade, “A review of rubric use in higher education,” Assess. Eval. High. Educ., vol. 35, no. 4, pp. 435–448, 2010.
R. J. Stiggins, Student-involved classroom assessment. Prentice Hall, 2001.
J. A. Arter and J. Chappuis, Creating & recognizing quality rubrics. Pearson Merrill Prentice Hall, 2007.
H. Fukui, “A Literature Review Related to the Rubric Development in University :Focusing on the practical evaluation criteria (Japanese),” J. Educ. Des., vol. 14, no. 2, pp. 20–28, 2023.
Y. Hoshi and S. Koshikawa, “Effects and Challenges of Using Rubrics for Self-Assessment in University Education (Japanese),” Japan Bull. Educ. Hum. Dev., vol. 20, no. 1, pp. 21–31, 2020.
MEXT, “Towards the qualitative transformation of university education in order to build a new future,” The Central Council for Education, 2012. [Online]. Available: http://www.mext.go.jp/b_menu/shingi/chukyo/chukyo0/toushin/1325047.htm. [Accessed: 15-Mar-2018].
MEXT, “Grand design of higher education for 2040 (report),” 2018. [Online]. Available: https://www.mext.go.jp/b_menu/shingi/chukyo/chukyo0/toushin/1411360.htm. [Accessed: 05-Oct-2020].
K. Takamatsu, K. Murakami, T. Kirimura, K. Bannaka, I. Noda, L. R.-J. Wei, K. Mitsunari, M. Seki, E. Matsumoto, M. Bohgaki, A. Imanishi, M. Omori, R. Adachi, M. Yamasaki, H. Sakamoto, K. Takao, J. Asahi, T. Nakamura, et al., “‘Eduinformatics’: A new education field promotion,” Bull. kobe Tokiwa Univ., vol. 11, pp. 27–44, 2018.
K. Takamatsu, Y. Kozaki, K. Murakami, A. Sugiura, K. Bannaka, K. Mitsunari, M. Omori, and Y. Nakata, “Review of Recent Eduinformatics Research,” in IEEE/IIAI International Congress on Applied Information Technology 2019, 2019, pp. 27–32.
K. Takamatsu, I. Noda, B. Kenya, T. Nakagawa, Y. Kozaki, K. Mitsunari, M. Omori, R. Adachi, and Y. Nakata, “A New Concept of ICT on Eduinformatics in Higher Education,” in Proceedings of Sixth International Congress on Information and Communication Technology: ICICT 2021, London, Volume 1 (Lecture Notes in Networks and Systems Book 235), Springer Nature, 2022, pp. 693–700.
K. Takamatsu, M. Katsuhiko, Y. Kozaki, K. Bannaka, I. Noda, K. Mitsunari, M. Omori, and Y. Nakata, “Eduinformatics: A New Academic Field Needed in the Age of Information and Communication Technology,” in Intelligent Sustainable Systems: Selected Papers of WorldS4 2021, Volume 1 (Lecture Notes in Networks and Systems Book 333), 2022, pp. 139–147.
K. Takamatsu, I. Noda, K. Bannaka, K. Murakami, Y. Kozaki, K. Mitsunari, M. Omori, and Y. Nakata, “A new Concept of Digital transformation,Institutional Research,and Information and Communication Technology in Higher Education based on Eduinformatics,” in Proceedings of Seventh International Congress on Information and Communication Technology, Springer Nature, 2023, pp. 233–24.
Y. Nakata, K. Bannaka, I. Noda, K. Murakami, Y. Kozaki, K. Mitsunari, M. Omori, and K. Takamatsu, “A New Concept of data-driven education based on Eduinformatics,” in Proceedings of Eight lnternational Congress on lnformation and Communication Technology: ICICT2023, London,Volume4, Springer Nature, 2024, p. in Press.
K. Takamatsu, I. Noda, K. Bannaka, K. Murakami, Y. Kozaki, A. Kishida, H. Kabutoya, K. Mitsunari, R. Adachi, M. Omori, and Y. Nakata, “Sustainability of Digital transformation (DX), Institutional Research (IR), and Information and Communication Technology (ICT) in Higher Education based on Eduinformatics,” in Intelligent Sustainable Systems: Selected Papers of WorldS4 2022, 2022, p. in press.
K. Takamatsu, T. Oishi, S. Matsumoto, K. Nishiyama, I. Noda, S. Imai, K. Bannaka, K. Murakami, A. Kishida, Y. Nakata, and M. Mori, “Practice report of Feasibility-Sustainability Analytics (FS Analytics) in Institutional Research (IR) based on Eduinformatics and new proposal to improve Sustainability in IR,” in Proceeding of the eleventh meeting on Japanese Institutional Research, 2022, pp. 12–17.
MEXT, “Status of reform of educational content and other aspects of education at universities (Japanese),” 2017. [Online]. Available: https://www.mext.go.jp/a_menu/koutou/daigaku/04052801/1398426.htm. [Accessed: 15-Apr-2022].
K. Takamatsu, K. Murakami, T. Kirimura, K. Bannaka, I. Noda, M. Yamasaki, L. Rahpael-Joe, l Wei, K. Mitsunari, T. Nakamura, and Y. Nakata, “A new way of visualizing curricula using competencies: Cosine similarity, multidimensional scaling methods, and scatter plotting,” in Advanced Applied Informatics (IIAI-AAI), 2017 6th International Institute of Applied Informatics (IIAI) International Congress on. Institute of Electrical and Electronics Engineers (IEEE), 2017, pp. 192–197.
K. Takamatsu, Y. Kozaki, K. Murakami, K. Bannaka, I. Noda, W. Lim, Raphael-Joel, K. Mitsunari, T. Nakamura, and Y. Nakata, “A New Way of Visualizing Curricula Using Competencies : Cosine Similarity and t-SNE,” in Advanced Applied Informatics (IIAI-AAI), 2018 7th International Institute of Applied Informatics (IIAI) International Congress on. Institute of Electrical and Electronics Engineers (IEEE), 2018, pp. 390–395.
E. Matsumoto, K. Takamatsu, M. Bohgaki, A. Imanishi, M. Seki, and Y. Nakata, “Development of Course Guidance Support Method for‘Competency-Based Education (CBE)’ in University and its Issues,” Bull. Kobe Tokiwa Univ., vol. 12, pp. 17–28, 2019.
T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv Prepr. arXiv1301.3781, 2013.
Q. Le and T. Mikolov, “Distributed representations of sentences and documents,” in International conference on machine learning, 2014, pp. 1188–1196.
A. Hamana, “Rubric-based assessment,” Ministry of Education, Culture, Sports, Science and Technology, Central Council for Education, Upper Secondary School Pedagogy Subcommittee, Document 6., 2012. [Online]. Available: https://www.mext.go.jp/b_menu/shingi/chukyo/chukyo3/047/siryo/__icsFiles/afieldfile/2012/12/07/1328509_05.pdf. [Accessed: 01-Apr-2022].
A. Itoh, K. Takamatsu, H. Ito, K. Nishiyama, T. Kirimura, T. Kunisaki, I. Noda, K. Bannaka, K. Murakami, M. Kondo, R. Kozaki, R. Kitamatsu, K. Kabutoya, M. Omori, and Y. Nakata, “Development of a reference rubric (R2) to promote the use of rubrics in higher education, and comparison of the rubric of each subject with the R2,” J. Learn. Anal., p. submitted, 2023.
