A Time-Constrained Analysis of Dynamic Early Warning Systems for Academic Risk Prediction
DOI:
https://doi.org/10.52731/lir.v006.489Keywords:
Early Warning Systems, Time-Constrained Prediction, Learning Analytics, Institutional ResearchAbstract
Implementing effective academic support for mandatory first-year courses requires precise decision-making about when to intervene, with whom, and with what level of certainty. This study extends our previous static prediction model (AUC=0.878 [1] using enrollment data alone) by addressing its key limitation: the inability to answer operational questions about intervention timing. Using data from a mandatory Information Literacy course at Hokuriku University (N=335, Economics and Management faculty, 2022-2023), we developed machine learning models that incrementally add dynamic formative assessment data from weeks 2-8 to static enrollment information. Under strict time-constraints preventing data leakage, we evaluated models using Recall@Precision≥0.90—a practical metric balancing intervention resource constraints with student rescue effectiveness. Results demonstrate that minimal behavioral features from weeks 2-8 (submission rates, task completion counts) significantly improve Recall@P≥0.90 from 1.6% to 3.2%, doubling rescue capacity while providing weeks of intervention lead time.
References
S. Tajiri, K. Takamatsu, N. Shiratori, T. Oishi, M. Mori, and M. Murota, "Predicting Performance in First-Year Required Courses Using Machine Learning Based on At-Enrollment Data," in Proc. IIAI AAI 2024 Winter (DSIR Winter), 2024.
A. Pardo, F. Han, and R. A. Ellis, "Combining university student self-regulated learning indicators and engagement with online learning events to predict academic performance," IEEE Transactions on Learning Technologies, vol. 10, no. 1, pp. 82–92, 2017.
N. Kondo, M. Okubo, and T. Hatanaka, "Early Detection of At-Risk Students Using Machine Learning Based on LMS Log Data," in Proc. 6th IIAI International Congress on Advanced Applied Informatics, pp. 198–201, 2017.
D. Gašević, S. Dawson, and G. Siemens, "Let’s Not Forget: Learning Analytics are About Learning," TechTrends, vol. 59, no. 1, pp. 64–71, 2015.
K. E. Arnold and M. D. Pistilli, "Course Signals at Purdue: Using Learning Analytics to Increase Student Success," in Proc. LAK’12, pp. 267–270, 2012.
S. Kaufman, S. Rosset, C. Perlich, and O. Stitelman, "Leakage in Data Mining: Formulation, Detection, and Avoidance," in Proc. KDD’11, pp. 556–563, 2011.
T. Saito and M. Rehmsmeier, "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets," PLOS ONE, 10(3): e0118432, 2015.
T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System," in Proc. KDD’16, pp. 785–794, 2016.
S. M. Lundberg and S.-I. Lee, "A Unified Approach to Interpreting Model Predic-tions," in Proc. NeurIPS 30, pp. 4765–4774, 2017.
