Keywords
cross-institutional validation
computing education
learning analytics
Abstract
This study investigates the extent to which early keystroke and event log data from programming exercises can predict final course performance across different institutions and course contexts in computer science education. We conducted a multi-institutional study involving four universities, encompassing various course levels and programming languages. Fine-grained programming behavior data was collected from the first 25% of course activities and used to train and validate predictive models across different institutional contexts. Our Gradient Boosting model achieved consistent performance across institutions, with F1 scores ranging from 0.77 to 0.82. Time spent on initial exercises, frequency of compilation errors, and early code growth rate emerged as the most important predictive features. The model showed higher accuracy for introductory courses compared to advanced ones. This study contributes to the field by demonstrating the feasibility of early, generalizable performance prediction using keystroke and event log data, providing insights into cross-institutional model validity, and identifying a set of context-agnostic features consistently predictive across different course types and institutions. These findings have implications for developing more robust and widely applicable learning analytics tools in computer science education, potentially enabling timely interventions to improve student outcomes.
References
Alireza Ahadi, Raymond Lister, Heikki Haapala, and Arto Vihavainen. 2015. Exploring Machine Learning Methods to Automatically Identify Students in Need of Assistance. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research. ACM, 121–130. https://doi.org/10.1145/2787622.2787717
Alireza Ahadi, Raymond Lister, Shahil Lal, and Arto Hellas. 2016. Learning Programming, Syntax Errors and Institution-Specific Factors. In Proceedings of the Australasian Computer Science Week Multiconference. ACM, 1–10. https://doi.org/10.1145/2843043.2843047
Alireza Ahadi, Raymond Lister, and Donna Teague. 2017. Falling Behind Early and Staying Behind When Learning to Program. Proceedings of the 25th Psychology of Programming Interest Group Annual Conference (2017), 1–10.
Kai Arakawa, Qiang Hao, Wesley Deneke, Indie Cowan, Steven Wolfman, and Abigayle Peterson. 2022. Early Identification of Student Struggles at the Topic Level Using Context-Agnostic Features. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 1. ACM, 147–153. https://doi.org/10.1145/3478431.3499298
David Azcona and Alan F. Smeaton. 2017. Targeting At-Risk Students Using Engagement and Effort Predictors in an Introductory Computer Programming Course. In European Conference on Technology Enhanced Learning. Springer, 361–366. https://doi.org/10.1007/978-3-319-66610-5_27
Jens Bennedsen and Michael E. Caspersen. 2007. Failure Rates in Introductory Programming. ACM SIGCSE Bulletin 39, 2 (2007), 32–36. https://doi.org/10.1145/1272848.1272879
Paulo Blikstein, Marcelo Worsley, Chris Piech, Mehran Sahami, Steven Cooper, and Daphne Koller. 2014. Programming Pluralism: Using Learning Analytics to Detect Patterns in the Learning of Computer Programming. Journal of the Learning Sciences 23, 4 (2014), 561–599. https://doi.org/10.1080/10508406.2014.954750
Adam S. Carter, Christopher D. Hundhausen, and Olusola Adesope. 2015. The Normalized Programming State Model: Predicting Student Performance in Computing Courses Based on Programming Behavior. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research. ACM, 141–150. https://doi.org/10.1145/2787622.2787710
Hendrik Drachsler and Wolfgang Greller. 2016. Privacy and Analytics: It's a DELICATE Issue a Checklist for Trusted Learning Analytics. Proceedings of the Sixth International Conference on Learning Analytics Knowledge (2016), 89–98. https://doi.org/10.1145/2883851.2883893
John Edwards, Juho Leinonen, and Arto Hellas. 2020. A Study of Keystroke Data in Two Contexts: Written Language and Programming Language Influence Predictability of Learning Outcomes. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. ACM, 413–419. https://doi.org/10.1145/3328778.3366863
Dragan Gaševi'c, Shane Dawson, and George Siemens. 2016. Let's Not Forget: Learning Analytics Are About Learning. TechTrends 60, 1 (2016), 64–71. https://doi.org/10.1007/s11528-015-0001-y
Michael Guttmann, Aleksandar Karakaš, and Denis Helic. 2024. Attribution of Work in Programming Teams with Git Reporter. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1. ACM, 436–442. https://doi.org/10.1145/3626252.3630785
Arto Hellas, Petri Ihantola, Andrew Petersen, Vangel V. Ajanovski, Mirela Gutica, Timo Hynninen, Antti Knutas, Juho Leinonen, Chris Messom, and Soohyun Nam Liao. 2018. Predicting Academic Performance: A Systematic Literature Review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education. ACM, 175–199. https://doi.org/10.1145/3293881.3295783
Christopher D. Hundhausen, Daniel M. Olivares, and Adam S. Carter. 2017. IDE-Based Learning Analytics for Computing Education: A Process Model, Critical Review, and Research Agenda. ACM Transactions on Computing Education 17, 3 (2017), 1–26. https://doi.org/10.1145/3105759
Petri Ihantola, Arto Vihavainen, Alireza Ahadi, Matthew Butler, J"urgen B"orstler, Stephen H. Edwards, Essi Isohanni, Ari Korhonen, Andrew Petersen, Kelly Rivers, Miguel 'Angel Rubio, Judy Sheard, Bronius Skupas, Jaime Spacco, Claudia Szabo, and Daniel Toll. 2016. Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies. In Proceedings of the 2015 ITiCSE on Working Group Reports. ACM, 41–63. https://doi.org/10.1145/2858796.2858798
Matthew C. Jadud. 2006. Methods and Tools for Exploring Novice Compilation Behaviour. In Proceedings of the Second International Workshop on Computing Education Research. ACM, 73–84. https://doi.org/10.1145/1151588.1151600
Aleksandar Karakaš and Denis Helic. 2024. Combining Local Testing with Automatic Commits: Benefits for Progress Tracking and CS2 Students' Learning Experience. In Proceedings of the 2024 Innovation and Technology in Computer Science Education V. 1. ACM, 108–114. https://doi.org/10.1145/3649217.3653561
Charles Koutcheme, Sami Sarsa, Arto Hellas, Lassi Haaranen, and Juho Leinonen. 2022. Methodological Considerations for Predicting At-Risk Students. In Proceedings of the 24th Australasian Computing Education Conference. ACM, 105–113. https://doi.org/10.1145/3511861.3511873
Juho Leinonen, Francisco Enrique Vicente Castro, and Arto Hellas. 2021. Does the Early Bird Catch the Worm? Earliness of Students' Work and Its Relationship with Course Outcomes. In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1. ACM, 373–379. https://doi.org/10.1145/3430665.3456383
Juho Leinonen, Francisco Enrique Vicente Castro, and Arto Hellas. 2022. Time-on-Task Metrics for Predicting Performance. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 1. ACM, 871–877. https://doi.org/10.1145/3478431.3499359
Leo Lepp"anen, Juho Leinonen, Petri Ihantola, and Arto Hellas. 2016. Predicting Academic Success Based on Learning Material Usage. In Proceedings of the 16th Koli Calling International Conference on Computing Education Research. ACM, 13–22. https://doi.org/10.1145/2999541.2999558
Soohyun Nam Liao, Daniel Zingaro, Kevin Thai, Christine Alvarado, William G. Griswold, and Leo Porter. 2019. A Robust Machine Learning Technique to Predict Low-Performing Students. ACM Transactions on Computing Education 19, 3 (2019). https://doi.org/10.1145/3277569
Katerina Mangaroska and Michail N. Giannakos. 2019. Learning Analytics for Learning Design: A Systematic Literature Review of Analytics-Driven Design to Enhance Learning. IEEE Transactions on Learning Technologies 12, 4 (2019), 516–534. https://doi.org/10.1109/TLT.2018.2868673
Jonathan C. Munson and Stephen E. Zitney. 2015. Behavior-Based Learning Curves for Predicting Student Performance in a First-Year Engineering Course. In 2015 IEEE Frontiers in Education Conference (FIE). 1–6. https://doi.org/10.1109/FIE.2015.7344313
Zacharoula Papamitsiou and Anastasios A. Economides. 2014. Learning Analytics and Educational Data Mining in Practice: A Systematic Literature Review of Empirical Evidence. Educational Technology Society 17, 4 (2014), 49–64.
Filipe Dwan Pereira, Samuel C. Fonseca, Elaine H. T. Oliveira, David B. F. Oliveira, Alexandra I. Cristea, and Leandro S. G. Carvalho. 2020. Deep Learning for Early Performance Prediction of Introductory Programming Students: A Comparative and Explanatory Study. Brazilian Journal of Computers in Education 28 (2020), 723–749. https://doi.org/10.5753/rbie.2020.28.0.723
Paul Prinsloo and Sharon Slade. 2017. An Elephant in the Learning Analytics Room: The Obligation to Act. Proceedings of the Seventh International Learning Analytics Knowledge Conference (2017), 46–55. https://doi.org/10.1145/3027385.3027406
Zac Pullar-Strecker, Filipe Dwan Pereira, Paul Denny, Andrew Luxton-Reilly, and Juho Leinonen. 2023. G is for Generalisation: Predicting Student Success from Keystrokes. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. ACM, 1028–1034. https://doi.org/10.1145/3545945.3569824
Nils Rys-Recker and Qiang Hao. 2024. Early identification of struggling students in large computer science courses: A replication study. In Proceedings of the 48th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE, 88–93. https://doi.org/10.1109/COMPSAC61105.2024.00022
Christopher Watson and Frederick W. B. Li. 2014. Failure Rates in Introductory Programming Revisited. Proceedings of the 2014 Conference on Innovation Technology in Computer Science Education (2014), 39–44. https://doi.org/10.1145/2591708.2591749
Christopher Watson, Frederick W. B. Li, and Jamie L. Godwin. 2013. Predicting Performance in an Introductory Programming Course by Logging and Analyzing Student Programming Behavior. 2013 IEEE 13th International Conference on Advanced Learning Technologies (2013), 319–323. https://doi.org/10.1109/ICALT.2013.99
Valdemar Švábenský, Kristián Tkáčik, Aubrey Birdwell, Richard Weiss, Ryan S. Baker, Pavel Čeleda, Jan Vykopal, Jens Mache, and Ankur Chattopadhyay. 2024. Detecting Unsuccessful Students in Cybersecurity Exercises in Two Different Learning Environments. In Proceedings of the 54th IEEE Frontiers in Education Conference (FIE '24). IEEE. To be published.
This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.
Copyright (c) 2024 Lexi Dixon, Tony Tang, Samuel Darmasti (Author)