Original title: Crediting assisted work inflates mastery: a preregistered comparison of evidence rules in intelligent tutoring logs
Kumar Srivastava · September 2026
Computer tutoring systems track what students do and the help they receive. Most systems give students credit for finishing a problem even if they needed hints along the way. This study asked if treating assisted work like independent work creates a false picture of what students actually know. The researchers tested four different rules for updating mastery on educational log data. Some rules credited students for finishing problems with help, while others only counted unassisted first tries. They found that counting only first attempts was best at predicting future unaided performanceHow well a student does on a problem without any hints or help.. Crediting any finished problem made students look far more skilled than they were, but those students struggled later on independent tasks. If a mastery score is supposed to show independent ability, the scoring rule matters.
Imagine a coach timing a runner's laps. One day the runner gets a head start from a teammate, but the coach still counts that lap as a personal best. Later, the coach is surprised when the runner cannot keep up in a race without help. In this paper, the tutoring system is like the coach, and the hints are like the head start. Counting helped problems as mastered inflates the estimate of what the student can do alone, just like counting the assisted lap overestimates the runner's true speed.
Teachers and tutoring systems rely on mastery estimates to decide what a student should study next. If those estimates count hints as success, they overstate what a student can do alone. That can lead to skipping needed practice or misjudging a student's readiness for harder material.
The study uses only the ASSISTments 2012–13 dataset, so results may not generalize to other tutoring systems or subjects. It compares only four evidence update rules; other rules might behave differently. The best AUC of 0.658 is still modest, meaning prediction is far from perfect. The abstract does not discuss whether the findings hold across different student populations or skill types.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Crediting assisted work inflates mastery: a preregistered comparison of evidence rules in intelligent tutoring logs — Kumar Srivastava — EdArXiv — doi:10.35542/osf.io/wzbqa_v1
Abstract licence: CC BY
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.