Pooled calibration hides fairness gaps of up to 10.5 points in hybrid essay scoring.

Automated scoring systems grade student essays and send tricky ones to human raters. However, researchers did not know if this routing treats all student groups fairly. To investigate, researchers audited three scoring systems across two essay datasets. They used a method called split conformal prediction. This technique turns the computer's confidence into a list of likely scores designed to contain the true grade at a chosen success rate. When using a single overall cutoff, the system hit its overall target but hid unfair gaps. For example, English language learners received 5.4 percentage points less coverage than promised under one system, but 3.3 points more under another. Surprisingly, the largest gap involved grade levels, differing by 10.5 points. Setting separate cutoffs for each group fixed these gaps across every test, but lowered how many essays from those groups were scored automatically, leaving automation rates between 2.4 and 0.2 percent for English language learners.

Why it matters

Automated essay scoring sends uncertain responses to human raters, but this routing can be unfair to specific student groups—like English learners or different grades—without anyone noticing because overall averages look fine. The paper shows how to detect these hidden disparities and fix them by setting separate thresholds per group, ensuring equitable treatment. This matters because millions of students are assessed this way, and unfair routing can skew their scores.

Questions answered