A large language model graded marketing posts best, but it fell for hashtags.

Grading student writing automatically is hard because computers often struggle to match how humans grade quality. Researchers tested this in a marketing tool called BrandSim. They compared human grades on student social media posts against three computer scoring methods: fixed rules, an AI known as a large language model, and a hybrid mix of both. Two human judges scored posts from students, and researchers checked how closely each computer method matched them. The AI worked best, achieving the highest agreement with humans and the smallest average error. Fixed rules performed the worst, and mixing both dragged down the AI's score. Interestingly, the AI still made odd mistakes, giving high scores to nearly blank posts that just included relevant hashtags. Adding intentionally bad posts also distorted the agreement numbers. The researchers concluded that grading systems must be tested directly on real student work.

Why it matters

Automated grading of marketing posts can save instructors hours, but only if the scores match human judgment. This study found that a large language model agreed with human raters better than rule-based systems did—yet it also gave high scores to near-empty posts with hashtags, revealing a blind spot.

Questions answered