Original title: NormasTCU — A Brazilian Portuguese IR dataset and an evaluation of LLM-as-a-judge for relevance assessment
Leandro Carísio Fernandes, Marcus Vinícius Borela de Castro, Leandro dos Santos Ribeiro, Leonardo Augusto da Silva Pacheco +1 · September 2026
Testing search systems needs lots of labeled data, but paying people to read and grade specialized texts takes time and money. Researchers also do not know if artificial intelligence can reliably judge search results in languages other than English. To test this, researchers created NormasTCU, a Brazilian Portuguese legal search dataset. They asked three large language models to grade how well documents matched search questions. Then, they checked if these AI scores could rank 15 search engines accurately. The AI models tended to be overly generous, scoring documents higher than humans did. They only matched human pair-level choices fairly to moderately well. However, when sorting search engines by top-ranking quality, the AI judgments worked very well. They even matched true reference rankings better than single human raters sometimes did. Still, the models struggled to measure precision and recall, meaning AI graders work for some search tests, but not all.
Imagine you are grading a stack of student essays. You have a rubric, but you tend to give slightly higher scores than another teacher. Despite that, when you both rank the essays from best to worst, your order is almost the same. Here, the AI system is like the lenient grader: it consistently gives higher relevance scores than humans, but the resulting ranking of documents is very similar to the human ranking.
Building search systems for Portuguese legal documents is expensive because human judges must manually assess relevance. This work shows that large language models can produce rankings nearly identical to human judges, despite a tendency to overestimate relevance. That means researchers can scale up relevance assessment at lower cost, but only when using rank-aware metrics like nDCG, not precision or recall.
The study is limited to one dataset of Portuguese legal documents, so results may not generalize to other languages or domains. LLM judgments show consistent positive bias and only fair-to-moderate agreement with humans at the pair level. The high ranking correlation for nDCG and MRR was not always statistically significant, and the models performed poorly for precision and recall metrics. The findings are also specific to the three models and two prompt techniques tested.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
NormasTCU — A Brazilian Portuguese IR dataset and an evaluation of LLM-as-a-judge for relevance assessment — Leandro Carísio Fernandes, Marcus Vinícius Borela de Castro, Leandro dos Santos Ribeiro et al. — Information Processing & Management — doi:10.1016/j.ipm.2026.105151
Abstract licence: CC BY
Explanation generated by Synthesis from the abstract — it is not the original paper.