Original title: How bibliometric indicators align with REF 2021 peer review across disciplines
Junjie Shen · September 2026
Experts often judge research quality by reading papers, but people increasingly use automated numbers, called bibliometric indicatorsNumbers that measure the impact or quality of research, like how many times a paper is cited or the prestige of the journal it's published in., to guess those expert scores. It is unclear how well these numbers match human reviews across different subjects. Researchers tested this by comparing overall institutional scores from the UK REF 2021 review with two kinds of data: how often articles are cited, and the prestige of the journals publishing them. They examined fifteen subject areas. They found that subjects have very different judging habits. Physics scores matched citation counts much better than journal prestige, with correlations of 0.85 compared to 0.42. In contrast, Business and Management Studies matched journal prestige far better, at 0.92 compared to 0.59. Overall, nine subjects aligned best with journal ranks, while six aligned with citations, showing that metric accuracy depends on each field's culture.
Imagine two restaurants are being judged by food critics. One critic always checks how many people are lining up outside (like citations), while the other looks at the restaurant's reputation (like journal prestige). In a busy city, the line length might be a great sign of quality, but in a quiet town, the restaurant's reputation matters more. In this paper, the researchers found that in some fields, citation counts are like the line length—very telling—while in others, journal prestige is the better guide. The right measure depends entirely on the field.
Research assessments like the UK's REF decide how billions in funding are distributed. If metrics are used without knowing how they behave in each field, entire disciplines could be misjudged. This study shows that the same indicator can be excellent in one field and misleading in another, so assessment systems must be tailored to each field's evaluation culture.
The study uses institutional-level GPA, not individual article scores, so it cannot show how well metrics predict the quality of single papers. It only covers UK REF 2021 data, and the findings may not generalize to other countries or assessment systems. The correlations are based on shares of outputs in the top 10%, which may not capture the full distribution of quality.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
How bibliometric indicators align with REF 2021 peer review across disciplines — Junjie Shen — SocArXiv — doi:10.31235/osf.io/znu6c_v1
Abstract licence: CC BY
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.