New principles for research assessment
Researchers are used to being assessed – we know that our merits are assessed when we apply for an academic position or a research project. Typically, we are assessed on narrow, quantitative criteria such as the number of scientific publications, h index, and the impact factor of academic journals. But new winds are blowing, and we may have to get used to being assessed on more qualitative merits.
Aarhus University has signed the Agreement on Reforming Research Assessment (ARRA) under the Coalition for Advancing Research Assessment (CoARA). So far, it may be difficult for employees to see that this has significantly changed AU’s approach to assessing researchers and research quality. However, implementation should be underway. Perhaps the university can find inspiration in the Independent Research Fund Denmark (DFF), which has also signed the agreement and is now taking its first steps toward seriously implementing new principles for assessment.
ARRA outlines several principles intended to reform research assessment practices to place greater emphasis on qualitative parameters, diversity, transparency, and the responsible use of quantitative indicators. CoARA and ARRA emerged from a growing recognition that a strong focus on a few quantitative indicators - such as publication counts, h‑index, and impact factors - does not adequately capture research quality and impact. Furthermore, there is a risk - and examples exist - that assessment and promotion systems based on these quantitative indicators foster an unhealthy research and collaboration culture, where important competencies such as teaching experience and engagement, collaborative abilities, and collegiality are undervalued. CoARA’s aim is instead to promote the development of assessment and promotion systems that support a positive research culture, where research increasingly focuses on societal, environmental, democratic, and economic challenges.
Signatories of ARRA commit, among other things, to implementing the following in research assessments:
• Recognising the diversity of contributions to, and careers in, research in accordance with the needs and nature of research
• Basing research assessment primarily on qualitative evaluation, supported by the responsible use of quantitative indicators
• Avoiding the use of rankings of research organisations and the inappropriate use of journal‑ and publication‑based metrics, especially the inappropriate use of Journal Impact Factor and h‑index
As many of you will know, DFF has recently introduced changes to its application procedures and assessment processes as a first step toward meeting ARRA’s requirements. In the current application rounds, DFF will place greater emphasis on the scientific quality and originality of applications, as well as the applicant’s competencies in relation to executing the specific research project. In addition, assessments will focus on applicants’ previous contributions to the scientific field and to the wider society, as well as to the research community, for example, peer review, leadership, collaboration and supervision. It will be interesting to follow whether, and how, this will influence grant outcomes in the upcoming application rounds.
Some may worry that these changes risk diluting scientific excellence when a long list of publications in high‑ranking journals is no longer the single most important marker of merit. I do not share that concern. High‑quality scientific output will continue to be an important assessment criterion. But now we must make a greater effort to describe the quality and impact of our research, and to show our contribution to healthy research environments that are characterized by collaboration towards new scientific insights, groundbreaking discoveries, and solutions to the major challenges of our time and our planet.