Research evaluation must evolve over time

The European Commission in Brussels wants a European-wide agreement on the evaluation of research that recognizes ethics and integrity as well as teamwork and a diversity of results.Credit: Bernal Revert / Alamy

Many publicly funded researchers are required to participate in national reviews of their work. Such assessments are popular with governments because they help ensure a certain degree of accountability for taxpayer dollars. Funders love them too, as they provide a useful benchmark for the level of research being carried out. Universities also benefit financially when they craft their research strategies around the requirements of assessments. In contrast, researchers generally regard evaluations as unnecessary for their work. Assessments can also be stressful and cumbersome and in some cases create tension between colleagues in academic and administrative positions.

With a few exceptions, the major components of assessment systems have remained largely the same since the fiscal years began in the 1980s. But some countries are considering reworking these systems to reflect the way science is done today. hui. Change has been long in coming, precipitated by initiatives such as the 2013 San Francisco Declaration on Research Evaluation, the 2015 Leiden Manifesto for Research Measures and the 2020 Hong Kong Principles for Research. evaluation of researchers. Official research evaluations are clearly behind schedule and need to catch up.

Last November, the European Commission announced its intention to conclude an EU-wide agreement on research evaluation. He proposes that the evaluation criteria reward ethics and integrity, teamwork and a diversity of results in addition to the quality and impact of research. Britain’s Future Research Assessment Program, which is expected to report by the end of this year, has also been tasked with proposing ways to ensure that assessments become more inclusive. These changes cannot come soon enough.

Measures of success

Research evaluation systems are the closest universities have to common business performance measures. Individual researchers are assessed on a range of measures, such as the number and quality of journal articles, books and monographs they have published; their research income; the number of students who obtained a postgraduate degree; and any non-academic impact of their work, such as its influence on society or politics. In the UK, for example, this information is compressed into a composite index and the results are used to allocate funds.

British public funding goes preferentially to university departments with the best performing researchers. But assessments that measure individual performance make it more difficult for institutions to recognize science conducted as a team, both within and between disciplines. In addition, research evaluations have tended to focus on final published results, as researchers increasingly produce more diverse results, including datasets, reproducibility studies, and recorded reports, in which researchers publish study designs before starting the experiments. Most current evaluations do not value mentorship and struggle to recognize the needs of researchers from minority communities.

And then there is the question of costs. The 2014 iteration of the UK Research Excellence Framework – the exercise takes place roughly every seven years – cost around £ 246 million (US $ 334 million). The lion’s share (£ 232million) went to universities. It included the costs of academic staff who served on review boards that assessed approximately 190,000 results in 36 areas; and the costs to institutions, which go to great lengths to prepare their staff, including conducting mock assessment exercises. Here, the small institutions lack the resources to compete with the better funded ones.

Researchers who study evaluation methods regularly offer ideas on how evaluations might change for the better. Last August, a working group of the International Network of Research Management Companies developed a framework called SCOPE. This encourages donors to design evaluation systems around the “values” they wish to evaluate. For example, rewarding competitive behavior may require a different set of criteria than encouraging collegiality. The SCOPE framework also proposes that funders work with assessors to design the assessment, and urge them to work with experts in research assessment – a defined area of ​​research.

The importance of co-design cannot be overstated: it will allow the views of different research actors to be represented and ensure that no voice dominates. Large research-intensive institutions often do well in conventional evaluations, as they focus their multi-year strategies on attracting and retaining researchers who meet the criteria for success in publication of results and income, among others.

Small institutions can’t always compete on these bases – but could win if future evaluations include new criteria, such as rewarding collaborations, or if the evaluations place less weight on the ability to secure research funding. A wider range of evaluation criteria could ensure that a greater diversity of institutions have the opportunity to do well. And this must be welcomed.

Large institutions should not feel threatened by these changes in any way. It is often said – in this journal and elsewhere – that making the research culture more welcoming requires systemic change. Research evaluation is at the heart of the research system. If the evaluation criteria can be made more representative of how research is carried out, this much-needed culture shift will take one more important step.

About Bernice Dyer

Check Also

NC Teach for America: – EducationNC

The following is part of my monthly column, One Day and One Goal: Expanding opportunity …