Scientific funding agencies are increasingly creating dedicated metascience teams that use scientific methods to study how research is funded, conducted and evaluated. In July, Office of Science and Technology Policy Director Michael Kratsios called on all federal science agencies to establish such departments, and Ireland has also created a similar team. The UK Metascience Unit, launched in 2024, offers an early example of how experiments conducted within a national funding system can inform policy and practice.
Testing the research-funding process
The UK Metascience Unit is jointly run by the UK government’s Department of Science and UK Research and Innovation (UKRI), a national funding body. As a co-founder of the first metascience team to be incorporated into a national research funding system, I have seen how experiments carried out within the funding process can make a difference.
Research-funding processes often evolve organically rather than through systematic testing and evaluation. Over the past two years, my team has conducted numerous experiments and data analyses to identify effective ways to fund and support research and innovation.
For example, we tested whether email invitations to grant reviewers that appealed to their sense of social obligation would increase acceptance rates. They did not. We also evaluated whether decentralized peer review, in which applicants review one another’s proposals, could speed up grant evaluations. We found that it could, because the number of reviewers automatically increases with the number of applicants, eliminating the need to hire external reviewers.
We continue to test new ideas, including attempts to improve staff desk rejection. At this stage, UKRI staff identify and remove uncompetitive applications before they reach peer review. The goal is to explore ways to reduce the demands placed on examiners.
Lessons from building a new unit
Because this was the first unit of its kind, the team had no established template to follow and had to learn as it went. One of the clearest lessons was that metascience units should remain small.
Metascience research shows that large teams tend to expand on existing ideas, whereas small teams tend to challenge or overturn them. The Metascience Unit’s role is to be disruptive and advocate for change.
I firmly believe that our unit achieved more with a team of six than it would have with a team of 60. When early experiments showed that email nudges were ineffective, we decided to focus our efforts on testing a new intervention rather than trying to improve an unpromising one. Changing direction would, I think, have been more difficult with a larger unit. Small teams have little choice but to be ruthless with their time.
Source: www.nature.com