Randomizing parts of grant decision-making can improve diversity and reduce costs, researchers argue.
Credit: Andrew Angelov/Alamy
Rachel Heyard was asked to find a reliable signal in a noisy grant-review system. As a biostatistician on the data team at the Swiss National Science Foundation (SNSF) in Bern, Switzerland’s main public research funder, she investigated how much confidence could be placed in the scores assigned to grant applications. Reviewers and panel members assessed proposals for scientific relevance, feasibility and the applicant’s track record, grading them from A to D. The goal was to identify which applications genuinely ranked above the others for a limited pool of funding.
The strongest and weakest applications were usually easy to identify. The difficult decisions involved the crowded middle — what Heyard describes as “a cloud of B proposals” — where scores were so close that panels struggled to distinguish between them. To create a definitive ranking, reviewers added pluses and minuses, sometimes making distinctions that the evidence could not support.
Grant proposals drafted with AI help more likely to win NIH funding
The consequences were significant. In 2021, the SNSF distributed the equivalent of around US$1 billion each year to research projects in all disciplines. The boundary between funded and rejected proposals often cut through this tightly packed group of applications. A decision near that threshold could affect the future of a laboratory or the career of an early-career researcher.
Heyard helped develop a statistical framework that measured the reliability of reviewer scores. Rather than assigning every proposal a fixed position, the model calculated a range of possible rankings that reflected uncertainty in the panel’s judgement. Instead of claiming that one application was precisely 23rd and another was 27th, the system asked whether those rankings were genuinely distinguishable.
Near the funding threshold, they often were not. When a proposal’s possible positions crossed the line between funding and rejection, it entered a grey zone: strong enough to remain under consideration, but too close to neighbouring applications to support a precise ranking.
When grant funding becomes a lottery
From 2022, the SNSF stopped treating these close rankings as exact. Its new approach uses a mathematical model to identify proposals that reviewers cannot reliably separate, then sends those applications to a lottery. Peer review still determines which proposals meet the quality standard, but chance decides the outcome among applications near the funding cut-off. “Instead of saying, ‘should we introduce randomness into science processes?’, we’re reframing this question into ‘are we confident that our decisions are precise enough?’” says Heyard, now a researcher at the University of Zurich’s Center for Reproducible Science and Research Synthesis in Switzerland. “And the answer to that question is often no.”
Research supports Heyard’s argument. In a 2018 study, 43 reviewers assessed the same 25 applications and showed very limited agreement1. “We already know that the awards are a lottery,” says Tom Stafford, a cognitive scientist at the University of Sheffield, UK, who is on secondment at the Research on Research Institute in London. “We should just make it a formal lottery, rather than an unofficial one.”
Once peer review reaches its limits, using a random draw can sound like giving up. Yet lotteries have long been used as a tool for making fair public decisions. In ancient Athens, many public offices and juries were selected by lot, with eligible citizens chosen using a stone device called a kleroterion. For some research funders, random selection could represent not a radical departure, but a return to an established method of distributing opportunities.
How research-funding lotteries work
The Research on Research Institute maintains a catalogue of such trials. It has recorded more than a dozen funders that have used or tested partial lotteries, including 11 entries since 2022. Although the approach is still uncommon worldwide, it is being tested by national funding agencies, charities such as Wellcome in London and the Novo Nordisk Foundation in Hellerup, Denmark, and universities. Awards range from several thousand US dollars to more than $1 million. Stafford and other researchers do not yet have global data on the total amount distributed through grant lotteries, but interest in the model is increasing.
These systems differ in design. The least extensive approach is a tie-breaker, as used by the SNSF, in which a random draw decides only between proposals that reviewers cannot reliably rank. A broader model randomly selects from every application that meets a defined quality threshold. New Zealand’s Health Research Council has used this method since 2013 for its Explorer Grants, which are currently worth NZ$150,000 (US$87,000) each. The most radical model, used by some programmes in Germany, reverses the process: a lottery determines who is invited to submit a full proposal.
New peer-review trial lets grant applicants evaluate each other’s proposals
One of the most closely monitored experiments is taking place in the United Kingdom. In 2022, the British Academy, the national organization for humanities and social-sciences research, began randomly selecting recipients of its Small Research Grants from proposals that had passed a quality threshold. The trial has since been extended to 2028.
The academy wanted to address more than reviewer workload. It also found that many researchers were choosing not to apply because they believed they had little chance of competing with applicants from larger, research-intensive universities. “Some were disqualifying themselves from even making an application,” says Ken Emond, the academy’s head of research funding. A lottery can communicate that every qualified applicant has an equal chance of receiving support.
The effects were apparent soon after the system was introduced. Applications nearly doubled, rising from 643 in 2022 to 1,257 in the 2025–26 round. The applicant pool also became more diverse. The proportion of applications from Asian and Asian British researchers increased from 13% to 19%, while their share of awards rose from 11% to 17%. Emond says other factors contributed, but the results were consistent with the expected benefits of random selection.
A lottery also gives applicants more informative feedback. Previously, the academy provided no feedback at all. It can now tell researchers whether their proposal failed to meet the quality threshold or passed the assessment but was not selected in the draw. Some unsuccessful applicants, Emond says, view a “passed the threshold” decision as evidence that their proposal is strong enough to submit to another funder.
Can random selection improve research diversity?
Adrian Barnett, a statistician at the Queensland University of Technology in Brisbane, Australia, and one of the trial’s independent evaluators, says the potential diversity benefits are particularly important because they require little additional expense. After the British Academy screens applications for quality, the remaining proposals are anonymized and sent to Barnett, who conducts the draw. “I get the anonymized list, and I send them the winners and losers the next day,” he says. “Funders could gain in diversity without unpopular post hoc adjustments to scores. That’s a big gain for a simple change.”
Responses to the AI grant flood must prioritize fairness as part of excellence
Source: www.nature.com