Artificial intelligence is increasingly being used to review academic papers, and research funders are considering whether it could also assess grant proposals. Although AI could make evaluations faster and more efficient, a study of 139 real grant applications found that AI-assisted assessment may favor established ideas over innovative research, potentially making it harder for novel work to receive funding.
In an analysis published in Scientometrics in August1, study author Diogo Machado asked OpenAI’s GPT-4o to evaluate 139 grant proposals. Machado, who leads the data science team at Technopolis, a science and policy consultancy based in Brighton, UK, gave the model a simple instruction: “Which proposal should we fund? Please answer only A or B.”
The model assessed proposals in pairs, comparing each proposal with every other proposal assigned to the same research area. Proposals with the most “wins” were placed at the top and selected for funding.
All of the proposals were genuine applications submitted in 2021 to the Austrian Science Foundation, Austria’s main public funder of basic research. Along with the full proposal text, each application included human reviewers’ assessments of novelty, scientific quality, team competency and feasibility, as well as the final funding decision.
Machado found that the AI model’s decisions matched those of human judges in 60% of cases. However, the human reviewers showed what he described as a “systematic reluctance” to fund proposals that received high marks for novelty.
“The risk is that AI-assisted review systems could steer funding towards research based on established ideas and overlook potential scientific advances,” Machado said.
Can AI recognize novelty?
The findings come as several major funding organizations examine the use of AI in grant evaluation. UK Research and Innovation, the UK’s national research funding agency, announced last month that it is considering AI-assisted evaluations, beginning with controlled trials comparing AI-generated assessments with past funding decisions.
Other funders have imposed restrictions. Wellcome, based in London, says that reviewers must not enter grant applications into generative AI tools because of privacy concerns. It can, however, be used for a “side job”, such as checking whether similar studies have already been conducted.
Understanding the biases that AI tools can introduce into research evaluation should help funders develop appropriate safeguards, Machado said.
His study did not examine why the bias against novelty occurred. Machado suspects it could be partly related to how large language models such as GPT-4o learn from repeated patterns in vast quantities of data. These models may be better at recognizing mainstream ideas because those ideas appear more often in their training data. New contributions, by definition, occur less frequently, leaving models with fewer patterns against which to assess their potential value.
The study also had a limitation: the AI agent was asked only which proposals should receive funding, rather than being given the same criteria used by human reviewers. Even if an AI tool is explicitly asked to assess novelty, however, there is no guarantee that it can do so reliably.
Novelty is difficult to standardize as a measure because it is highly subjective, said Susan Guthrie, director of science and emerging technologies at Rand Europe, a nonprofit research institute in Cambridge, England.
Source: www.nature.com