Why comparing countries’ research systems is difficult

Different countries collect and analyze research data in different ways. Relevant information can be difficult to find or access, and the same concept can be defined and measured differently from one country to another.

Scientific output — including publications, citations and patents — can be quantified relatively easily. But these indicators reveal little about the factors that influenced the results. Standardized statistics, such as research-investment data published by the Organization for Economic Co-operation and Development (OECD), make it easier to compare countries. However, they are not sufficient for analyzing the intertwined effects of funding models, institutional management and support for researchers over time.

Some indicators, such as how funding shapes researchers’ careers, are rarely measured systematically. Even when countries report similar measures, differences in data collection can make comparisons difficult.

For example, a country reporting information on basic research might publish detailed expenditure statistics by sector and funding source. Another might report only total expenditures, including applied research. These differences make it difficult to form a coherent picture of research systems around the world.

A framework for analyzing national research systems

To address these problems, the group I lead has developed an approach to analyzing national research systems. It combines data from a wide range of sources and develops countermeasures for aspects of research systems that are often overlooked. We hope that researchers, funders and international organizations will use the framework and identify ways to improve it.

Our project, the China Basic Research Report, examines research systems in 25 countries and regions, including the European Union (www.cfrr.net). The selected regions have research systems of varying size, structure and stage of development.

China provides a useful case study because its scientific landscape has changed over the past four decades through continued investment in funding, institutions, human resources and research infrastructure.

Using open data and new research metrics

Our research has been made possible by the rapid expansion of openly available research information, including publication and funding databases, data repositories and other sources. This has made it easier to compare different aspects of research across countries. We used large-scale computational techniques to analyze datasets that had previously been difficult to study together.

Another important part of our work was developing new metrics to capture areas of research that are not typically measured. For example, we used large-scale language models to identify signs of novelty in research questions and methods — specifically, the extent to which research goes beyond established knowledge.

We also linked publications to patent citations to track connections between basic research and technology development. In addition, we combined funding and workforce data to examine how young researchers are contributing to scientific output in China.

Our analysis found that this group is supported by the long-term expansion of funding for young researchers from the National Natural Science Foundation of China.