How the Paper2Agent Multi-Agent Pipeline Works

A central orchestrator agent coordinates specialized sub-agents through a six-step workflow. Each sub-agent is defined by a structured prompt specifying its role, permitted tools—such as file read/write, shell execution and web access—and expected output schema.

The orchestrator dispatches sub-agents sequentially across pipeline stages and in parallel within stages when multiple tutorials are processed concurrently. Data flows between stages through standardized JSON reports and file conventions.

1. 1. Locate and Download the Research Codebase

   Paper2Agent first attempts to identify the associated code repository from the manuscript, references or supplementary materials. If automatic identification fails, returns multiple candidates or the user prefers a specific repository, the repository URL can be provided directly.

   Once identified, the repository is cloned or downloaded together with associated resources, including supplementary data and configuration files. The outputs are the cloned repository and detected programming language.

2. 2. Set Up an Isolated Environment

   The environment manager sub-agent creates a clean, isolated virtual environment for the repository. Its input is the cloned repository, and its outputs are the isolated environment and test configuration files.

3. 3. Discover Tutorials and Educational Resources

   The tutorial scanner sub-agent reviews the repository to locate useful reference and educational materials. It identifies candidate tutorials, classifies relevant files and produces a JSON index for downstream tooling.

4. 4. Execute and Audit Tutorials

   The tutorial executor runs selected tutorials end to end with their example data. It captures inputs, outputs, figures and runtime constraints while recording implicit assumptions that should be made explicit.

   The inputs are tutorial source files, the activated virtual environment and the scanner report. The outputs are executed notebooks and per-tutorial execution reports.

5. 5. Extract, Test and Refine MCP Tools

   This stage uses two sub-agents in sequence:

   • Tutorial tool extractor–implementor: Converts each executed tutorial into a standalone Python module containing reusable functions. It identifies generalizable analysis steps, parameterizes hard-coded values such as file paths, thresholds and column names, enforces file-based inputs and outputs, and decorates each function as an MCP tool.
   • Test verifier–improver: Creates per-function test files using the tutorial’s example data as ground truth. Tests verify that expected output files are generated. Functions that repeatedly fail have their MCP tool decorators removed and are excluded from the final server.

   The inputs are executed notebooks, the virtual environment and the scanner report. The outputs are tool modules, per-function test files, test logs and summaries.

6. 6. Assemble the MCP Server

   The orchestrator integrates validated tool modules into a unified MCP server with a manifest, versioning and basic security defaults. The resulting server is ready to be used by an orchestrator or co-scientist agent.

Specialized Paper2Agent Sub-Agents

Each sub-agent is instantiated as an independent Claude LLM session with a role-specific system prompt and a defined set of permitted tools, including file read/write, shell execution and code search.

Environment Manager

The environment manager creates clean, reproducible environments for research codebases. It analyzes project setup requirements, provisions an isolated workspace, installs dependencies and verifies that the code runs without conflicts. Standardized environment setup improves reliability and reproducibility across systems.

Tutorial Scanner

The tutorial scanner reviews public codebases to identify and organize educational resources. It distinguishes genuine tutorials from other files, highlights resources suitable for reuse and produces summaries showing which materials should be retained or set aside.

Tutorial Executor

The tutorial executor runs approved tutorials end to end to generate gold-standard outputs and reference data for tool extraction. It resolves execution errors, preserves numerical results, figures, tables and generated data files, and records execution metadata. These outputs serve as authoritative references for test creation and validation.

Tutorial Tool Extractor–Implementor

This sub-agent converts tutorials into reusable tools. It identifies tasks that generalize beyond example data and implements each as a clean, single-purpose function with clear inputs, outputs and defaults.

It parameterizes hard-coded values, enforces file-based inputs, saves essential results and figures, and returns a standardized summary of generated artifacts. The objective is to reproduce tutorial results on the original data while preparing the functions for new datasets.

Test Verifier–Improver

The test verifier–improver creates, runs and refines tests for tutorial implementations. It uses the tutorials’ own examples to evaluate numerical and visualization results.

A test passes when expected files are generated, numerical results match tutorial outputs exactly with a 3% tolerance for floating-point values, and generated figures match reference visualizations using perceptual hashing with a Hamming distance below 20.

The agent iteratively generates tests, executes them, diagnoses failures and applies fixes, with a maximum of six attempts per function. If a function repeatedly fails, its MCP decorator is removed, a failure comment is added and the function is excluded from the MCP server. Results and logs are recorded for transparency.

AlphaGenome Agent: From Research Paper to MCP Server

We applied Paper2Agent to the AlphaGenome paper to generate an AlphaGenome MCP server and connected it with Claude Code to create the AlphaGenome agent. The generated server is remotely hosted on Hugging Face Spaces.

To assess reproducibility, the AlphaGenome agent was evaluated with 15 original tutorial-based queries and 15 novel queries. Agent responses were compared with ground-truth answers.

One evaluation prompt asked the agent to interpret an LDL genetic association:

Use AlphaGenome to interpret why chr1:109274968:G>T associates with LDL cholesterol. Identify the causal genes and assess regulatory effects across modalities in liver. Generate a publication-ready report with figures. My AlphaGenome API key is: <API_KEY>. Reason step by step.

Detailed benchmark queries are available in the Paper2Agent GitHub repository.

Benchmarking AlphaGenome Against Claude + Repo and Biomni

For the tutorial-based and novel AlphaGenome benchmarks, evaluation followed four general steps:

1. Generate ground-truth answers for each query using manually curated and executed code.
2. Capture the agent’s response together with performance metrics such as runtime and cost.
3. Manually review and grade the response against the ground truth.
4. Summarize performance across the benchmark dataset.

Unless otherwise specified, evaluations used claude-sonnet-4-20250514 as the underlying LLM. Evaluations ran locally on a MacBook Air with an M2 chip, an 8-core CPU, an 8-core GPU and 8 GB of unified memory. Model APIs were used as needed.

Each query ran in non-interactive command-line mode, and all output was captured in JSON format. For example:

claude --model "claude-sonnet-4-20250514" --print --output-format "json"

The 30 open-ended AlphaGenome queries required the agent to:

1. Independently formulate an analysis plan.
2. Compose multiple tool calls, such as comparing variant-effect predictions across tissues, integrating motif and QTL evidence, or evaluating candidate variants.
3. Synthesize the results into a biological conclusion.

Two domain experts scored the queries using a predefined rubric based on key-entity matching, including genes, variants, tissues and biological conclusions.

AlphaGenome Agent Evaluation Setup

Each benchmark query for the Paper2Agent-generated AlphaGenome agent was wrapped in a system prompt instructing the agent to use available AlphaGenome MCP tools, return a structured JSON response containing the final answer and step-by-step reasoning, and retrieve the API key from the project environment. The agent received no additional context beyond the MCP tool definitions and query.

Claude + Repo Baseline

For the Claude + Repo baseline, Claude Code had access to a local clone of the AlphaGenome repository. The system prompt instructed the agent to write and execute Python code using the AlphaGenome library while prohibiting it from copying answers from tutorial notebooks or documentation. The agent returned a structured JSON response containing the final answer and executed code.

Biomni Baseline

We used the API-based version of Biomni. Its system prompt directed the agent to the AlphaGenome repository and API key and required a structured JSON response. Full prompt templates are provided in the Supplementary Note. Benchmarking tools and analyses are available in the Paper2Agent repository.

Generation and Analysis of the TISSUE Agent

We applied Paper2Agent to the TISSUE paper to generate a TISSUE MCP server and connected it with Claude Code to create the TISSUE agent.

To assess reproducibility, we compared TISSUE agent outputs with analyses performed by human researchers using the same mouse somatosensory cortex spatial transcriptomics data. The human researchers followed the tutorial in the TISSUE GitHub repository.

Generation and Analysis of the Scanpy Agent

Paper2Agent was applied to the Scanpy software package to generate a Scanpy agent focused on preprocessing and clustering workflows for single-cell RNA sequencing analysis. The resulting MCP server was deployed and integrated with Claude Code.

To construct the workflow in MCP prompts, we used the following instruction:

Based on the tools you have, construct an MCP prompt to replicate the tutorial in the correct order. Always inspect the data first, and only deviate from the default settings if adhering to them would yield incorrect results.

This ensured that the generated prompts encoded the standard Scanpy preprocessing and clustering pipeline in a reproducible and interpretable way.

Reproducibility was evaluated by comparing agent outputs with results obtained by human researchers following official Scanpy reference tutorials. Three publicly available 10x Genomics PBMC single-cell RNA-seq datasets and four additional datasets were used for benchmarking.

Across these datasets, the agent reproduced key workflow steps, including:

• Gene filtering
• Normalization
• Principal component analysis
• Neighborhood graph construction
• Clustering

The resulting analyses were consistent with human-executed workflows.

Large-Scale Evaluation of Paper2Agent

To evaluate generalizability, scalability and robustness, we processed three corpora end to end with Paper2Agent without manual cleanup, code modification or intervention.

1. Computational biology papers: 100 papers retrospectively sampled from the bioRxiv bioinformatics category by iterating backward chronologically from December 2025. The sample was not filtered for documentation quality, repository maintenance or code completeness, reflecting the natural heterogeneity of research code.
2. Data- and discovery-focused papers: 26 papers—13 from bioRxiv and 13 from Nature in 2025—reporting experimental results, datasets or discoveries with accompanying supplementary materials. These papers evaluated Paper2Agent’s structured resource layer across manuscript text, supplementary files and metadata.
3. Non-biology computational papers: 10 papers spanning diverse domains and programming paradigms: grf, SAELens, Binoculars, SAM2, TabPFN, GenericML, CausalImpact, Nashpy, emcee and conformal-selection.

Successful agentification was defined as the generated MCP server completing tool extraction, execution and automated validation end to end without human intervention.

Benchmark Questions and Ground Truth

For the 74 successfully agentified computational biology papers, we derived 300 tutorial-based benchmark questions. Ground-truth answers were obtained by executing the original code and verifying results against tutorial outputs.

For the 26 data- and discovery-focused papers, we curated 100 synthesis-based questions requiring integration across the main text and supplementary materials. These included reinterpretation tasks, such as reanalyzing conclusions using Spearman correlation when the paper originally reported Pearson correlation, and cross-referencing across tables, figures and narrative text.

For the 10 non-biology computational papers, we constructed 42 execution-based benchmark questions. Detailed prompts are provided in the Supplementary Note.

Baselines and Evaluation Metrics

For computational biology papers, the primary baseline was Claude Code with direct repository access, referred to as Claude + Repo. The agent received the full code repository and paper but no MCP tools, structured resources or prompts. It answered the same queries by writing and executing code. Biomni was not included because of its high cost.

Primary evaluations used Sonnet 4, claude-sonnet-4-20250514. An additional Claude + Repo baseline used Sonnet 4.6 on the same 300 questions.

For data- and discovery-focused papers, the baseline was Claude with browser-use capabilities and direct access to the paper URL, representing a human-assisted LLM setup that could browse and read the paper directly.

We report mean accuracy ± s.e.m. using a bootstrap procedure. Comparisons with baselines used paired t-tests on per-run accuracy for tutorial-based benchmarks and bootstrap hypothesis tests with 10,000 resamples for the 100-paper evaluation. Accuracy differences are reported with 95% confidence intervals.

API costs were tracked by logging all LLM API calls during MCP construction and downstream question answering. For MCP construction, we report total cost—the sum of all sub-agent API calls—and the time from pipeline initiation to MCP server creation. For query-time evaluation, cost and latency were measured per query, from submission to the final answer.

False-Positive and Hallucination Evaluation

To evaluate false-positive behavior, we created an adversarial out-of-scope benchmark by randomly permuting paper–question pairs across the 26 data- and discovery-focused papers. Each question was paired with a paper that did not contain the relevant information.

Paper2Agent was evaluated under two conditions:

1. With an explicit rejection instruction: “If the question is not related to the files, say ‘I don’t know’.”
2. Without an explicit rejection instruction.

We measured the correct rejection rate, defined as the fraction of out-of-scope queries for which the agent declined to answer instead of producing a hallucinated response.

Testing for Tutorial-Specific Memorization

We inspected generated MCP server implementations across the 100 computational biology papers for evidence of tutorial-specific memorization. The review examined extracted tools for:

• Hard-coded tutorial constants
• Fixed file paths
• Cached outputs
• Dataset-dependent heuristics

We also implemented an automated reviewer agent that scans generated MCP server Python files for hard-coded values, fixed dataset paths, cached outputs and tutorial-specific logic. This provided a systematic implementation-level check for shortcut learning in addition to manual inspection.

Ablation Studies

We evaluated five ablated variants of the full system using AlphaGenome and 30 tutorial and novel benchmark questions.

• Monolithic agent: All tasks—including environment setup, tutorial scanning, tool extraction and testing—were performed by one agent in a single 200,000-token context window without specialized sub-agents.
• Non-parallel multi-agent system: The same four sub-agents were used but forced to run sequentially, isolating the contribution of parallel orchestration to runtime efficiency.
• No test verifier–improver: The test verifier–improver was removed, and extracted tools were deployed without iterative validation against tutorial outputs.
• Markdown skill files: MCP tools were replaced with Markdown skill files using Claude Code’s Skills feature. These files encode tool-use instructions as structured text rather than executable tools.
• Alternative scaffolding: The Claude Code backend was replaced with OpenCode to test whether MCP construction quality depended on the agent scaffolding.

All ablation evaluations used claude-sonnet-4-20250514 as the base model.

Robustness to Upstream Code Defects

To evaluate robustness to upstream code defects, we created adversarial variants of three representative repositories:

• AlphaGenome, a Python notebook-based repository
• POP-TOOLS, a Python command-line repository
• mlearner, an R command-line repository

For each repository, we independently injected four categories of execution-level errors:

1. Missing dependencies: Required packages were removed from environment specification files such as requirements.txt, DESCRIPTION or README installation instructions.
2. Broken file paths: Input paths were changed to nonexistent directories or filenames.
3. Typographical errors: Function names, package names or variable names were misspelled in executable code cells or scripts.
4. Deprecated API calls: Valid function calls were replaced with deprecated or incompatible alternatives.

These changes produced 12 adversarial configurations in total. Modifications were applied before Paper2Agent was run and were not disclosed to the agent.

Paper2Agent was tasked with running the standard agentification pipeline. We recorded whether it detected the injected errors, the repair strategies it used and whether the final MCP server passed all validation tests. Detailed examples of injected errors and observed behavior are provided in the Supplementary Note.

Are Executable Tutorials Required?

To assess whether executable tutorials are necessary, we removed all executable tutorials from POP-TOOLS while retaining the README and source code. We then ran the standard Paper2Agent pipeline and assessed whether the resulting MCP server exposed functional tools and reproduced human-executed POP-TOOLS command-line outputs across five analysis tasks.

AI Co-Scientist Analysis of the rs887314 Psoriasis Locus

We used three Paper2Agent-generated agents to prioritize and validate candidate causal genes associated with the psoriasis risk variant rs887314:

• An AlphaGenome agent for computational variant-effect prediction
• An MPRA-coupled scCRISPRi agent providing cis-regulatory element perturbation effects on gene expression in primary human CD4+ T cells
• A CD4+ T-cell Perturb-seq agent providing transcriptome-wide gene-expression profiles after individual gene knockdowns under three culture conditions

The CD4+ T-cell Perturb-seq dataset included:

• Rest: Cells maintained without restimulation and harvested after 8 hours.
• Stim8hr: Cells restimulated with 12.5 µl ml⁻¹ ImmunoCult human CD3/CD28/CD2 T-cell activator and harvested after 8 hours.
• Stim48hr: Cells restimulated with the same activator and harvested after 48 hours.

All agents exposed their underlying datasets, metadata and supplementary tables as structured, queryable resources.

The AlphaGenome agent was prompted to score rs887314 across all available prediction modalities, with analyses restricted to CD4+ T cells. For each gene within the local genomic window surrounding rs887314, the agent returned a predicted expression-impact score, and genes were ranked according to those predicted effects.

We then instructed the AI co-scientist, under human-in-the-loop supervision, to use the MPRA scCRISPRi and CD4+ T-cell Perturb-seq papers and data. Full prompts and documentation of human interventions are provided in the Supplementary Note.

Paper2Agent Agent Availability

The Paper2Agent-generated AlphaGenome agent is publicly available on Hugging Face Spaces.

The Paper2Agent framework, benchmark questions and GitHub repositories are available in the Paper2Agent repository.

Reporting Summary

Further information on the research design is available in the Nature Portfolio Reporting Summary linked to this article.