How a Multi-Agent AI Health Assistant Uses Wearable Data, Medical Knowledge and Health Coaching

This study evaluates a multi-agent personal health assistant (PHA) designed to answer personalized health questions using wearable data, medical records, medical knowledge and behavioural coaching. The system combines three specialized agents: a data science (DS) agent, a domain expertise (DE) agent and a health coaching (HC) agent. A central orchestrator assigns tasks, coordinates the agents and reviews their responses before producing a final answer.

Real-world wearable and health data

To evaluate the framework in a real-world setting, the researchers used the WEAR-ME study, a large-scale study of metabolic health involving 1,165 participants⁴³. Participants consented to link their Fitbit accounts to Google Health Studies, allowing wearable data to be collected during the study and for up to three months before enrollment.

After joining the study, participants were asked to:

Complete questionnaires about demographics, health history and personal health insights.
Schedule a blood draw at a Quest Patient Service Center.
Wear a Fitbit or Pixel Watch during the day and night for at least three out of every four days.

The resulting WEAR-ME dataset includes questionnaire responses, Fitbit records and blood test results for each participant. Further dataset information is provided in Supplementary Section B.2.

The WEAR-ME study complied with privacy regulations and was approved by the Advarra Institutional Review Board (IRB #Pro 00074093). All participants provided informed electronic consent for their data to be used in research and publications.

The dataset supported several evaluation tasks, including data science, domain expertise, health coaching and multi-agent personal health assistant assessments. Working with clinicians, the researchers identified the ten most common health profiles among participants. They then randomly selected five participants representing each profile, creating 50 user personas. These personas were used to prepare evaluation samples for the DS, DE, HC and PHA agents. The ten profiles and persona examples are provided in Supplementary Section B.2.

Choice of base language model

The experiments used the Gemini 2 family of models, including Gemini Flash and Gemini Pro⁴⁴. Gemini has demonstrated strong performance on medical tasks^8,45,46. To test whether the proposed architecture was independent of a particular model, the researchers also evaluated the agents using OpenAI GPT and Anthropic Claude models.

The aim of the study was not to train a new large language model (LLM), but to design an agentic system around an existing base model. Gemini was selected because it enabled the researchers to verify that:

The evaluation data had not contaminated the model’s training data.
Real participant health data would not be logged by the model or used to train future models.

The sub-agent and team designs were intended to be model agnostic. The researchers therefore expect the proposed methods to generalize to other LLM families.

User-centred design of the personal health assistant

The PHA was designed using three complementary sources of information about end-user needs:

Real-world consumer health queries.
Survey responses about user goals, satisfaction and engagement.
Expert workshops used to organize the findings into actionable requirements.

Together, these sources formed a user-centred design pipeline. Naturalistic health questions captured users’ interests, structured surveys explored their goals and experiences, and expert analysis converted the findings into requirements for the AI health assistant.

Consumer health queries

The researchers compiled consumer health queries from Google Search, Gemini and the Fitbit community forum between January 2024 and February 2025. From October 2024 through February 2025, they also launched Insights Explorer through Fitbit Labs⁴⁹.

Participation in Insights Explorer was voluntary. Users had to opt in and provide in-app consent for their data to be used for research and development. Participants could ask health questions and receive responses from a prototype generative AI conversational agent.

The researchers assembled 1,370 de-identified queries covering a broad range of everyday health and wellness topics. Examples are listed in Supplementary Table 1.

Fitbit Labs survey data

To complement the query data, the researchers collected survey responses from 555 Insights Explorer Fitbit Labs participants. The survey examined:

Personal health goals and desired outcomes.
Health and wellness topics explored with the prototype.
User experience and satisfaction.
Expectations that were or were not met.
Engagement, continued use and perceived benefits.
Previous experience with wearable sensing, AI and conversational agents.

Further details of the survey are provided in Supplementary Table 1.

Expert workshop

The researchers conducted a one-hour critical user journey workshop with 14 experts from user experience, product, research and engineering. The workshop focused on summarizing users’ goals and the tasks associated with each goal.

Two researchers independently coded the same subset of queries and survey data. They jointly developed a hierarchical codebook of user journeys, including categories and subcategories. The researchers then categorized the remaining queries and expanded the codebook when necessary using a standard thematic analysis process⁵⁰.

The analysis identified four core user journeys:

Interpreting personal data.
Seeking health knowledge.
Receiving wellness advice.
Assessing symptoms.

These varied needs require more than one monolithic AI system. The researchers therefore developed a multi-agent personal health assistant with three specialized roles: data analysis, health domain expertise and behavioural coaching.

The design was influenced by the theory of modular cognition⁵¹, which proposes that the mind uses specialized subsystems for different tasks, as well as by the structure of professional health teams in which experts with complementary skills work together.

Data science agent development

The DS agent answers questions about personal and population-level health data. Previous research has investigated LLMs for data science tasks^2,28–35, but the ability of these systems to interpret open-ended, personalized wearable-data questions and produce statistically sound analyses remains uncertain.

The researchers developed what they describe as a personal, open-ended data science agent capable of analyzing personal and population-level time-series wearable data and producing numerical health insights. The agent allows users to ask personalized questions about their health data and provides data-driven responses. An evaluation benchmark was also open-sourced.

The DS agent focuses on capabilities that are useful and tractable to evaluate:

Interpreting numerical data from wearable streams and medical records.
Aligning statistical results with the user’s question and context.
Accounting for timeframes, statistical methods and analytical assumptions.
Recognizing whether the available data is sufficient for a reliable conclusion.

Turning open-ended questions into analysis plans

For broad or ambiguous questions, the DS agent converts the user’s request into a specific, unambiguous and feasible analysis plan. This involves:

Identifying conceptual variables.
The agent identifies concepts in the question that should be represented in the data. For example, “Have I become fitter over the past few months?” might be represented using average daily steps and resting heart rate.
Defining data transformations.
The agent determines how to operationalize variables across personal and population-level tables, including:
1. Filtering relevant periods, such as activities from the previous three months.
2. Cleaning missing or incorrectly recorded wearable data.
3. Joining tables, such as activity and daily summary records by date.
4. Aggregating data, such as calculating monthly average steps and resting heart rate.
Checking data sufficiency.
The agent identifies missing or insufficient information that could make the analysis unreliable.
Selecting statistical methods.
The agent applies tests and calculations that reflect the user’s intent.

Because open-ended questions can have more than one valid interpretation, the DS agent is designed to provide at least one appropriate answer rather than list every possible analysis. Users can continue the conversation to explore alternative interpretations. If required data is unavailable, the agent can check with the user and request additional information.

Code generation, execution and feedback

The DS agent uses a two-stage architecture:

Analysis plan generation.
Code generation followed by iterative execution and refinement.

This separation distinguishes the semantic question of what to do from the technical question of how to implement it. The approach is intended to reduce factual and logical errors that can occur when an LLM attempts to complete a complex analysis from start to finish^53–57.

In the first stage, the planner receives a summary of the available data, including table schemas, column descriptions, data types, semantic tags and summary statistics. It then produces a structured plan describing the conceptual variables, filters, joins, aggregations and statistical model.

In the second stage, the agent generates executable code based on the plan. The code runs in a sandboxed environment. If execution fails because of a syntax error or runtime exception, the agent receives the original question, analysis plan and error message and attempts to correct the code. This iterative process continues until the code executes successfully and produces a numerical result.

The DS agent architecture is illustrated in Fig. 2a, with additional prompt and implementation details in Supplementary Section C.2.

Domain expertise agent development

The DE agent addresses medical knowledge questions and daily symptoms. Although earlier work has evaluated the medical knowledge of LLMs^36–38, important capabilities of a health-domain expert remain underexplored. These include interpreting health data in environmental and population contexts, summarizing information across modalities and reasoning over wearable data alongside medical records.

The DE agent combines wearable data, medical records and contextual information to answer complex health questions. Its purpose is to synthesize, contextualize and validate information against authoritative sources and user-specific data rather than simply retrieve facts.

The researchers defined four core DE capabilities:

Health information accuracy.
The agent should provide reliable medical and wellness information across personal health and clinical domains.
Diagnostic reasoning.
The agent should generate and rank plausible diagnoses using symptoms and relevant medical history shared during a conversation.
Contextualization and personalization.
The agent should adapt health information to the user’s demographics, existing conditions and health goals.
Multimodal health-data synthesis.
The agent should integrate wearable measures, laboratory results, medical history and other health information into coherent summaries.

Tool use and iterative health reasoning

The DE agent uses a multi-step planning, reasoning and acting architecture, as shown in Fig. 3a. It receives a user question, wearable data and medical records. Available tools include Web Search, the NCBI API and the Data Commons API. A Python Sandbox can also execute API calls.

The agent follows an iterative “reason, investigate and examine” process:

It decomposes the health question into verifiable steps, such as establishing a clinical baseline or identifying relevant medical literature.
It selects tools to investigate the question, such as Data Commons for population statistics or NCBI for medical studies.
It examines the results and uses them to determine the next step.
It repeats the cycle until sufficient evidence has been synthesized into a comprehensive response.

This workflow creates a traceable reasoning process. By combining authoritative information with personal health data, the agent is designed to produce grounded, personalized health insights rather than an unsupported black-box response. The researchers note that these capabilities were studied in a controlled academic setting and do not represent a real-world product subject to a full regulatory assessment.

Health coaching agent development

The HC agent provides personalized health and wellness advice and helps users define goals and motivation through conversation. Earlier studies have explored AI health coaching^39–41, but the researchers aimed to support open-ended coaching across a broad range of wellness topics.

The HC agent incorporates motivational interviewing and personalized recommendations. Its modular structure balances information gathering, active listening and advice delivery. Example conversations are provided in Extended Data Figs. 3 and 4.

Expert input for the coaching design

The researchers conducted formative interviews with 11 health experts, including six men and five women aged 24–45. The experts represented fields such as sleep, fitness and life coaching. Each had an advanced degree, at least five years of professional experience and knowledge of health-coaching best practices.

The semi-structured interviews lasted approximately one hour and examined effective health-coaching behaviours, including the relationship between conversation content and conversational style. Two authors conducted thematic analysis, independently developed theme codebooks and iterated until reaching convergence.

The resulting HC capabilities included:

Goal and purpose identification: understanding the user’s goal and motivation, maintaining a goal-oriented conversation and returning to the goal when the conversation drifts.
Active listening: summarizing and paraphrasing the conversation to confirm understanding and goal alignment.
Context clarification: identifying constraints, preferences and previously attempted solutions.
User empowerment: building trust, supporting independent decision-making and reinforcing positive progress.
Personalized recommendations: providing specific, measurable, attainable, relevant and time-bound recommendations based on SMART goal principles⁵⁹.
Feedback incorporation: offering initial recommendations while inviting feedback and updating the advice in response.

Conversation flow modules

Single-prompt systems can struggle to balance questioning and advice. During pilot testing, some systems offered recommendations before gathering enough context, while others continued asking questions without transitioning to actionable guidance.

To address these problems, the researchers added two specialized modules:

Recommendation module: determines when enough context has been gathered, keeps the conversation focused and confirms when it is appropriate to provide advice.
Conclusion module: identifies a natural endpoint, summarizes the interaction and reinforces key takeaways.

The final HC architecture combines a personalized coaching module with the recommendation and conclusion modules. The coaching module is inspired by motivational interviewing⁶⁰ and guides the agent to identify the user’s goal, explore constraints, request feedback and then provide recommendations.

Personal health assistant: multi-agent collaboration

The DS, DE and HC agents have complementary capabilities and can collaborate on the same user question. A central orchestrator receives the initial request, identifies the user’s needs, assigns subtasks to one or more agents, combines their outputs and reviews the response before it reaches the user.

This approach reflects modular cognition and the structure of human expert teams. Treating each agent as a specialized cognitive module can support broader coverage of health needs and make the final response more interpretable by linking outputs to specific functions.

Four principles of the PHA design

Address comprehensive user needs.
Users ask about data interpretation, health information, wellness and symptoms. A single agent may not cover all these needs effectively.
Provide adaptive support.
Different questions require different combinations of agents. The system should dynamically assign agents rather than rely on a fixed architecture.
Reduce user burden.
The system should use available wearable data and medical records instead of asking users to provide information that can already be calculated. For example, it should calculate sleep duration from wearable records when possible.
Maintain architectural simplicity.
When agents have similar capabilities, a simpler design can reduce latency and computational cost and make maintenance easier.

PHA workflow

The PHA consists of a Gemini-based orchestrator and three specialist agents: the DS agent, the DE agent and the HC agent. The orchestrator uses a sequential and iterative workflow with four main stages.

User-need understanding.
The orchestrator analyzes the question and infers the user’s primary goal, aligning it with one or more of the four core user journeys.
Agent orchestration and task assignment.
It assigns a main agent and determines whether supporting agents are needed. For example, a question about improving sleep based on last week’s data may assign the HC agent as the main agent and the DS agent to analyze sleep patterns first.
Query reflection.
The main agent reviews the combined outputs for coherence, completeness and accuracy. This process helps prevent the system from requesting information that is already available.
Memory update.
After each interaction, persistent memory records important entities, generated insights, user goals, barriers and preferences. This supports continuity and personalization across conversations.

Agent evaluation framework

The researchers created a mixed-method evaluation framework to identify the strengths and weaknesses of each agent. The framework examined:

Single-turn and multi-turn interactions.
Single-modality and multimodality data processing.
Automatic and human evaluation.
Feedback from health experts and end users.

The framework evaluated the following capabilities:

DS agent: analysis-plan generation and accurate executable code.
DE agent: health questions, diagnostic conversations, personalized responses and multimodal health-data reasoning.
HC agent: end-user coaching effectiveness and adherence to expert coaching principles.
PHA: holistic conversational quality from end-user and expert perspectives.

DS agent evaluation

The DS agent was evaluated in two tasks designed to measure statistical robustness and numerical accuracy.

Task 1: analysis-plan generation

The first task measured whether the agent could convert an open-ended question, such as “Am I getting fitter recently?”, into a precise statistical analysis plan aligned with the user’s intent.

The evaluation used wearable data from WEAR-ME, including:

A daily summary table containing activity and sleep measures.
An activities table containing recorded physical activities such as running, biking and swimming.
A population summary table containing daily wearable percentiles stratified by gender and age group.

Because analysis quality depends on decisions about missing data, statistical models and variable definitions, the researchers developed an LLM autorater and tested it against human expert assessments.

Analysis-plan rubric

Four data science experts developed a rubric covering six dimensions:

Timeframe: whether the analysis uses an appropriate period.
Data transformations: whether columns and transformations are used correctly and reproducibly.
Data sufficiency: whether enough rows and non-missing values are available.
Summary statistics: whether the selected statistics suit the question.
Statistical tests: whether the tests are appropriate and account for distributional assumptions.
Overall alignment: whether the analysis addresses the user’s intent.

The final rubric contained 26 items arranged in a conditional hierarchy. Some items applied only when an earlier criterion was satisfied. For example, the appropriateness of a statistical test was assessed only when the underlying hypothesis was sound. Errors and omissions reduced the score, with a maximum deduction of 16 points per evaluation.

Query–analysis plan benchmark

The researchers created an expert-rated benchmark containing 141 unique query–analysis plan pairs generated from 100 health questions. The questions were sampled from the end-user query set, and the plans were generated using Gemini 1.5–2.0 and GPT-4o.

Ten expert data scientists provided 354 approach annotations. Each pair was assessed by two to four experts, producing 6,157 rubric-item annotations. The experts had PhDs in computer science and five to 20 years of experience. Average agreement was 80.2%, with a Bennett’s S of 0.622⁷⁰.

LLM autorater validation

The researchers developed and validated an autorater using Gemini 2.0 and 2.5 Flash. The test set included 96 evaluations covering 25 unique queries, while the development set included 75 queries. The final autorater achieved an intraclass correlation coefficient of 0.838 against human expert scores, indicating high reliability for scaled evaluation⁶⁸.

The DS agent’s analysis plans were then compared with those produced by the baseline Gemini 2.0 Flash model.

Task 2: code generation and execution

The second DS evaluation measured whether the agent could convert a precisely specified analysis plan into executable, functionally correct and robust code. The evaluation included limited data, complex transformations, appropriate statistical methods and accurate answers to user questions.

Experts converted analysis plans into function headers and docstrings. The implementations were written in Python using libraries including NumPy, pandas and SciPy and were tested in a sandbox environment.

Seven data scientists created 173 unit tests across 25 query–approach pairs. The experts spent approximately 75 hours developing the coding problems, implementations and tests. The test suites assessed executable code, logical correctness, edge-case handling, transformations, statistical methods and numerical accuracy.

The pass rate was calculated as the number of successful tests divided by the 173 total tests. If code failed, the model received the error message and could retry up to five times.

Model-agnostic DS evaluation

To determine whether performance improvements depended on Gemini, the researchers ran the same DS architecture and prompts on GPT and Claude models. Both analysis-plan generation and code generation were evaluated using identical queries, rubrics, autoraters and unit-test suites.

For each base model, the researchers compared the model alone with the same model enhanced by the DS agent architecture. The purpose was to test whether the framework generalized beyond Gemini rather than claim a new peak result for another LLM.

DE agent evaluation

The DE agent was evaluated across four capabilities using independent tasks.

Task 1: health knowledge and factuality

The first task measured the agent’s ability to answer nuanced medical and wellness questions using expert-level multiple-choice examinations. The selected domains were endocrinology, cardiology, fitness and sleep medicine.

These areas reflect common health needs, including chronic disease management and high-interest preventive topics. Their case-based questions also test reasoning beyond simple fact retrieval.

The evaluation used more than 2,000 questions:

Endocrinology: 570 StatPearls questions from the American Board of Internal Medicine Endocrinology, Diabetes and Metabolism examination preparation materials, including 173 expert, 198 difficult and 199 moderate or easy questions⁷².
Cardiology: 399 questions created from BoardVital’s ABIM-based preparation bank, including 100 hard, 199 moderate and 100 easy questions⁷³.
Fitness coaching: 542 questions compiled by Khasentino et al. from Certified Strength and Conditioning Specialist-style question banks⁸.
Sleep medicine: 634 questions from BoardVitals sleep medicine Maintenance of Certification review banks⁸.

The DE agent was compared with the same Gemini backbone operating without the agentic framework.

Task 2: diagnostic conversations

This task examined differential diagnosis through conversational reasoning. Rather than focusing on complex clinical case studies, the researchers used common symptoms that people might report themselves.

A retrospective observational study collected self-reported health cases from 2,000 consumers aged 18–65. The study was approved by the Advarra IRB under protocol #GH-SCD-001. Participants had experienced symptoms from one of 39 predefined categories during the previous three months and completed an online survey describing their symptoms, prior health events, online information-seeking behaviour and eventual diagnosis.

The DE agent was compared with a reimplemented Gemini-based state-of-the-art differential-diagnosis agent⁴. Each system received the symptom descriptions and generated the ten most likely diagnoses. A separate Gemini 2.5 Pro autorater compared the predictions with the reported diagnoses.

Task 3: contextualized and personalized health responses

This evaluation measured whether the DE agent could tailor medical answers to a user’s circumstances. The researchers created 50 health questions designed to elicit personalized responses. Each question included information from one or both of the following categories:

Demographic information: age, biological sex and race or ethnicity.
Existing health conditions: diabetes, cardiovascular disease and impaired kidney, thyroid or liver function.

The DE agent’s responses were compared with responses from Gemini 2.0 Pro without the agentic framework. All responses were normalized to a standard format to reduce presentation bias.

Seventeen end users aged 25–50 evaluated the responses in randomized, blinded order. They spent a total of 13 hours assessing personalization, relevance, credibility and trustworthiness using Boolean rubric questions and side-by-side comparisons.

Task 4: multimodal personal health-data reasoning

The fourth DE task tested whether the agent could combine wearable data and medical records into a comprehensive health summary without receiving a specific user question or chief complaint.

The input included:

Unstructured text: contextual and background information from participant onboarding surveys.
Structured tables: wearable measures such as resting heart rate and heart-rate variability.
Laboratory documents: participant laboratory results converted into PDF files.

The researchers sampled 30 participants from the persona pool, with equal representation across ten health profiles. The DE agent and baseline Gemini model generated a health summary for each participant using the complete data profile.

Five expert clinicians independently evaluated the summaries in a blinded, side-by-side comparison. They spent a total of 77 hours assessing clinical relevance, interpretation accuracy and actionability using a detailed rubric.

HC agent evaluation

The HC agent was evaluated using real-world-style conversations and feedback from both end users and coaching experts.

Task 1: coaching capabilities from end-user perspectives

The researchers assessed whether users found the HC agent engaging, competent and preferable to a baseline. Measures included agent rankings, satisfaction, engagement, conversation endings, coaching capabilities, conversational style and the actionability of recommendations.

Thirty-one participants, including 25 men and six women aged 18–64, completed 1.5-hour sessions. Participants interacted with multiple agents in a blinded, counterbalanced order using a Latin square design. The comparison focused on the HC agent and a baseline Gemini system instructed to act as a helpful personal coach. No significant difference in response time was observed between the systems.

Participants selected an open-ended scenario that matched their personal interests. Each conversation began with the same opening sentence for the selected scenario, after which participants continued freely until they reached a natural resolution or ended an unproductive interaction.

After each interaction, participants completed assessments based on established instruments, including the CUQ, WAI, CEMI and SMART goal-setting measures^59,77–79. These assessed usability, collaborative rapport, motivational techniques, goal formulation and the quality of the coaching experience.

Task 2: expert validation of the coaching design

Three health experts from the formative interview study reviewed the conversation transcripts while blinded to the agent condition. They rated the six coaching themes, as well as CUQ, WAI, CEMI and SMART measures.

The experts also used an AICM developed by the health-coaching team to assess supportiveness, friendliness, clarification, rapport development, information gathering, personalized intervention and appropriate summarization. They provided comparative rankings for motivational interviewing, recommendation quality and credibility.

PHA evaluation

The complete PHA was compared with two baseline systems using the same Gemini backbone.

Baseline systems

Single-agent PHIA:
a reimplementation of a previous personal health insight architecture capable of code generation, information retrieval, online search, behavioural-data analysis and health recommendations². Although this system can make multiple LLM calls through a ReAct framework, it uses one agent to combine data science, domain expertise and coaching.
Parallel multi-agent system:
a static architecture in which the DS, DE and HC agents independently answer every user question in parallel. An orchestrator combines their responses. This approach uses the capabilities of all three agents but may not adapt to the user’s specific needs or benefit from sequential collaboration.

Task 1: end-user multi-agent experience

The first PHA evaluation measured broad, multi-turn health conversations from an end-user perspective. The researchers assessed helpfulness, coherence, empathy and the integration of data analysis, medical knowledge and health coaching.

To improve ecological validity, participants role-played WEAR-ME personas using real-world health profiles and data. Fifty personas represented ten common health profiles, with five personas per profile. The profiles ranged from healthy individuals training toward a goal to people managing conditions such as hypertension or diabetes.

Primary-care physicians helped develop background stories and health needs aligned with each persona’s data. All three systems received the same persona information so that performance differences reflected the agent architecture rather than access to data.

Live conversation protocol

Forty-five end users participated, including 17 women, 25 men and three participants who did not identify as male or female. Participants were aged 21–67 and represented a broad range of health conditions.

Each participant was assigned a matching WEAR-ME persona and held real-time, multi-turn conversations with all three systems. The systems were presented in counterbalanced order using a Latin square design, and participants were blinded to the agent conditions. Random response delays were introduced so that latency would not influence evaluations.

Conversations ended when the participant felt the persona’s needs had been addressed, the conversation reached a natural endpoint or 15 minutes had elapsed.

End-user evaluation

After each conversation, participants completed evaluations covering overall quality, data science, domain knowledge, health coaching and the integration of those capabilities. After using all three systems, they ranked them comparatively.

The evaluation produced 135 individual agent ratings and 45 comparative rankings. Participants spent a total of 90 hours on the study.

Task 2: expert multi-agent evaluation

Three health-coaching experts reviewed the live conversation logs while blinded to agent conditions. Their assessments covered the correctness of data analysis, clinical validity of health information, appropriateness and safety of coaching strategies, coordination between agents and overall conversational quality.

The experts assessed each system for every conversation and then ranked the three systems comparatively. The study generated approximately 140 hours of expert evaluation, 405 individual agent ratings and 135 comparative rankings.

Task 3: scaled synthesized dialogues

Live evaluations provide ecological validity but are limited by participant time and the number of conversations. To expand the evaluation, the researchers generated conversations between persona-instructed user models and each of the three systems.

A Gemini 2.5 Flash model was instructed to act as each WEAR-ME persona using its data, background and goals. It held three independent multi-turn conversations with each system, producing 150 conversation scripts across 50 personas and three systems.

Twenty generalist end users rated the scripts using the end-user rubric. This produced 3,000 individual ratings, 1,000 comparative rankings and approximately 500 hours of human effort.

Five health experts independently reviewed the same scripts using the expert rubric. Their evaluation produced 750 individual ratings, 250 comparative rankings and approximately 270 hours of human effort.

Computational cost

The researchers compared computational cost using LLM call count and response time per round in the scaled PHA evaluation:

Single-agent PHIA: 3.0 calls per round and 36.4 ± 1.6 seconds.
PHA: 6.5 calls per round and 205.3 ± 15.0 seconds.
Parallel multi-agent baseline: 7.0 calls per round and 244.4 ± 11.4 seconds.

The PHA required more computation than the single-agent system. However, dynamic task assignment and sequential collaboration required fewer calls and produced faster responses than the parallel multi-agent baseline, despite the additional reflection stage.

Statistical analysis

For comparisons involving three or more systems, the researchers fitted generalized linear mixed models with a log link. System was treated as a fixed effect, while rater and, when applicable, conversation script were treated as random effects.

Pairwise comparisons used Tukey’s honestly significant difference adjustment. Each rubric represented a prespecified and conceptually distinct quality dimension. An across-rubric Holm–Bonferroni robustness analysis was also conducted to assess overall type I error.

Comparative rankings were analyzed using a Friedman omnibus test followed by pairwise Wilcoxon signed-rank tests with Holm–Bonferroni correction. Effect sizes and confidence intervals were reported alongside P values when applicable. All tests were two-sided. Unless otherwise specified, error bars represented the standard error of the mean.

Ethics and participant privacy

The human-participant research was reviewed under three protocols:

WEAR-ME study: Advarra IRB #Pro 00074093.
Retrospective symptom study: Advarra IRB #GH-SCD-001.
HC agent and PHA user studies: WCG IRB #20244970.

All research involving human participants followed relevant ethical guidelines and regulations. Participants provided informed consent before taking part.

Reporting summary

Additional information about the research design is available in the Nature Portfolio Reporting Summary linked to the article.

Source: www.nature.com