Stanford University
Schmiedmayer LabDivision of Computational Medicine

Research

Multimodal and Agentic AI for Patient-Centered Care and Biomedical Research

Wearables capture everyday physiology and behavior, while patients’ interactions with care reveal symptoms, concerns, and changing priorities. We study how patient-facing agents can use these observations to explain changes in health between visits, and how AI co-scientists can test hypotheses across cohorts with analyses that researchers can inspect and reproduce.

Editorial collage of a wearable, a phone, and layers of clinical records.
Browse publications →

01Multimodal Computational Methods

How can language models reason jointly over wearable observations and clinical context?

Understanding changes in health requires methods for reasoning jointly over longitudinal time-series signals and clinical context.

OpenTSLM

Time series as input to language models

OpenTSLM encodes heart rate and oxygen saturation alongside text, then passes both to a language model to interpret the signals.
Langer et al. · Figure 1 ↗

Learning from signals and text

Treating time series as a native input lets models relate physiological patterns to clinical questions while retaining temporal information beyond predefined summaries.

Training models to describe these observations requires text grounded in the underlying signals. Our TimeCap work addresses this bottleneck through deterministic signal extraction and cross-channel synthesis, generating over seven million captions without a language model in the annotation process.

Wearable studies in cardiovascular health

60M+hours of wearable observations in OpenMHC

Our large-scale wearable studies, including My Heart Counts, provide the empirical foundation. Our open data release OpenMHC pairs wearable observations with reproducible benchmarks.

Through the DIGITS consortium, we will test how physiological models transfer across devices and sites. Shared standards and metadata representations will support reproducible multisite studies.

My Heart Counts demonstration showing study tasks and questionnaires.Study tasks Study tasks
My Heart Counts heart-health dashboard with demonstration scores.Heart health Heart health

Looking ahead

Task-conditioned signal retrieval

Our future research will investigate multimodal representations connecting wearable signals, care events, and patient interactions within agentic systems. We will extend our on-device inference methods to personalized support with local processing of sensitive health data, while enabling selective access to longer histories.

02Learning and Evaluation from Patient Interactions

How can patient interactions guide model training and evaluation?

Patient interactions reveal which problems agents need to solve and provide a basis for constructing clinically relevant training and evaluation tasks.

Patient information needs in Stanford portal messages

Colored clusters of points showing related information needs expressed in patient portal messages.
Each point represents a distinct patient need. Lutz et al., manuscript in preparation.

Patient questions and behavior change

Our preliminary analysis of Stanford portal messages identifies recurring patient needs along the cardiovascular-kidney-metabolic (CKM) continuum. Patient and clinician interviews for Plainly, our open-source patient-facing AI platform, inform priorities for agent behavior and communication.

Our work on GPT-Coach combined wearable data with people’s goals and circumstances to personalize physical activity support. In Bloom, we evaluated LLM-augmented behavior-change interactions in a four-week randomized study (N = 54). Together with our work on behaviorally grounded fine-tuning, these studies connect model adaptation, interaction design, and prospective evaluation.

Editorial illustration of a young adult pausing on a park bench, looking up from their phone.

Digital wellbeing and interventions

Our work on translating digital interventions provides a framework for studying accessibility and sustained use alongside technical capability. The one sec Stanford study examines brief pauses before social-media use. We study sleep, activity, and wellbeing, not just time spent on a phone.

As feeds and AI companions adapt to their users, we ask how interventions can preserve people’s control over their own choices.

Looking ahead

Patient-informed learning and evaluation

Our long-term research will investigate how patient interactions can guide agent training, task selection, and personalized communication. Communication patterns will inform benchmarks and learning methods that personalize interactions without treating existing replies as ideal responses. Research on information selection and presentation will test how text, voice, and visual summaries improve understanding.

03Patient-Centered Agentic Systems

How can agents support evolving patient needs under effective supervision?

Patient-facing support must reflect individual care histories and concerns. Our long-term goal is sustained support across visits, with agents connecting physiological evidence to changing patient priorities.

Plainly chat demonstration with a synthetic record and patient question.
Plainly voice demonstration explaining a synthetic health record.
Plainly demo showing the SpineAI intake questionnaire with symptom and duration questions.

Support between visits

Our research on record-grounded explanations connects AI reasoning over electronic health data with physician evaluation of responses to patient questions.

Our Plainly research platform makes this program testable through text and voice interaction. Frequent requests for information between visits guided the selection of spine surgery and reproductive health for our ongoing studies.

Clinical supervision of agent actions

Through STEWARD, a project in ARPA-H’s ADVOCATE program, we will study methods for deciding when an agent should pause for clinician review. These methods must account for gaps in a patient’s history as well as clinical guidance, institutional policies, and regulatory requirements.

AI co-scientists for biomedical studies

As part of the NHLBI-funded AI Data Science Center at Stanford, we investigate research agents for analyses spanning large-scale, multimodal datasets. With Heartwood, investigators can review an agent’s proposed actions before it runs analysis code.

Wearable studies such as My Heart Counts and benchmarks such as OpenMHC motivate a longer-term goal: to test whether patterns observed in one population hold across other cohorts or devices, with the data and analysis behind each finding available for researchers to check.

Heartwood action review showing proposed commands for a synthetic cohort analysis.
Heartwood · GitHub ↗

Looking ahead

Prospective studies in cardiometabolic care

We will investigate architectures combining wearable signals, clinical records, and patient interactions to maintain context and personalize support over time. Our work in mobile cardiovascular research and the multicenter ENGAGE-HF trial provides experience designing and conducting prospective studies. Our open models and ongoing studies support prospective evaluation of patient understanding, communication, and clinician workload.

Research figure

Publications

42 publications

Preprint

TimeNet: An Extensible Unified Data Infrastructure for Next-Generation Temporal Foundation Models

Martin Maritsch, Timo Stoffregen, Thomas Kaar, et al.All 39 authors

Martin Maritsch, Timo Stoffregen, Thomas Kaar, Behsad Riemer, Maxwell A. Xu, Max Rosenblattl, Juncheng Liu, Nicolas Zumarraga, Yu Yvonne Wu, Denys Herasymuk, Sparsh Rastogi, Hyungjun Yoon, Bosong Huang, Arvind Pillai, Dmytro Lopushanskyy, Tony Chen, Robin Deuber, Yichen Liu, Shvat Messica, Dan Li, Jian Lou, Yuwei Zhang, Jaeho Kim, Renée Rosillo Garcia, Fan Wu, Robert Müller, Elgar Fleisch, Flora D. Salim, Dimitris Spathis, Yuzhe Yang, Aaqib Saeed, Daniel McDuff, Ming Jin, Markus Kreft, Kevin O'Sullivan, Robert Jakob, Azul Garza, Paul Schmiedmayer, Patrick Langer

Conference paper

OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data

Patrick Langer, Thomas Kaar, Max Rosenblattl, et al.All 22 authors

Patrick Langer, Thomas Kaar, Max Rosenblattl, Maxwell A. Xu, Winnie Chow, Martin Maritsch, Robert Jakob, Ning Wang, Juncheng Liu, Aradhana Verma, Brian Han, Daniel Seung Kim, Henry Chubb, Scott Ceresnak, Aydin Zahedivash, Alexander Tarlochan Singh Sandhu, Fatima Rodriguez, Daniel McDuff, Elgar Fleisch, Oliver Aalami, Filipe Barata, Paul Schmiedmayer

Proceedings of the 43rd International Conference on Machine Learning (ICML 2026) · 306 · 62642-62676

Preprint

How Well Do Multimodal Models Reason on ECG Signals?

Maxwell A. Xu, Harish Haresamudram, Catherine W. Liu, et al.All 14 authors

Maxwell A. Xu, Harish Haresamudram, Catherine W. Liu, Patrick Langer, Jathurshan Pradeepkumar, Wanting Mao, Sunita J. Ferns, Aradhana Verma, Jimeng Sun, Paul Schmiedmayer, Xin Liu, Daniel McDuff, Emily B. Fox, James M. Rehg

Journal article

Design and Rationale of the My Heart Counts Cardiovascular Health Study: A Large-Scale, Fully Digital Biobank, and Randomized Trial of Large Language Model-Driven Coaching of Physical Activity

Paul Schmiedmayer, Anders Johnson, Narayan Schuetz, et al.All 18 authors

Paul Schmiedmayer, Anders Johnson, Narayan Schuetz, Lukas Kollmer, Paul Goldschmidt, Juan Delgado-SanMartin, Kelly W. Zhang, Sriya D. Mantena, Alexander Tolas, Samuel Montalvo, Mariana Ramirez-Posada, Jack W. O'Sullivan, Marily Oppezzo, Abby C. King, Fatima Rodriguez, Euan Ashley, Allan Lawrie, Daniel Seung Kim

American Journal of Preventive Cardiology · 28 · 101565

Conference paper

OpenMHC: Accelerating the Science of Wearable Foundation Models

Narayan Schuetz, Yuze Bai, Lianggang Pan, et al.All 19 authors

Narayan Schuetz, Yuze Bai, Lianggang Pan, Edgar Eggert, Favour Nerrise, Juan Delgado-SanMartin, Max Rosenblattl, Milana Gurbanova, Mohammad Asadi, Anders Johnson, Paul Schmiedmayer, Dennis Wang, Allan Lawrie, Daniel Seung Kim, Xin Liu, Akshay Paruchuri, Ehsan Adeli, Euan Ashley, Kelly W. Zhang

Advances in Neural Information Processing Systems (NeurIPS 2026), Evaluations & Datasets Track

Journal article

Fine-Tuning LLMs in Behavioral Psychology for Scalable Health Coaching

Sriya D. Mantena, Anders Johnson, Marily Oppezzo, et al.All 15 authors

Sriya D. Mantena, Anders Johnson, Marily Oppezzo, Narayan Schütz, Alexander Tolas, Ritu Doijad, C. Mikael Mattson, Allan Lawrie, Mariana Ramirez-Posada, Paul Schmiedmayer, Eleni Linos, Abby C. King, Fatima Rodriguez, Daniel Seung Kim, Euan A. Ashley

npj Cardiovascular Health · 2(1) · 48

Journal article

Utility of Smart Watches for Identifying Arrhythmias in Children

Aydin Zahedivash, Henry Chubb, Heather Giacone, et al.All 17 authors

Aydin Zahedivash, Henry Chubb, Heather Giacone, Nicole K. Boramanand, Anne M. Dubin, Anthony Trela, Erin Lencioni, Kara S. Motonaga, William Goodyer, Brittany Navarre, Vishnu Ravi, Paul Schmiedmayer, Vasiliki Bikia, Oliver Aalami, Xuefeng B. Ling, Marco Perez, Scott R. Ceresnak

Communications Medicine · 3(1) · 167

Conference paper

Global Software Engineering in a Global Classroom

Paul Schmiedmayer, Robert Chatley, Jan Philip Bernius, Stephan Krusche, Konstantin Chaika, Kirill Krinkin, Bernd Brügge

Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · 113-121