Walter Mathis

Yale University
Country flagUnited States

Research Interests

ApplyKite TwinLearns as you apply

Make your next move.

Prepare your application

Review drafts before you send them.

Explore related searches

Articles (9)

Background Existing large language model (LLM) evaluations rely on accuracy benchmarks that fail to capture whether models reason well while making diagnoses. Studies that do analyse reasoning focus on post hoc explanations accompanying model outputs rather than distinct, clinician-visible artifacts such as detailed reasoning traces. This creates a translational gap in domains such as psychiatry, where diagnosis relies on narrative interpretation, diagnostic reasoning, and clinical judgment under uncertainty. Methods We conducted a mixed-methods evaluation of four state-of-the-art LLMs using a clinician-curated dataset of 196 psychiatric case vignettes, including 135 published cases and 61 novel clinician-authored vignettes. Diagnostic accuracy was assessed using multiple metrics (top-1 accuracy, top-5 accuracy, recall@5, and mean reciprocal rank) based on ranked lists of five differential diagnoses per vignette. Clinical reasoning quality was evaluated on a randomly selected subset of 30 vignettes through clinician assessment of model-generated diagnostic reasoning traces alongside qualitative commentary from board-certified psychiatrists. We examined the association between clinician-rated reasoning quality and diagnostic correctness and included an illustrative comparison with psychiatry residents. Findings Clinician-rated diagnostic reasoning quality was strongly associated with diagnostic correctness in mixed-effects logistic regression analyses (β = 1·80; p < 0·001), whereas data extraction quality alone was not. Across the full vignette set, models demonstrated moderate to high diagnostic accuracy. The highest-performing model achieved a top-5 accuracy of 0·801 and also received the highest clinician-rated reasoning scores. In an illustrative comparison, model diagnostic accuracy fell within the range observed for psychiatry residents. Interpretation Diagnostic reasoning quality captures clinically meaningful variation in LLM performance beyond accuracy metrics. Psychiatry may represent a stringent testbed for evaluating reasoning in narrative-driven clinical domains. Evaluation frameworks for LLM-based clinical decision support should incorporate structured assessment of reasoning processes, not accuracy alone. Research in Context Evidence before this study We searched PubMed and Scopus for studies evaluating large language models for psychiatric diagnosis and/or differential diagnosis from text vignettes. Searches were run from database inception to February 6, 2026, using terms including (“large language model” OR LLM “artificial intelligence” OR “generative AI” OR AI OR ChatGPT OR GPT OR Claude OR Gemini OR DeepSeek OR Llama) AND (psychiatr* OR mental OR DSM OR “differential diagnosis” OR diagnos*) AND (vignette OR case OR “case report”). We included empirical studies that evaluated model diagnostic outputs using psychiatric cases/vignettes and excluded editorials/commentaries and studies that did not report case-level diagnostic performance. We did not formally assess risk of bias/quality; studies were heterogeneous in vignette sources, model access, and outcome definitions, precluding quantitative pooling. Prior studies have typically examined small vignette sets, focused on narrow diagnostic domains, evaluated single models, or relied primarily on outcome-based accuracy metrics. When diagnostic reasoning has been assessed, it has usually been inferred from post hoc explanations accompanying model outputs rather than evaluated as a distinct, clinician-visible artifact. Clinician-grounded evaluations of diagnostic reasoning across multiple contemporary models remain limited. Added value of this study This study provides a large-scale, clinician-grounded evaluation of diagnostic accuracy and diagnostic reasoning quality across four contemporary large language models using a diverse dataset of psychiatric case vignettes. Rather than relying solely on outcome-based explanations, we directly evaluated model-generated d

Year:

2026

Chronic hepatitis B (CHB) is a major cause of liver‐related morbidity and mortality in the United States. Patients with CHB require long‐term antiviral treatment and consistent follow‐up, but often face numerous barriers to accessing care and medications. In this study, we used the Medicare Part D database and the Rural–Urban Continuum code to explore specialty and geographic characteristics of healthcare providers that manage Medicare patients with CHB. Between 2013 and 2021, more than 7000 prescribers prescribed over 2.4 million 30‐day prescriptions of these CHB therapies, of which 2 million (85%) were in metropolitan counties. The number of 30‐day prescriptions increased by 5.4% annually. The number of prescriptions by GI increased by 12.5% a year and prescriptions by APPs increased by 12.2% a year, while prescriptions by ID decreased by 14.0% annually. In non‐metropolitan counties, APPs experienced −5% APC between 2013 and 2021, and PCPs experienced −12.5% APC between 2016 and 2021. In this study, we found that there has been a noticeable shift in the specialties prescribing medications for patients with hepatitis B. Gastroenterologists and APPs became significantly more involved as prescribers. The increase in APP management of patients with CHB is a welcoming development, especially in light of the physician workforce shortage. It is important to create solutions such as co‐management to ensure that patients with CHB receive consistent care without further contributing to supply–demand mismatch in gastroenterology.

Year:

2025

Year:

2025

Year:

2024

Year:

2024

This paper presents TimelinePTC, a web-based tool developed to improve the collection and analysis of Pathways to Care (PTC) data in first episode psychosis (FEP) research. Accurately measuring the duration of untreated psychosis (DUP) is essential for effective FEP treatment, requiring detailed understanding of the patient’s journey to care. However, traditional PTC data collection methods, mainly manual and paper-based, are time-consuming and often fail to capture the full complexity of care pathways. TimelinePTC addresses these limitations by providing a digital platform for collaborative, real-time data entry and visualization, thereby enhancing data accuracy and collection efficiency. Initially created for the Specialized Treatment Early in Psychosis (STEP) program in New Haven, Connecticut, its design allows for straightforward adaptation to other healthcare contexts, facilitated by its open-source codebase. The tool significantly simplifies the data collection process, making it more efficient and user-friendly. It automates the conversion of collected data into a format ready for analysis, reducing manual transcription errors and saving time. By enabling more detailed and consistent data collection, TimelinePTC has the potential to improve healthcare access research, supporting the development of targeted interventions to reduce DUP and improve patient outcomes.

Year:

2024

Year:

2024

We investigated the impact of COVID-19 restrictions on the duration of untreated psychosis (DUP). First-episode psychosis admissions (n=101) to STEP Clinic in Connecticut showed DUP reduction (p=.0015) in the pandemic, with the median reducing from 208 days during the pre-pandemic to 56 days in the early pandemic period and subsequently increasing to 154 days (p=.0281). Time from psychosis onset to anti-psychotic prescription decreased significantly in the pandemic (p=.0183), with the median falling from 117 to 35 days. This cohort study demonstrates an association between greater pandemic restrictions and marked DUP reduction and provides insights for future early detection efforts.

Year:

2024

Year:

2023

Collaborators (20)

Matteo Malgaroli

Assistant Professor

NYU School of Medicine

UNITED STATES
View details

Stephen Congly

University of Calgary

CANADA
View details

Giovanni Appolon

San Diego State University

UNITED STATES
View details

Nimish Patel

Associate Professor

-

UNITED STATES
View details

Salih Selek

Professor

University of Texas System

UNITED STATES
View details

Stefano De Paoli

Abertay University

UNITED KINGDOM
View details

Sarah Riley

Assistant Professor

Yale University School of Medicine

UNITED STATES
View details

Qingyu Chen

Assistant Professor

National Institutes of Health

UNITED STATES
View details

Hadar Hazan

Assistant professor

SUNY Polytechnic Institute

UNITED STATES
View details

Brian Zaboski

Assistant Professor

Yale University

UNITED STATES
View details

Hua Xu

Yale University

UNITED STATES
View details

Maria Ferrara

Assistant Professor

University of Ferrara

ITALY
View details

Jiyeong Kim

Assistant Professor

University of California, San Francisco

UNITED STATES
View details

Christopher Pittenger

Professor (secondary)

Yale University

UNITED STATES
View details

Nasser Sharareh

Faculty Director

University of Utah Health

UNITED STATES
View details

Robert Brown

Gladys and Roland Harriman Professor of Medicine

Joan and Sanford I Weill Medical College of Cornell University

UNITED STATES
View details

Carolyn Rodriguez

Professor and Associate Dean

Stanford University

UNITED STATES
View details

Inmaculada Hernandez

Professor

University of California, San Diego

UNITED STATES
View details

Vinod Srihari

Professor

Yale University School of Medicine

UNITED STATES
View details

Lucas Berenbrok

University of Pittsburgh

UNITED STATES
View details