Walter Mathis
Research Interests
Make your next move.
Prepare your application
Review drafts before you send them.
Explore related searches
Articles (9)
Background Existing large language model (LLM) evaluations rely on accuracy benchmarks that fail to capture whether models reason well while making diagnoses. Studies that do analyse reasoning focus on post hoc explanations accompanying model outputs rather than distinct, clinician-visible artifacts such as detailed reasoning traces. This creates a translational gap in domains such as psychiatry, where diagnosis relies on narrative interpretation, diagnostic reasoning, and clinical judgment under uncertainty. Methods We conducted a mixed-methods evaluation of four state-of-the-art LLMs using a clinician-curated dataset of 196 psychiatric case vignettes, including 135 published cases and 61 novel clinician-authored vignettes. Diagnostic accuracy was assessed using multiple metrics (top-1 accuracy, top-5 accuracy, recall@5, and mean reciprocal rank) based on ranked lists of five differential diagnoses per vignette. Clinical reasoning quality was evaluated on a randomly selected subset of 30 vignettes through clinician assessment of model-generated diagnostic reasoning traces alongside qualitative commentary from board-certified psychiatrists. We examined the association between clinician-rated reasoning quality and diagnostic correctness and included an illustrative comparison with psychiatry residents. Findings Clinician-rated diagnostic reasoning quality was strongly associated with diagnostic correctness in mixed-effects logistic regression analyses (β = 1·80; p < 0·001), whereas data extraction quality alone was not. Across the full vignette set, models demonstrated moderate to high diagnostic accuracy. The highest-performing model achieved a top-5 accuracy of 0·801 and also received the highest clinician-rated reasoning scores. In an illustrative comparison, model diagnostic accuracy fell within the range observed for psychiatry residents. Interpretation Diagnostic reasoning quality captures clinically meaningful variation in LLM performance beyond accuracy metrics. Psychiatry may represent a stringent testbed for evaluating reasoning in narrative-driven clinical domains. Evaluation frameworks for LLM-based clinical decision support should incorporate structured assessment of reasoning processes, not accuracy alone. Research in Context Evidence before this study We searched PubMed and Scopus for studies evaluating large language models for psychiatric diagnosis and/or differential diagnosis from text vignettes. Searches were run from database inception to February 6, 2026, using terms including (“large language model” OR LLM “artificial intelligence” OR “generative AI” OR AI OR ChatGPT OR GPT OR Claude OR Gemini OR DeepSeek OR Llama) AND (psychiatr* OR mental OR DSM OR “differential diagnosis” OR diagnos*) AND (vignette OR case OR “case report”). We included empirical studies that evaluated model diagnostic outputs using psychiatric cases/vignettes and excluded editorials/commentaries and studies that did not report case-level diagnostic performance. We did not formally assess risk of bias/quality; studies were heterogeneous in vignette sources, model access, and outcome definitions, precluding quantitative pooling. Prior studies have typically examined small vignette sets, focused on narrow diagnostic domains, evaluated single models, or relied primarily on outcome-based accuracy metrics. When diagnostic reasoning has been assessed, it has usually been inferred from post hoc explanations accompanying model outputs rather than evaluated as a distinct, clinician-visible artifact. Clinician-grounded evaluations of diagnostic reasoning across multiple contemporary models remain limited. Added value of this study This study provides a large-scale, clinician-grounded evaluation of diagnostic accuracy and diagnostic reasoning quality across four contemporary large language models using a diverse dataset of psychiatric case vignettes. Rather than relying solely on outcome-based explanations, we directly evaluated model-generated d
Year:
2026
Chronic hepatitis B (CHB) is a major cause of liver‐related morbidity and mortality in the United States. Patients with CHB require long‐term antiviral treatment and consistent follow‐up, but often face numerous barriers to accessing care and medications. In this study, we used the Medicare Part D database and the Rural–Urban Continuum code to explore specialty and geographic characteristics of healthcare providers that manage Medicare patients with CHB. Between 2013 and 2021, more than 7000 prescribers prescribed over 2.4 million 30‐day prescriptions of these CHB therapies, of which 2 million (85%) were in metropolitan counties. The number of 30‐day prescriptions increased by 5.4% annually. The number of prescriptions by GI increased by 12.5% a year and prescriptions by APPs increased by 12.2% a year, while prescriptions by ID decreased by 14.0% annually. In non‐metropolitan counties, APPs experienced −5% APC between 2013 and 2021, and PCPs experienced −12.5% APC between 2016 and 2021. In this study, we found that there has been a noticeable shift in the specialties prescribing medications for patients with hepatitis B. Gastroenterologists and APPs became significantly more involved as prescribers. The increase in APP management of patients with CHB is a welcoming development, especially in light of the physician workforce shortage. It is important to create solutions such as co‐management to ensure that patients with CHB receive consistent care without further contributing to supply–demand mismatch in gastroenterology.
Year:
2025
Year:
2025
Year:
2024
Year:
2024
This paper presents TimelinePTC, a web-based tool developed to improve the collection and analysis of Pathways to Care (PTC) data in first episode psychosis (FEP) research. Accurately measuring the duration of untreated psychosis (DUP) is essential for effective FEP treatment, requiring detailed understanding of the patient’s journey to care. However, traditional PTC data collection methods, mainly manual and paper-based, are time-consuming and often fail to capture the full complexity of care pathways. TimelinePTC addresses these limitations by providing a digital platform for collaborative, real-time data entry and visualization, thereby enhancing data accuracy and collection efficiency. Initially created for the Specialized Treatment Early in Psychosis (STEP) program in New Haven, Connecticut, its design allows for straightforward adaptation to other healthcare contexts, facilitated by its open-source codebase. The tool significantly simplifies the data collection process, making it more efficient and user-friendly. It automates the conversion of collected data into a format ready for analysis, reducing manual transcription errors and saving time. By enabling more detailed and consistent data collection, TimelinePTC has the potential to improve healthcare access research, supporting the development of targeted interventions to reduce DUP and improve patient outcomes.
Year:
2024
Year:
2024
We investigated the impact of COVID-19 restrictions on the duration of untreated psychosis (DUP). First-episode psychosis admissions (n=101) to STEP Clinic in Connecticut showed DUP reduction (p=.0015) in the pandemic, with the median reducing from 208 days during the pre-pandemic to 56 days in the early pandemic period and subsequently increasing to 154 days (p=.0281). Time from psychosis onset to anti-psychotic prescription decreased significantly in the pandemic (p=.0183), with the median falling from 117 to 35 days. This cohort study demonstrates an association between greater pandemic restrictions and marked DUP reduction and provides insights for future early detection efforts.
Year:
2024
Year:
2023
Collaborators (20)
Matteo Malgaroli
Assistant Professor
NYU School of Medicine
Stephen Congly
University of Calgary
Giovanni Appolon
San Diego State University
Nimish Patel
Associate Professor
-
Salih Selek
Professor
University of Texas System
Stefano De Paoli
Abertay University
Sarah Riley
Assistant Professor
Yale University School of Medicine
Qingyu Chen
Assistant Professor
National Institutes of Health
Hadar Hazan
Assistant professor
SUNY Polytechnic Institute
Brian Zaboski
Assistant Professor
Yale University
Hua Xu
Yale University
Maria Ferrara
Assistant Professor
University of Ferrara
Jiyeong Kim
Assistant Professor
University of California, San Francisco
Christopher Pittenger
Professor (secondary)
Yale University
Nasser Sharareh
Faculty Director
University of Utah Health
Robert Brown
Gladys and Roland Harriman Professor of Medicine
Joan and Sanford I Weill Medical College of Cornell University
Carolyn Rodriguez
Professor and Associate Dean
Stanford University
Inmaculada Hernandez
Professor
University of California, San Diego
Vinod Srihari
Professor
Yale University School of Medicine
Lucas Berenbrok
University of Pittsburgh
Make your next move.
Prepare your application
Review drafts before you send them.