Dan Roth

Professor

University of Pennsylvania
Country flagUnited States

Research Interests

ApplyKite TwinLearns as you apply

Make your next move.

Prepare your application

Review drafts before you send them.

Explore related searches

Articles (3)

Year:

2025

Year:

2025

Large, pre-trained language models (PLMs) such as BERT and GPT have drastically changed the Natural Language Processing (NLP) field. For numerous NLP tasks, approaches leveraging PLMs have achieved state-of-the-art performance. The key idea is to learn a generic, latent representation of language from a generic task once, then share it across disparate NLP tasks. Language modeling serves as the generic task, one with abundant self-supervised text available for extensive training. This article presents the key fundamental concepts of PLM architectures and a comprehensive view of the shift to PLM-driven NLP techniques. It surveys work applying the pre-training then fine-tuning, prompting, and text generation approaches. In addition, it discusses PLM limitations and suggested directions for future research.

Year:

2023

Collaborators (2)

Wei-Chiu Ma

Assistant Professor

Cornell University

UNITED STATES
View details

Thien Nguyen

Assistant Professor

University of Oregon

UNITED STATES
View details