Dan Roth
Professor
Research Interests
Make your next move.
Prepare your application
Review drafts before you send them.
Explore related searches
Articles (3)
Year:
2025
Year:
2025
Large, pre-trained language models (PLMs) such as BERT and GPT have drastically changed the Natural Language Processing (NLP) field. For numerous NLP tasks, approaches leveraging PLMs have achieved state-of-the-art performance. The key idea is to learn a generic, latent representation of language from a generic task once, then share it across disparate NLP tasks. Language modeling serves as the generic task, one with abundant self-supervised text available for extensive training. This article presents the key fundamental concepts of PLM architectures and a comprehensive view of the shift to PLM-driven NLP techniques. It surveys work applying the pre-training then fine-tuning, prompting, and text generation approaches. In addition, it discusses PLM limitations and suggested directions for future research.
Year:
2023
Collaborators (2)
Wei-Chiu Ma
Assistant Professor
Cornell University
Thien Nguyen
Assistant Professor
University of Oregon
Make your next move.
Prepare your application
Review drafts before you send them.