Dan Roth

Cross-Lingual Ability of Multilingual BERT: An Empirical Study

International Conference on Learning Representations · 2019 · 172 citations

Recent work has exhibited the surprising cross-lingual abilities of multilingual BERT (M-BERT) -- surprising since it is trained without any cross-lingual objective and with no aligned data. In this work, we provide a comprehensive study of the contribution of different components in M-BERT to its cross-lingual ability. We study the impact of linguistic properties of the languages, the architecture of the model, and the learning objectives. The experimental study is done in the context of three…

BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language

2021 · 73 citations

Transformer-based language models have taken the NLP world by storm. However, their potential for addressing important questions in language acquisition research has been largely ignored. In this work, we examined the grammatical knowledge of RoBERTa Using the behavioral probing paradigm, we found that a smaller version of RoBERTa-base that never predicts unmasked tokens, which we term BabyBERTa, acquires grammatical knowledge comparable to that of pre-trained RoBERTa-base -and does so with appr…

Extracting seizure frequency from epilepsy clinic notes: a machine reading approach to natural language processing

Journal of the American Medical Informatics Association · 2022 · 62 citations

OBJECTIVE: Seizure frequency and seizure freedom are among the most important outcome measures for patients with epilepsy. In this study, we aimed to automatically extract this clinical information from unstructured text in clinical notes. If successful, this could improve clinical decision-making in epilepsy patients and allow for rapid, large-scale retrospective research. MATERIALS AND METHODS: We developed a finetuning pipeline for pretrained neural models to classify patients as being seizur…

“Who said it, and Why?” Provenance for Natural Language Claims

2020 · 12 citations

In an era where generating content and publishing it is so easy, we are bombarded with information and are exposed to all kinds of claims, some of which do not always rank high on the truth scale. This paper suggests that the key to a longer-term, holistic, and systematic approach to navigating this information pollution is capturing the provenance of claims. To do that, we develop a formal definition of provenance graph for a given natural language claim, aiming to understand where the claim ma…

Zero-Shot Extraction of Seizure Outcomes from Clinical Notes Using Generative Pretrained Transformers

Journal of Healthcare Informatics Research · 2025-04-29 · 6 citations

Emerging evidence has shown that pre-trained encoder transformer models can extract information from unstructured clinic note text but require manual annotation for supervised fine-tuning. Large, Generative Pre-trained Transformer (GPT) models may streamline this process. In this study, we explore GPTs in zero- and few-shot learning scenarios to analyze clinical health records. We prompt-engineered Llama2 13B to optimize performance in extracting seizure freedom from epilepsy clinic notes and co…