About

Christopher Honey is a computational cognitive neuroscientist whose research group studies how our brains enable us to perceive and learn about sequences of information, such as the sequence of words in a sentence. He completed his undergraduate work in Applied Mathematics and English Literature at the University of Cape Town and received his PhD in Psychological and Brain Sciences from Indiana University, where his thesis research was conducted in the laboratory of Olaf Sporns. Following his doctoral studies, he completed a postdoctoral fellowship with Uri Hasson at Princeton University. He then started a laboratory at the University of Toronto before moving to Johns Hopkins University in 2016, where he began as an Assistant Professor of Psychological and Brain Sciences.

Research topics

• Computer Science
• Artificial Intelligence
• Natural Language Processing
• Physics

Selected publications

• Characterizing Verbatim Short-Term Memory in Neural Language Models

  arXiv (Cornell University) · 2022 · 1 citations

  • Computer Science
  • Computer Science
  • Natural Language Processing

  When a language model is trained to predict natural language sequences, its prediction at each moment depends on a representation of prior context. What kind of information about the prior context can language models retrieve? We tested whether language models could retrieve the exact words that occurred previously in a text. In our paradigm, language models (transformers and an LSTM) processed English text in which a list of nouns occurred twice. We operationalized retrieval as the reduction in surprisal from the first to the second list. We found that the transformers retrieved both the identity and ordering of nouns from the first list. Further, the transformers' retrieval was markedly enhanced when they were trained on a larger corpus and with greater model depth. Lastly, their ability to index prior tokens was dependent on learned attention patterns. In contrast, the LSTM exhibited less precise retrieval, which was limited to list-initial tokens and to short intervening texts. The LSTM's retrieval was not sensitive to the order of nouns and it improved when the list was semantically coherent. We conclude that transformers implemented something akin to a working memory system that could flexibly retrieve individual token representations across arbitrary delays; conversely, the LSTM maintained a coarser and more rapidly-decaying semantic gist of prior tokens, weighted toward the earliest items.

  DOI

• Characterizing Verbatim Short-Term Memory in Neural Language Models

  2022 · 3 citations

  • Computer Science
  • Computer Science
  • Natural Language Processing

  When a language model is trained to predict natural language sequences, its prediction at each moment depends on a representation of prior context. What kind of information about the prior context can language models retrieve? We tested whether language models could retrieve the exact words that occurred previously in a text. In our paradigm, language models (transformers and an LSTM) processed English text in which a list of nouns occurred twice. We operationalized retrieval as the reduction in surprisal from the first to the second list. We found that the transformers retrieved both the identity and ordering of nouns from the first list. Further, the transformers' retrieval was markedly enhanced when they were trained on a larger corpus and with greater model depth. Lastly, their ability to index prior tokens was dependent on learned attention patterns. In contrast, the LSTM exhibited less precise retrieval, which was limited to list-initial tokens and to short intervening texts. The LSTM's retrieval was not sensitive to the order of nouns and it improved when the list was semantically coherent. We conclude that transformers implemented something akin to a working memory system that could flexibly retrieve individual token representations across arbitrary delays; conversely, the LSTM maintained a coarser and more rapidly-decaying semantic gist of prior tokens, weighted toward the earliest items.

  DOI

• Long Short-Term Memory with Slower Information Decay

  2022 · 1 citations

  • Computer Science
  • Computer Science
  • Physics

  Learning to process long-range dependencies has been a challenge for recurrent neural networks. Despite improvements achieved by long shortterm memory (LSTMs), its gating mechanism results in exponential decay of information, limiting their capacity of capturing long-range dependencies. In this work, we present a power law forget gate, which instead has a slower rate of information decay. We propose a power law-based LSTM (pLSTM) based on the LSTM but with a power law forget gate. We test empirically the pLSTM on the copy task, sentiment classification, and sequential MNIST, all with long-range dependency tasks. The pLSTM solves these tasks outperforming an LSTM, specially for long-range dependencies. Further, the pLSTM learns sparser and more robust representations.

  DOI

• The “Narratives” fMRI dataset for evaluating models of naturalistic language comprehension

  Scientific Data · 2021 · 153 citations

  • Computer Science
  • Natural Language Processing
  • Computer Science

  The "Narratives" collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging.

  DOI

Frequent coauthors

• Tal Linzen

  3 shared

• Kristijan Armeni

  3 shared

• Javier S. Turek

  2 shared

• Nicole Beckage

  2 shared

• Vy A. Vo

  2 shared

• Janice Chen

  Johns Hopkins University

  1 shared

• Po-Hsuan Chen

  1 shared

• Theodore L. Willke

  1 shared

Labs

• Honey LabPI

Similar researchers at Johns Hopkins University

• Richard Huganirh-210
• Steven L. Salzbergh-191
• Arturo Casadevallh-152
• Peter Campochiaroh-139
• Alan Yuilleh-134