Rong Ge

About

Rong Ge is the Cue Family Associate Professor in the Computer Science Department at Duke University. He earned his Ph.D. from the Computer Science Department of Princeton University under the supervision of Sanjeev Arora. Following his doctoral studies, he was a postdoctoral researcher at Microsoft Research in New England. His research broadly spans theoretical computer science and machine learning, with a focus on understanding and formalizing hidden structures in data and designing efficient algorithms to uncover them. He studies problems arising in the analysis of text, images, and other data types, employing techniques such as non-convex optimization and tensor decompositions. His work aims to provide provable algorithms for machine learning problems, contributing to the theoretical foundations of modern machine learning methods including deep learning.

Research topics

• Computer Science
• Parallel computing
• Artificial Intelligence
• Computer Security
• Embedded system
• Operating system
• Computer architecture

Selected publications

• ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods

  2024-01-01 · 8 citations

  articleOpen access

  The rapid scaling of large language models (LLMs) has raised concerns about the transparency and fair use of the data used in their pretraining.Detecting such content is challenging due to the scale of the data and limited exposure of each instance during training.We propose RECALL, (Relative Conditional Log-Likelihood), a novel membership inference attack (MIA) to detect LLMs' pretraining data by leveraging their conditional language modeling capabilities.RECALL examines the relative change in…

  PublisherOA PDFDOI

• For Better or For Worse? Learning Minimum Variance Features With Label Augmentation

  arXiv (Cornell University) · 2024-02-10

  preprintOpen accessSenior author

  Data augmentation has been pivotal in successfully training deep learning models on classification tasks over the past decade. An important subclass of data augmentation techniques - which includes both label smoothing and Mixup - involves modifying not only the input data but also the input label during model training. In this work, we analyze the role played by the label augmentation aspect of such methods. We first prove that linear models on binary classification data trained with label augm…

  PublisherOA PDFDOI

• Mean-Field Analysis for Learning Subspace-Sparse Polynomials with Gaussian Input

  arXiv (Cornell University) · 2024-02-14

  preprintOpen accessSenior author

  In this work, we study the mean-field flow for learning subspace-sparse polynomials using stochastic gradient descent and two-layer neural networks, where the input distribution is standard Gaussian and the output only depends on the projection of the input onto a low-dimensional subspace. We establish a necessary condition for SGD-learnability, involving both the characteristics of the target function and the expressiveness of the activation function. In addition, we prove that the condition is…

  PublisherOA PDFDOI

• Linear Transformers are Versatile In-Context Learners

  arXiv (Cornell University) · 2024-02-21

  preprintOpen accessSenior author

  Recent research has demonstrated that transformers, particularly linear attention models, implicitly execute gradient-descent-like algorithms on data provided in-context during their forward inference step. However, their capability in handling more complex problems remains unexplored. In this paper, we prove that each layer of a linear transformer maintains a weight vector for an implicit linear regression problem and can be interpreted as performing a variant of preconditioned gradient descent…

  PublisherOA PDFDOI

• How Does Gradient Descent Learn Features -- A Local Analysis for Regularized Two-Layer Neural Networks

  arXiv (Cornell University) · 2024-06-03

  preprintOpen accessSenior author

  The ability of learning useful features is one of the major advantages of neural networks. Although recent works show that neural network can operate in a neural tangent kernel (NTK) regime that does not allow feature learning, many works also demonstrate the potential for neural networks to go beyond NTK regime and perform feature learning. Recently, a line of work highlighted the feature learning capabilities of the early stages of gradient-based training. In this paper we consider another mec…

  PublisherOA PDFDOI

Recent grants

• Collaborative Research: II-NEW: Marcher - A Heterogeneous High Performance Computing Infrastructure for Research and Education in Green Computing

  NSF · $252k · 2015–2018

• CSR: Small: Collaborative Research: EEDAG: Exploring Energy-Efficient Parallel Tasks Generation and Scheduling for Heterogeneous Multicore Systems

  NSF · $54k · 2011–2013

• AF: Large: Collaborative Research: Nonconvex Methods and Models for Learning: Towards Algorithms with Provable and Interpretable Guarantees

  NSF · $508k · 2017–2022

Frequent coauthors

• Sham M. Kakade

  35 shared

• Sanjeev Arora

  26 shared

• Xizhou Feng

  Menlo School

  26 shared

• Kirk W. Cameron

  Virginia Tech

  19 shared

• Ziliang Zong

  Texas State University

  18 shared

• Anima Anandkumar

  California Institute of Technology

  14 shared

• Zizhong Chen

  13 shared

• Majid Janzamin

  Twitter (United States)

  12 shared

Labs

• Rong Ge LabPI

  The Rong Ge Lab focuses on theoretical computer science and machine learning, particularly in analyzing text, images, and other forms of data using techniques such as non-convex optimization and tensor decompositions.

Awards & honors

• Collaborative Reseach: Transferable, Hierarchical, Expressiv…
• NSF CAREER: Optimization Landscape for Non-convex Functions…

Similar researchers at Duke University

• John H. Reifh-68
• Pankaj K. Agarwalh-62
• Alberto Bartesaghih-56