About

Jaime Fernández Fisac is an Assistant Professor of Electrical and Computer Engineering at Princeton University. He joined the Princeton faculty in August 2020 after working on safety and interaction for autonomous vehicles at Waymo. His research interests focus on making autonomous systems smart enough to function safely in a world full of people, encompassing applications from drones and autonomous cars entering urban environments to large-scale artificial intelligence algorithms influencing daily human experiences. His work aims to develop safe, reliable, and robust autonomous systems capable of operating effectively amidst uncertainty and human interaction, contributing to the advancement of safe robotics and AI systems.

Research topics

• Computer Science
• Artificial Intelligence
• Psychology
• Physics
• Data science
• Cognitive psychology
• Statistical physics

Selected publications

• Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguity

  arXiv (Cornell University) · 2024 · 3 citations

  Senior authorCorresponding
  • Computer Science
  • Computer Science
  • Psychology

  Large language models (LLMs) exhibit advanced reasoning skills, enabling robots to comprehend natural language instructions and strategically plan high-level actions through proper grounding. However, LLM hallucination may result in robots confidently executing plans that are misaligned with user goals or even unsafe in critical scenarios. Additionally, inherent ambiguity in natural language instructions can introduce uncertainty into the LLM's reasoning and planning processes.We propose introspective planning, a systematic approach that align LLM's uncertainty with the inherent ambiguity of the task. Our approach constructs a knowledge base containing introspective reasoning examples as post-hoc rationalizations of human-selected safe and compliant plans, which are retrieved during deployment. Evaluations on three tasks, including a newly introduced safe mobile manipulation benchmark, demonstrate that introspection substantially improves both compliance and safety over state-of-the-art LLM-based planning methods. Furthermore, we empirically show that introspective planning, in combination with conformal prediction, achieves tighter confidence bounds, maintaining statistical success guarantees while minimizing unnecessary user clarification requests. The webpage and code are accessible at https://introplan.github.io.

  DOI

• Emergent Coordination Through Game-Induced Nonlinear Opinion Dynamics

  2023 · 10 citations

  Senior authorCorresponding
  • Computer Science
  • Computer Science
  • Statistical physics

  We present a multi-agent decision-making framework for the emergent coordination of autonomous agents whose intents are initially undecided. Dynamic non-cooperative games have been used to encode multi-agent interaction, but ambiguity arising from factors such as goal preference or the presence of multiple equilibria may lead to coordination issues, ranging from the “freezing robot” problem to unsafe behavior in safety-critical events. The recently developed nonlinear opinion dynamics (NOD) [1] provide guarantees for breaking deadlocks. However, choosing the appropriate model parameters automatically in general multi-agent settings remains a challenge. In this paper, we first propose a novel and principled procedure for synthesizing NOD based on the value functions of dynamic games conditioned on agents' intents. In particular, we provide for the two-player two-option case precise stability conditions for equilibria of the game-induced NOD based on the mismatch between agents' opinions and their game values. We then propose an optimization-based trajectory optimization algorithm that computes agents' policies guided by the evolution of opinions. The efficacy of our method is illustrated with a simulated toll station coordination example.

  DOI

• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

  arXiv (Cornell University) · 2022 · 548 citations

  • Computer Science
  • Artificial Intelligence
  • Computer Science

  Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.

  DOI

Frequent coauthors

• Haimin Hu

  6 shared

• Kai–Chieh Hsu

  4 shared

• Naomi Ehrich Leonard

  4 shared

• Bartolomeo Stellato

  3 shared

• Stefan Clarke

  3 shared

• Zixu Zhang

  East China Jiaotong University

  3 shared

• Gabriele Dragotto

  3 shared

• Jonathan DeCastro

  Toyota Research Institute

  2 shared

Labs

• Safe Robotics LaboratoryPI

Awards & honors

• NSF CAREER Award (2024)
• Sony Focussed Research Award (2023)
• Google Research Scholar Award (2022)
• Leon O. Chua Award for Outstanding Achievement in Nonlinear…

Similar researchers at Princeton University

• Jonathan Cohenh-162
• Michael Straussh-149
• Thomas E. Shenkh-136
• Michael Levineh-131
• Romain Teyssierh-130