About

Julia Stoyanovich is an Institute Associate Professor of Computer Science and Engineering, an Associate Professor of Data Science, and the Director of the Center for Responsible AI at New York University. She is also a member of the Visualization and Data Analytics Research Center at NYU. Her research interests include AI ethics and legal compliance, data management, and AI systems. Stoyanovich works towards making 'Responsible AI' synonymous with 'AI' by engaging in academic research, education, and technology policy, and by speaking about the benefits and harms of AI to practitioners and the public. She has been recognized with the Presidential Early Career Award for Scientists and Engineers (PECASE) and is a Senior member of the Association of Computing Machinery (ACM). Her work includes involvement in technology policy and regulation in the US and internationally, such as serving on the New York City Automated Decision Systems Task Force. Stoyanovich received her M.S. and Ph.D. degrees in Computer Science from Columbia University and her B.S. in Computer Science and Mathematics & Statistics from the University of Massachusetts at Amherst. Her research encompasses AI ethics, data management, and responsible AI systems, with notable contributions to understanding the societal impacts of AI and developing frameworks for transparency and accountability in automated decision-making.

Research topics

• Computer Science
• Machine Learning
• Artificial Intelligence
• Information Retrieval
• Accounting
• Data science
• Political Science
• Clinical psychology
• Business
• Social psychology
• Psychology
• Applied psychology
• Public relations

Selected publications

• How Much Effort Is Enough? Fairness in Algorithmic Recourse Through the Lens of Substantive Equality of Opportunity

  2025-10-29 · 1 citations

  articleOpen accessSenior author
  PublisherDOI

• Still More Shades of Null: An Evaluation Suite for Responsible Missing Value Imputation

  Proceedings of the VLDB Endowment · 2025-05-01

  articleSenior author

  Data missingness is a practical challenge of sustained interest to the scientific community. In this paper, we present Shades-of-Null, an evaluation suite for responsible missing value imputation. Our work is novel in two ways (i) we model realistic and socially-salient missingness scenarios that go beyond Rubin's classic Missing Completely at Random (MCAR), Missing At Random (MAR) and Missing Not At Random (MNAR) settings, to include multi-mechanism missingness (when different missingness patterns co-exist in the data) and missingness shift (when the missingness mechanism changes between training and test) (ii) we evaluate imputers holistically, based on imputation quality and imputation fairness, as well as on the predictive performance, fairness and stability of the models that are trained and tested on the data post-imputation. We use Shades-of-Null to conduct a large-scale empirical study involving 29,736 experimental pipelines, and find that while there is no single best-performing imputation approach for all missingness types, interesting trade-offs arise between predictive performance, fairness and stability, based on the combination of missingness scenario, imputer choice, and the architecture of the predictive model. We make Shades-of-Null publicly available, to enable researchers to rigorously evaluate missing value imputation methods on a wide range of metrics in plausible and socially meaningful scenarios.

  PublisherDOI

• Estimating the impact of the Russian invasion on the displacement of graduating high school students in Ukraine

  Humanities and Social Sciences Communications · 2025-06-16 · 2 citations

  articleOpen accessSenior author

  Abstract On 24 February 2022, Russia began a full-scale invasion of Ukraine. The war has dramatically impacted every area of life in Ukraine, including education. In this paper, we curate a uniquely comprehensive dataset of standardized exam outcomes used for admissions to higher education institutions in Ukraine—analogous to the Standardized Aptitude Test (SAT) in the United States—to provide strong estimates of student displacement and the first analysis of student drop-off , or decline of participation in the Ukrainian education system, following the Russian full-scale invasion. We conducted descriptive statistical analysis, which included computing and comparing means across groups of students, conditioned on geographic location, migration pattern, and demographics, coupled with data visualization. We found that, among the graduating Ukrainian high school students in 2022, approximately 36,500 (16%) were displaced, with 64% of them moving abroad, primarily to Poland, Germany, and Czechia. Most displaced students originated from the front-line war regions, and either moved abroad or migrated towards the central and western parts of Ukraine. Further, we found a 21% decline in graduating high school students taking the standardized higher education entrance exam in 2022, as compared to 2021. This drop-off from the common educational pathway consists of approximately 41,500 students. With these findings taken together, we estimate that at least 78,000—a staggering 34%—of high school seniors have been directly impacted by the Russian invasion of Ukraine. We also study the impacts on subgroups and at the intersection of socio-economic status (as measured by urban vs. rural location) and gender, and find that intersectionality exacerbates the impacts, with men from rural areas being particularly adversely impacted. We conclude this article by reflecting on several policies pursued by the Ukrainian government and its institutions, aimed at minimizing disruptions to the school year and retaining students. Our analysis has important implications for governmental organizations like the Ukrainian government and the European Union, and human rights organizations like the UN Refugee Agency and the International Organization for Migration who wish to understand the impact of the Russian invasion on the education system in Ukraine.

  PublisherOA PDFDOI

• "Would You Want an AI Tutor?" Understanding Stakeholder Perceptions of LLM-based Systems in the Classroom

  ArXiv.org · 2025-02-02

  preprintOpen access

  Large Language Models (LLMs) have gained traction in educational settings, often framed as virtual tutors or teaching assistants. Following early skepticism and bans, many schools and universities have begun integrating these systems into curricula. Yet decisions about whether and how to deploy LLM-based tools are frequently made without systematic engagement with the full range of stakeholders they affect. In this paper, we argue that understanding stakeholder perceptions of LLM-based systems in the classroom is not a matter of measuring approval or acceptance, but of identifying whose concerns are surfaced, in which contexts, and with what implications for responsible design and governance. We introduce Contextualized Perceptions for the Adoption of LLMs in Education (Co-PALE), a stakeholder-first framework that connects educational context, responsible AI principles, and categories of perception to support more deliberate decision-making about the adoption of LLM-based tools. We ground Co-PALE through a targeted analysis of prior work to diagnose recurring gaps in how stakeholder perceptions are studied, and through contextually distinct educational scenarios that illustrate how the same technology raises different concerns for different stakeholders. We further examine how university faculty and K--12 parents make sense of the framework through focus groups, using their reflections to surface tensions and uncertainties. Co-PALE supports more systematic reasoning about whether, where, and for whom LLM-based tools should be deployed in education.

  PublisherOA PDFDOI

• SHAP-based Explanations are Sensitive to Feature Representation

  2025-06-23 · 6 citations

  articleOpen access

  Local feature-based explanations are a key component of the XAI toolkit.These explanations compute feature importance values relative to an "interpretable" feature representation.In tabular data, feature values themselves are often considered interpretable.This paper examines the impact of data engineering choices on local feature-based explanations.We demonstrate that simple, common data engineering techniques, such as representing age with a histogram or encoding race in a specific way, can manipulate feature importance as determined by popular methods like SHAP.Notably, the sensitivity of explanations to feature representation can be exploited by adversaries to obscure issues like discrimination.While the intuition behind these results is straightforward, their systematic exploration has been lacking.Previous work has focused on adversarial attacks on feature-based explainers by biasing data or manipulating models.To the best of our knowledge, this is the first study demonstrating that explainers can be misled by standard, seemingly innocuous data engineering techniques.

  PublisherOA PDFDOI

• The Cambridge Report on Database Research

  ArXiv.org · 2025-04-15 · 1 citations

  preprintOpen access

  On October 19 and 20, 2023, the authors of this report convened in Cambridge, MA, to discuss the state of the database research field, its recent accomplishments and ongoing challenges, and future directions for research and community engagement. This gathering continues a long standing tradition in the database community, dating back to the late 1980s, in which researchers meet roughly every five years to produce a forward looking report. This report summarizes the key takeaways from our discussions. We begin with a retrospective on the academic, open source, and commercial successes of the community over the past five years. We then turn to future opportunities, with a focus on core data systems, particularly in the context of cloud computing and emerging hardware, as well as on the growing impact of data science, data governance, and generative AI. This document is not intended as an exhaustive survey of all technical challenges or industry innovations in the field. Rather, it reflects the perspectives of senior community members on the most pressing challenges and promising opportunities ahead.

  PublisherOA PDFDOI

• SAFENUDGE: Safeguarding Large Language Models in Real-time with Tunable Safety-Performance Trade-offs

  2025-01-01 · 1 citations

  articleOpen accessSenior author

  Large Language Models (LLMs) have been shown to be susceptible to jailbreak attacks, or adversarial attacks used to illicit high risk behavior from a model, highlighting the critical need to safeguard widely-deployed models.Safeguarding approaches, which include finetuning models or having LLMs "self-reflect," may lengthen the inference time of a model, incur a computational penalty, reduce the semantic fluency of an output, and restrict "normal" model behavior.Importantly, these Safety-Performance Trade-offs (SPTs) remain an understudied area.In this work, we make three contributions: (1) We introduce SAFENUDGE, a novel safeguard that combines Controlled Text Generation and "nudging."SAFENUDGE triggers during text-generation while a jailbreak attack is being executed, and can reduce successful jailbreak attempts by between 28.1% and 37.3% by guiding the LLM towards a safe response.It adds minimal latency to inference and has a negligible impact on the semantic fluency of outputs.Second, it supports tunable SPTs, meaning practitioners can set their own tolerance for tradeoffs balancing safety and restrictions to normal model behavior.Third, we release the source code for SAFENUDGE at https:// github.com/joaopfonseca/SafeNudge.It is open source and compatible with the Hugging Face transformers library.

  PublisherOA PDFDOI

• Safeguarding Large Language Models in Real-time with Tunable Safety-Performance Trade-offs

  arXiv (Cornell University) · 2025-01-02

  preprintOpen accessSenior author

  Large Language Models (LLMs) have been shown to be susceptible to jailbreak attacks, or adversarial attacks used to illicit high risk behavior from a model. Jailbreaks have been exploited by cybercriminals and blackhat actors to cause significant harm, highlighting the critical need to safeguard widely-deployed models. Safeguarding approaches, which include fine-tuning models or having LLMs "self-reflect", may lengthen the inference time of a model, incur a computational penalty, reduce the semantic fluency of an output, and restrict ``normal'' model behavior. Importantly, these Safety-Performance Trade-offs (SPTs) remain an understudied area. In this work, we introduce a novel safeguard, called SafeNudge, that combines Controlled Text Generation with "nudging", or using text interventions to change the behavior of a model. SafeNudge triggers during text-generation while a jailbreak attack is being executed, and can reduce successful jailbreak attempts by 30% by guiding the LLM towards a safe responses. It adds minimal latency to inference and has a negligible impact on the semantic fluency of outputs. Further, we allow for tunable SPTs. SafeNudge is open-source and available through https://pypi.org/, and is compatible with models loaded with the Hugging Face "transformers" library.

  PublisherOA PDFDOI

• Safeguarding Large Language Models in Real-time with Tunable Safety-Performance Trade-offs

  Qeios · 2025-01-16

  preprintOpen accessSenior author

  Large Language Models (LLMs) have been shown to be susceptible to _jailbreak attacks_, or adversarial attacks used to illicit high risk behavior from a model. Jailbreaks have been exploited by cybercriminals and blackhat actors to cause significant harm, highlighting the critical need to safeguard widely-deployed models. Safeguarding approaches, which include fine-tuning models or having LLMs “self-reflect”, may lengthen the inference time of a model, incur a computational penalty, reduce the semantic fluency of an output, and restrict “normal” model behavior. Importantly, these Safety-Performance Trade-offs (SPTs) remain an understudied area. In this work, we introduce a novel safeguard, called SAFENUDGE, that combines Controlled Text Generation with “nudging,” or using text interventions to change the behavior of a model. SAFENUDGE _triggers during text-generation while a jailbreak attack is being executed_, and can reduce successful jailbreak attempts by 30% by guiding the LLM towards a safe responses. It adds minimal latency to inference and has a negligible impact on the semantic fluency of outputs. Further, we allow for tunable SPTs. SAFENUDGE is open-source and available through https://pypi.org/, and is compatible with models loaded with the Hugging Face transformerslibrary.

  PublisherOA PDFDOI

• Making Transparency Advocates: An Educational Approach Towards Better Algorithmic Transparency in Practice

  Proceedings of the AAAI Conference on Artificial Intelligence · 2025-04-11

  articleOpen accessSenior author

  Concerns about the risks and harms posed by artificial intelligence (AI) have resulted in significant study into algorithmic transparency, giving rise to a sub-field known as Explainable AI (XAI). Unfortunately, despite a decade of development in XAI, an existential challenge remains: progress in research has not been fully translated into the actual implementation of algorithmic transparency by organizations. In this work, we test an approach for addressing the challenge by creating transparency advocates, or motivated individuals within organizations who drive a ground-up cultural shift towards improved algorithmic transparency. Over several years, we created an open-source educational workshop on algorithmic transparency and advocacy. We delivered the workshop to professionals across two separate domains to improve their algorithmic transparency literacy and willingness to advocate for change. In the weeks following the workshop, participants applied what they learned, such as speaking up for algorithmic transparency at an organization-wide AI strategy meeting. We also make two broader observations: first, advocacy is not a monolith and can be broken down into different levels. Second, individuals' willingness for advocacy is affected by their professional field. For example, news and media professionals may be more likely to advocate for algorithmic transparency than those working at technology start-ups.

  PublisherOA PDFDOI

Recent grants

• CAREER: Querying Evolving Graphs

  NSF · $105k · 2018–2019

• NSF-BSF: III: Small: Collaborative Research: Databases Meet Computational Social Choice

  NSF · $234k · 2018–2022

• CAREER: Querying Evolving Graphs

  NSF · $498k · 2018–2024

• Collaborative Research: III: MEDIUM: Responsible Design and Validation of Algorithmic Rankers

  NSF · $400k · 2023–2027

• BIGDATA: F: Collaborative Research: Foundations of Responsible Data Management

  NSF · $485k · 2017–2019

Frequent coauthors

• Serge Abiteboul

  35 shared

• Gerome Miklau

  26 shared

• Bill Howe

  24 shared

• Susan B. Davidson

  University of Pennsylvania

  19 shared

• Ke Yang

  16 shared

• Benny Kimelfeld

  Technion – Israel Institute of Technology

  16 shared

• Sihem Amer-Yahia

  Laboratoire d'Informatique de Grenoble

  15 shared

• Vera Zaychik Moffitt

  14 shared

Labs

• Center for Responsible AIPI

Awards & honors

• Presidential Early Career Award for Scientists and Engineers…
• Senior member of the Association of Computing Machinery (ACM…
• NSF CAREER : Querying Evolving Graphs (2018)
• Member of the NYC automated decision systems task force, app…
• Co-PI on a NSF-BSF grant : Databases Meet Computational Soci…