A large-scale evaluation of commonsense knowledge in humans and large language models

PNAS Nexus · 2026-02-16

Commonsense knowledge, a major constituent of AI, is primarily evaluated in practice by human-prescribed ground-truth labels. An important, albeit implicit, assumption of these labels is that they accurately capture what any human would think, effectively treating human common sense as homogeneous. However, recent empirical work has shown that humans vary enormously in what they consider commonsensical; thus what appears self-evident to one benchmark designer may not be so to another. Here, we propose a method for assessing commonsense knowledge in AI, specifically in large language models (LLMs) that incorporates empirically observed heterogeneity among humans by measuring the correspondence between a model's judgment and that of a human population. We first find that, when treated as independent survey respondents, most LLMs remain below the human median in their individual commonsense competence. Second, when used as simulators of a hypothetical population, LLMs correlate with real humans only modestly in the extent to which they agree on the same set of statements. In both cases, smaller, open-weight models are surprisingly more competitive than larger, proprietary frontier models. Our evaluation framework, which ties commonsense knowledge to its cultural basis, contributes to the growing call for adapting AI models to human collectivities that possess different, often incompatible, social stocks of knowledge.

Integrative experiments identify how punishment affects welfare in public goods games

Science · 2026-04-09 · 1 citations

Despite decades of research, the conditions under which punishment promotes cooperation remain unclear. Through an integrative experiment varying 14 design parameters of public goods games across 360 experimental conditions (147,618 decisions from 7100 participants), we reveal substantial heterogeneity in punishment effectiveness: Its impact on welfare ranges from 43% improvement to 44% reduction depending on the game parameters. To characterize these patterns, we developed models that outperformed human forecasters in predicting punishment effectiveness in new experiments. Communication emerges as the most important factor, followed by contribution framing (opt out versus opt in), contribution type (variable versus all-or-nothing), game length, and outcome visibility, though these factors often interact. The results reframe the debate from whether punishment works to when it does, demonstrating how integrative experiments enable discovery of generalizable patterns in social phenomena.

Media Bias Detector: Designing and Implementing a Tool for Real-Time Selection and Framing Bias Analysis in News Coverage

2025-04-25 · 6 citations

Mainstream media, through their decisions on what to cover and how to frame the stories they cover, can mislead readers without using outright falsehoods.Therefore, it is crucial to have tools that expose these editorial choices underlying media bias.In this paper, we introduce the Media Bias Detector, a tool for researchers, journalists, and news consumers.By integrating large language models, we provide near real-time granular insights into the topics, tone, political lean, and facts of news articles aggregated to the publisher level.We assessed the tool's impact by interviewing 13 experts from journalism, communications, and political science, revealing key insights into usability and functionality, practical applications, and AI's role in powering media bias tools.We explored this in more depth with a follow-up survey of 150 news consumers.This work highlights opportunities for AI-driven tools that empower users to critically engage with media content, particularly in politically charged environments.

Computational social science: past, present, and future

Edward Elgar Publishing Limited eBooks · 2025-12-11

Rethinking news framing with large language models

Scientific Reports · 2025-11-28

Mainstream media, with its broad reach, plays a central role in shaping public opinion and thus warrants close scrutiny. Subtle forms of media bias-such as selective fact presentation and tone-can meaningfully influence public attitudes, even when reporting remains factually accurate. Although effects such as these have been widely studied by scholars of framing, much of the existing research focuses on specific topics and relies on manually constructed or pre-existing frames, limiting both scalability and generalizability. Here we introduce a novel framework that leverages large language models (LLMs) to generate synthetic news articles by systematically varying the selection and tone of the content while holding factual accuracy and other features constant. We evaluate the impact of these alternative framings in a large, pre-registered randomized experiment (N = 2,141), and find that selective presentation of accurate information can significantly shift individuals' policy views and emotional responses across a diverse collection of topics. These effects are consistently stronger for negative than positive framings and are more pronounced among individuals who say they are less informed about the topic. Our findings demonstrate the persuasive power of subtle bias in mainstream news as well as the value of LLMs as tools for scalable, controlled investigations of media effects.

Unpacking media bias in the growing divide between cable and network news

Scientific Reports · 2025-05-21 · 2 citations

The potential for a large, diverse population to coexist peacefully is thought to depend on the existence of a public sphere in which citizens are exposed to similar facts about similar topics. A generation ago, broadcast television news was widely considered to serve this function; however, since the rise of cable news in the 1990s, critics and scholars have worried that the corresponding fragmentation and segregation of audiences has caused this baseline of common understanding to be lost. Recent work documents that millions of Americans are loyal consumers of cable TV news stations. However, the implications of partisan segregation in TV news consumption depend on bias in content-which topics TV news programs talk about and the language they use to talk about them. Here, we measure bias in the production of TV news at scale by analyzing nearly a decade of TV news (Dec. 2012-Oct. 2022) on the largest cable and broadcast stations. We quantify the share of attention each station devoted to more than 20 politically significant topics as well as the linguistic similarity of different stations' news coverage of those topics. We find that while broadcast news continues to cover similar topics with similar language, cable news stations have become increasingly distinct, both from broadcast news and from each other, diverging in terms of both content and language. This trend is driven by hard news as much as partisan commentary programs. Our results show that changes in the supply, not just consumption, of TV news are contributing to Americans' polarizing media diets.

A research agenda for encouraging prosocial behaviour on social media

Nature Human Behaviour · 2025-03-10 · 2 citations

Hypothetical nudges provide directional but noisy estimates of real behavior change

Communications Psychology · 2025-11-18 · 1 citations

Hypothetical scenarios provide an extremely useful alternative to field experiments for scholars interested in nudging behavior change, comprising a substantial proportion of the literature. Yet the extent to which hypotheticals accurately estimate real-world treatment effects is not well understood. To investigate, we identified five recent field studies of real-world nudges in distinct domains and designed four styles of hypothetical scenarios to approximate each one. This setup allows for clear comparison of old field data with new hypothetical data. Across our 20 experiments (N = 16,114), hypothetical scenarios nearly always estimated the correct direction of treatment effects. However, they varied widely in estimating magnitudes, making them unreliable inputs to real-world policy applications such as cost-benefit analyses. Our findings underscore the promising value of hypotheticals, but also the need for greater investigation into strategies to calibrate their estimates. Twenty experiments show that hypothetical scenarios (across four styles) correctly estimated whether or not behavioral interventions (i.e., nudges) would encourage behaviors in five field settings, but unreliably estimated the size of those effects.

The role of topic choice in cross-partisan conversations

2025-05-31

Affective polarization in the United States—animosity between Republicans and Democrats—has escalated for decades, threatening the health of American democracy. Intergroup Contact Theory suggests that talking across party lines can reduce polarization, yet recent studies disagree on whether confronting or avoiding political disagreement is the more effective strategy for cross-partisan conversation. We address this debate using a large-scale “integrative” experiment that systematically varies levels of disagreement and political relevance across a diverse set of topics. While some discussion topics reduced affective polarization more than others, these differences were not explained by how “political” the topics were or how much participants disagreed. In fact, variation within topics far exceeds variation between them. Participant reports suggest that how individuals engage with one another—through listening, openness, and perspective-taking—may play a larger role in shaping conversational success than either the topic itself or the extent of disagreement.

Hypothetical nudges provide directional but noisy estimates of real behavior change

2025-09-18 · 1 citations

Hypothetical scenarios provide an extremely useful alternative to field experiments for scholars interested in nudging behavior change, comprising a substantial proportion of the literature. Yet the extent to which hypotheticals accurately estimate real-world treatment effects is not well understood. To investigate, we identified five recent field studies of real-world nudges in distinct domains and designed four styles of hypothetical scenarios to approximate each one. This setup allows for clear comparison of old field data with new hypothetical data. Across our 20 experiments (N=16,114), hypothetical scenarios nearly always estimated the correct direction of treatment effects. However, they varied widely in estimating magnitudes, making them unreliable inputs to real-world policy applications such as cost-benefit analyses. Our findings underscore the promising value of hypotheticals, but also the need for greater investigation into strategies to calibrate their estimates.