About

Carole Lee is a Professor in the Department of Philosophy at the University of Washington. Her fields of interest include the philosophy of science, philosophy of social science, science and technology, and the social structure of science. She has been involved in research examining racial disparities in peer review scoring of NIH grants and has partnered with other UW researchers to study bias detection in grant application peer review. Her work has been recognized through awards such as the first prize in the NIH Peer Review Challenge and her selection for the Simpson Center’s Society of Scholars. Professor Lee has contributed to the academic community through teaching courses on the history and philosophy of science, social structure of science, philosophical issues in cognitive sciences, and practical reasoning. She maintains an active research profile, with a personal website providing further information about her work. Her research focuses on understanding the social and structural aspects of scientific practice, including issues of bias and disparities in scientific funding and review processes.

Research topics

• Sociology
• Computer Science
• Social Science
• Political Science
• Medicine
• Geography
• Data science
• Nursing
• Psychology
• Economic geography
• Biology
• Gerontology
• Demography

Selected publications

• TOP 2025: An Update to the Transparency and Openness Promotion Guidelines

  2025-02-03 · 19 citations

  preprintOpen access

  Open science remains vital to the progress and functioning of the global research enterprise. Published in 2015, the Transparency and Openness Promotion Guidelines (TOP 2015) was developed as a policy framework to enhance the verifiability of empirical research claims in journal articles. It has been widely used and adopted by publishers and academic journals, but despite its uptake, concerns have been raised about aspects of the TOP 2015 framework and its implementation. In response to the above, the purpose of this manuscript is to introduce an official update to the TOP Guidelines. The final version—TOP 2025—provides updated guidelines for promoting the verifiability of published empirical research claims.

  PublisherDOI

• Study Protocol: The Role of Disagreement in Scientific Conservatism

  SSRN Electronic Journal · 2024-01-01

  articleOpen access
  PublisherDOI

• Catalyzing communities of research rigour champions

  Brain Communications · 2024-01-01

  articleOpen access

  The biomedical sciences must maintain and enhance a research culture that prioritizes rigour and transparency. The US National Institute of Neurological Disorders and Stroke convened a workshop entitled 'Catalyzing Communities of Research Rigor Champions' that brought together a diverse group of leaders in promoting research rigour and transparency (identified as 'rigour champions') to discuss strategies, barriers and resources for catalyzing technical, cultural and educational changes in the biomedical sciences. This article summarizes 2 days of panels and discussions and provides an overview of critical barriers to research rigour, perspectives behind reform initiatives and considerations for stakeholders across science. Additionally, we describe applications of network science to foster, maintain and expand cultural changes related to scientific rigour and opportunities to embed rigourous practices into didactic courses, training experiences and degree programme requirements. We hope this piece provides a primer for the wider research community on current discussions and actions and inspires individuals to build, join or expand collaborative networks within their own institutions that prioritize rigourous research practices.

  PublisherOA PDFDOI

• TOP 2025: An Update to the Transparency and Openness Promotion Guidelines

  2024-09-18 · 4 citations

  preprintOpen access

  Open science remains vital to the progress and functioning of the global research enterprise. Published in 2015, the Transparency and Openness Promotion Guidelines (TOP 2015) was developed as a policy framework to enhance the verifiability of empirical research claims in journal articles. It has been widely used and adopted by publishers and academic journals, but despite its uptake, concerns have been raised about aspects of the TOP 2015 framework and its implementation. In response to the above, the purpose of this manuscript is to introduce an official update to the TOP Guidelines. The final version—TOP 2025—provides updated guidelines for promoting the verifiability of published empirical research claims.

  PublisherDOI

• A new approach to grant review assessments: score, then rank

  Research Integrity and Peer Review · 2023-07-24 · 9 citations

  articleOpen access

  BACKGROUND: In many grant review settings, proposals are selected for funding on the basis of summary statistics of review ratings. Challenges of this approach (including the presence of ties and unclear ordering of funding preference for proposals) could be mitigated if rankings such as top-k preferences or paired comparisons, which are local evaluations that enforce ordering across proposals, were also collected and incorporated in the analysis of review ratings. However, analyzing ratings and rankings simultaneously has not been done until recently. This paper describes a practical method for integrating rankings and scores and demonstrates its usefulness for making funding decisions in real-world applications. METHODS: We first present the application of our existing joint model for rankings and ratings, the Mallows-Binomial, in obtaining an integrated score for each proposal and generating the induced preference ordering. We then apply this methodology to several theoretical "toy" examples of rating and ranking data, designed to demonstrate specific properties of the model. We then describe an innovative protocol for collecting rankings of the top-six proposals as an add-on to the typical peer review scoring procedures and provide a case study using actual peer review data to exemplify the output and how the model can appropriately resolve judges' evaluations. RESULTS: For the theoretical examples, we show how the model can provide a preference order to equally rated proposals by incorporating rankings, to proposals using ratings and only partial rankings (and how they differ from a ratings-only approach) and to proposals where judges provide internally inconsistent ratings/rankings and outlier scoring. Finally, we discuss how, using real world panel data, this method can provide information about funding priority with a level of accuracy in a well-suited format for research funding decisions. CONCLUSIONS: A methodology is provided to collect and employ both rating and ranking data in peer review assessments of proposal submission quality, highlighting several advantages over methods relying on ratings alone. This method leverages information to most accurately distill reviewer opinion into a useful output to make an informed funding decision and is general enough to be applied to settings such as in the NIH panel review process.

  PublisherOA PDFDOI

• Gender-based homophily in collaborations across a heterogeneous scholarly landscape

  PLoS ONE · 2023 · 23 citations

  • Sociology
  • Computer Science
  • Social Science

  In this article, we investigate the role of gender in collaboration patterns by analyzing gender-based homophily-the tendency for researchers to co-author with individuals of the same gender. We develop and apply novel methodology to the corpus of JSTOR articles, a broad scholarly landscape, which we analyze at various levels of granularity. Most notably, for a precise analysis of gender homophily, we develop methodology which explicitly accounts for the fact that the data comprises heterogeneous intellectual communities and that not all authorships are exchangeable. In particular, we distinguish three phenomena which may affect the distribution of observed gender homophily in collaborations: a structural component that is due to demographics and non-gendered authorship norms of a scholarly community, a compositional component which is driven by varying gender representation across sub-disciplines and time, and a behavioral component which we define as the remainder of observed gender homophily after its structural and compositional components have been taken into account. Using minimal modeling assumptions, the methodology we develop allows us to test for behavioral homophily. We find that statistically significant behavioral homophily can be detected across the JSTOR corpus and show that this finding is robust to missing gender indicators in our data. In a secondary analysis, we show that the proportion of women representation in a field is positively associated with the probability of finding statistically significant behavioral homophily.

  PublisherDOI

• Refinement: Measuring informativeness of ratings in the absence of a gold standard

  British Journal of Mathematical and Statistical Psychology · 2022-03-16 · 2 citations

  articleSenior author

  We propose a new metric for evaluating the informativeness of a set of ratings from a single rater on a given scale. Such evaluations are of interest when raters rate numerous comparable items on the same scale, as occurs in hiring, college admissions, and peer review. Our exposition takes the context of peer review, which involves univariate and multivariate cardinal ratings. We draw on this context to motivate an information-theoretic measure of the refinement of a set of ratings - entropic refinement - as well as two secondary measures. A mathematical analysis of the three measures reveals that only the first, which captures the information content of the ratings, possesses properties appropriate to a refinement metric. Finally, we analyse refinement in real-world grant-review data, finding evidence that overall merit scores are more refined than criterion scores.

  PublisherDOI

• A new approach to peer review assessments: Score, then rank

  Research Square · 2022-10-27

  preprintOpen access

  Abstract Background: In many peer review settings, proposals are selected for funding onthe basis of some summary statistics – such as the mean, median, or percentile –of review scores. There are numerous challenges to working with scores. Theseinclude low inter-rater reliability, epistemological differences, susceptibility tovarying levels of leniency or harshness of reviewers, and the presence of ties. Adifferent approach that is able to mitigate some of these issues would be toadditionally collect rankings such as top-k preferences or paired comparisons andincorporate them in the analysis of review scores. Rankings and pairedcomparisons are scale-free and can enforce demarcation between proposals bydesign. However, analyzing scores and rankings simultaneously has not been doneuntil recently due to the lack of tools for principled modeling. Methods: We first introduce an innovative protocol for collecting rankingsamong top quality proposals. This rankings collection is done as an add-on to thetypical peer review procedures focused on scores and does not require reviewersto rank all proposals. We then present statistical methodology for obtaining anintegrated score for each proposal, and from the integrated scores an inducedpreference ordering, that captures both types of peer review inputs: scores andrankings. Our statistical methodology allows for the collected rankings to differfrom the score-implied rankings; this feature is essential when the two qualityassessments disagree which, as we find empirically, often happens in peer review.We illustrate how our method quantifies the uncertainty in order to betterunderstand reviewer preferences among similarly scored proposals. Results: Using artificial “toy” examples and real peer review data, wedemonstrate that incorporating top-k rankings into scores allows us to betterlearn when reviewers can distinguish between proposals. We also examine therobustness of this system to partial rankings, inconsistencies between ratings andrankings, and outliers. Finally, we discuss how, using panel data, this method canprovide information about funding priority that provides a level of accuracy in aformat that is well suited for the types of decisions research funders make. Conclusions: The gathering of both rating and ranking data and the use ofintegrated scores and its induced preference ordering can have many advantagesover methods relying on ratings alone, leveraging more information to mostaccurately distill reviewer opinion into a useful output to make the most informedfunding decision.

  PublisherOA PDFDOI

• Certified Amplification: An Emerging Scientific Norm and Ethos

  Philosophy of Science · 2022-05-30 · 6 citations

  articleOpen access1st authorCorresponding

  Abstract Merton envisioned his norms of science at a time when peer-reviewed journals controlled scientific communication. Technologies for sharing and finding content have since divorced the certification and amplification of science, generating systemic vulnerabilities. Certified amplification —a new Mertonian-styled norm—enjoins their recoupling and introduces a taxonomy of strategies adopted by institutions to close the certification-amplification gap, including the proportioning of the one to the other. Examples illustrating each taxonomic type collectively paint a picture of an ethos employing a rich range of certification and amplification techniques and emerging in a decentralized fashion across heterogeneous objects, communication modalities, and institutions.

  PublisherOA PDFDOI

• When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review

  OSF Preprints (OSF Preprints) · 2021-01-01 · 7 citations

  articleOpen access1st authorCorresponding

  Reproducibility code for the paper "When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review" by Elena A. Erosheva, Patrícia Martinková, and Carole J. Lee, published in Journal of the Royal Statistical Society: Series A (Statistics in Society) https://doi.org/10.1111/rssa.12681 Funding information: NSF, Grant/Award Number: #1759825; Czech Academy of Sciences: RVO, Grant/Award Number: 67985807; Czech Science Foundation, Grant/Award Number: 21- 03658S; COST Action, Grant/Award Number: TD1306. Disclaimer: This material is based upon work supported by the National Science Foundation under Grant Number #1759825. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

  PublisherOA PDFDOI

Frequent coauthors

• Mary E. Larimer

  University of Washington

  90 shared

• Megan E. Patrick

  University of Michigan–Ann Arbor

  79 shared

• Melissa A. Lewis

  University of North Texas

  78 shared

• Clayton Neighbors

  University of Houston

  69 shared

• Anne M. Fairlie

  University of Washington

  57 shared

• Isaac C. Rhew

  University of Washington

  50 shared

• Jason R. Kilmer

  University of Washington

  44 shared

• Scott Graupensperger

  35 shared

Labs

• Carole Lee LabPI

Education

• PhD, Philosophy

  University of Michigan

  2006

• BA, Philosophy

  Wellesley College

  1999

Awards & honors

• Carole Lee chosen for Simpson Center’s Society of Scholars (…
• Carole Lee (co-PI) awarded NSF grant with Elena Erosheva (PI…
• CAROLE LEE WINS FIRST PRIZE IN THE NIH PEER REVIEW CHALLENGE…
• Awards and Achievements (February 6, 2015)
• Faculty (March 31, 2014)