Learning from Null Effects: A Bottom-Up Approach

Political Analysis · 2022 · 23 citations

• Computer Science
• Political Science
• Computer Science

Abstract A critical barrier to generating cumulative knowledge in political science and related disciplines is the inability of researchers to observe the results from the full set of research designs that scholars have conceptualized, implemented, and analyzed. For a variety of reasons, studies that produce null findings are especially likely to be unobserved, creating biases in publicly accessible research. While several approaches have been suggested to overcome this problem, none have yet proven adequate. We call for the establishment of a new discipline-wide norm in which scholars post short “null results reports” online that summarize their research designs, findings, and interpretations. To address the inevitable incentive problems that earlier proposals for reform were unable to overcome, we argue that decentralized research communities can spur the broader disciplinary norm change that would bring advantage to scientific advance. To facilitate our contribution, we offer a template for these reports that incorporates evaluation of the possible explanations for the null findings, including statistical power, measurement strategy, implementation issues, spillover/contamination, and flaws in theoretical priors. We illustrate the template’s utility with two experimental studies focused on the naturalization of immigrants in the United States and attitudes toward Syrian refugees in Jordan.

System Error: Where Big Tech Went Wrong and How We Can Reboot

Perspectives on Science and Christian Faith · 2022 · 41 citations

• Computer Science
• Computer Security
• Computer Science

SYSTEM ERROR: Where Big Tech Went Wrong and How We Can Reboot by Rob Reich, Mehran Sahami, and Jeremy M. Weinstein. New York: HarperCollins Publishers, 2021. 352 pages. Hardcover; $27.99. ISBN: 9780063064881. *Remember when digital technology and the internet were our favorite things? When free Facebook accounts connected us with our friends, and the internet facilitated democracy movements overseas, including the Arab Spring? So do the authors of this comprehensive book. "We shifted from a wide-eyed optimism about technology's liberating potential to a dystopian obsession with biased algorithms, surveillance capitalism, and job-displacing robots" (p. 237). *This transition has not escaped the notice of the students and faculty of Stanford University, the elite institution most associated with the rise (and sustainment) of Silicon Valley. The three authors of this book teach a popular course at Stanford on the ethics and politics of technological change, and this book effectively brings their work to the public. Rob Reich is a philosopher who is associated with Stanford's Institute for Human-Centered Artificial Intelligence as well as their Center for Ethics in Society. Mehran Sahami is a computer science professor who was with Google during the startup years. Jeremy Weinstein is a political science professor with experience in government during the Obama administration. *The book is breathtakingly broad, explaining the main technical and business issues concisely but not oversimplifying, and providing the history and philosophy for context. It accomplishes all this in 264 pages, but also provides thirty-six pages of notes and references for those who want to dive deeper into some topics. The most important section is doubtless the last chapter dealing with solutions, which may be politically controversial but are well supported by the remainder of the book. *Modern computer processors have enormous computational power, and a good way to take advantage of that is to do optimization, the subject of the first chapter. Engineers love optimization, but not everything should be done as quickly and cheaply as possible! Optimization requires the choice of some quantifiable metric, but often available metrics do not exactly represent the true goal of an organization. In this case, optimizers will choose a proxy metric which they feel logically or intuitively should be correlated with their goal. The authors describe the problems which result when the wrong proxy is selected, and then excessive optimization drives that measure to the exclusion of other possibly more important factors. For example, social media companies that try to increase user numbers to the exclusion of other factors may experience serious side effects, such as the promotion of toxic content. *After that discussion on the pros and cons of optimization, the book dives into the effects of optimizing money. Venture capitalists (VCs) have been around for years, but recent tech booms have swelled their numbers. The methodology of Objectives and Key Results (OKR), originally developed by Andy Grove of Intel, became popular among the VCs of Silicon Valley, whose client firms, including Google, Twitter, and Uber, adopted it. OKR enabled most of the employees to be evaluated against some metric which management believed captured the essence of their job, so naturally the employees worked hard to optimize this quantity. Again, such a narrow view of the job has led to significant unexpected and sometimes unwanted side effects. *The big tech companies are threatened by legislation designed to mitigate some of the harm they have created. They have hired a great many lobbyists, and even overtly entered the political process where possible. In California, when Assembly Bill 5 reclassified many independent contractors as employees, the affected tech companies struck back with Proposition 22 to overturn the law. An avalanche of very expensive promotion of Proposition 22 resulted in its passage by a large margin. *It is well known that very few politicians have a technical background, and the authors speculate that this probably contributes to the libertarian leaning prominent in the tech industry. The authors go back in history to show how regulation has lagged behind technology and industrial practice. An interesting chapter addresses the philosophical question of whether democracy is up to the task of governing, or whether government by experts, or Plato's "philosopher kings" would be better. *Part II of the book is the longest, addressing the fairness of algorithms, privacy, automation and human job replacement, and free speech. The authors point out some epic algorithm failures, such as Amazon being unable to automate resumé screening to find the best candidates, and Google identifying Black users as gorillas. The big advances in deep learning neural nets result from clever algorithms plus the availability of very large databases, but if you've got a database showing that you've historically hired 95% white men for a position, training an algorithm with that database is hardly going to move you into a future with greater diversity. Even more concerning are proprietary black-box algorithms used in the legal system, such as for probation recommendations. Why not just let humans have the last word, and be advised by the algorithms? The authors remind us that one of the selling points of algorithmic decision making is to remove human bias; returning the humans to power returns that bias as well. *Defining fairness is yet another ethical and philosophical question. The authors give a good overview of privacy, which is protected by law in the European Union by the General Data Protection Regulation. Although there is no such federal law in America, California has passed a similar regulation called the California Consumer Privacy Act. At this point, it's too soon to evaluate the effect of such regulations. *The automation chapter is entitled "Can humans flourish in a world of smart machines?" and it covers many philosophical and ethical issues after providing a valuable summary of the current state of AI. Although machines are able to defeat humans in games like chess, go, and even Jeopardy, more useful abilities such as self-driving cars are not yet to that level. The utopian predictions of AGI (artificial general intelligence, or strong AI), in which the machine can set its own goals in a reasonable facsimile of a human, seem quite far off. But the current state of AI (weak AI) is able to perform many tasks usefully, and automation is already displacing some human labor. The authors discuss the economics, ethics, and psychology of automation, as human flourishing involves more than financial stability. The self-esteem associated with gainful employment is not a trivial thing. The chapter raises many more important issues than can be mentioned here. *The chapter on free speech also casts a wide net. Free speech as we experience it on the internet is vastly different from the free speech of yore, standing on a soap box in the public square. The sheer volume of speech today is incredible, and the power of the social media giants to edit it or ban individuals is also great. Disinformation, misinformation, and harassment are rampant, and polarization is increasing. *Direct incitement of violence, child pornography, and video of terrorist attacks are taken down as soon as the internet publishers are able, but hate speech is more difficult to define and detect. Can AI help? As with most things, AI can detect the easier cases, but it is not effective with the more difficult ones. From a regulatory standpoint, section 230 of the Communications Decency Act of 1996 (CDA 230) immunizes the platforms from legal liability due to the actions of users. Repealing or repairing CDA 230 may be difficult, but the authors make a good case that "it is realistic to think that we can pursue some commonsense reforms" (p. 225). *The final part of the book is relatively short, but addresses the very important question: "Can Democracies Rise to the Challenge?" The authors draw on the history of medicine in the US as an example of government regulation that might be used to reign in the tech giants. Digital technology does not have as long a history as medicine, so few efforts have been made to regulate it. The authors mention the Association for Computing Machinery (ACM) Software Engineering Code of Ethics, but point out that there are no real penalties for violation besides presumably being expelled from the ACM. Efforts to license software engineers have not borne fruit to date. *The authors argue that the path forward requires progress on several fronts. First, discussion of values must take place at the early stages of development of any new technology. Second, professional societies should renew their efforts to increase the professionalism of software engineering, including strengthened codes of ethics. Finally, computer science education should be overhauled to incorporate this material into the training of technologists and aspiring entrepreneurs. *The authors conclude with the recent history of attempts to regulate technology, and the associated political failures, such as the defunding of the congressional Office of Technology Assessment. It will never be easy to regulate powerful political contributors who hold out the prospect of jobs to politicians, but the authors make a persuasive case that it is necessary. China employs a very different authoritarian model of technical governance, which challenges us to show that democracy works better. *This volume is an excellent reference on the very active debate on the activities of the tech giants and their appropriate regulation. It describes many of the most relevant events of the recent past and provides good arguments for some proposed solutions. We need to be thinking and talking about these issues, and this book is a great

Community policing does not build citizen trust in police or reduce crime in the Global South

Science · 2021 · 117 citations

• Political Science
• Sociology
• Criminology

Is it possible to reduce crime without exacerbating adversarial relationships between police and citizens? Community policing is a celebrated reform with that aim, which is now adopted on six continents. However, the evidence base is limited, studying reform components in isolation in a limited set of countries, and remaining largely silent on citizen-police trust. We designed six field experiments with Global South police agencies to study locally designed models of community policing using coordinated measures of crime and the attitudes and behaviors of citizens and police. In a preregistered meta-analysis, we found that these interventions led to mixed implementation, largely failed to improve citizen-police relations, and did not reduce crime. Societies may need to implement structural changes first for incremental police reforms such as community policing to succeed.

Reporting all results efficiently: A RARE proposal to open up the file drawer

Proceedings of the National Academy of Sciences · 2021 · 29 citations

• Computer Science
• Political Science
• Computer Science

While the social sciences have made impressive progress in adopting transparent research practices that facilitate verification, replication, and reuse of materials, the problem of publication bias persists. Bias on the part of peer reviewers and journal editors, as well as the use of outdated research practices by authors, continues to skew literature toward statistically significant effects, many of which may be false positives. To mitigate this bias, we propose a framework to enable authors to report all results efficiently (RARE), with an initial focus on experimental and other prospective empirical social science research that utilizes public study registries. This framework depicts an integrated system that leverages the capacities of existing infrastructure in the form of public registries, institutional review boards, journals, and granting agencies, as well as investigators themselves, to efficiently incentivize full reporting and thereby, improve confidence in social science findings. In addition to increasing access to the results of scientific endeavors, a well-coordinated research ecosystem can prevent scholars from wasting time investigating the same questions in ways that have not worked in the past and reduce wasted funds on the part of granting agencies.

Forced Displacement and Asylum Policy in the Developing World

International Organization · 2021 · 59 citations

• Political Science
• Political Science
• Development economics

Abstract Little theoretical or empirical work examines migration policy in the developing world. We develop and test a theory that distinguishes the drivers of policy reform and factors influencing the direction of reform. We introduce an original data set of de jure asylum and refugee policies covering more than ninety developing countries that are presently excluded from existing indices of migration policy. Examining descriptive trends in the data, we find that unlike in the global North, forced displacement policies in the global South have become more liberal over time. Empirically, we test the determinants of asylum policymaking, bolstering our quantitative results with qualitative evidence from interviews in Uganda. A number of key findings emerge. Intense, proximate civil wars are the primary impetus for asylum policy change in the global South. Liberalizing changes are made by regimes led by political elites whose ethnic kin confront discrimination or violence in neighboring countries. There is no generalizable evidence that developing countries liberalize asylum policy in exchange for economic assistance from Western actors. Distinct frameworks are needed to understand migration policymaking in developing versus developed countries.

Attitudes Toward Migrants in a Highly Impacted Economy: Evidence From the Syrian Refugee Crisis in Jordan

Comparative Political Studies · 2020 · 132 citations

• Political Science
• Political Science
• Development economics

With international migration at a record high, a burgeoning literature has explored the drivers of public attitudes toward migrants. However, most studies to date have focused on developed countries, which have relatively fewer migrants and more capacity to absorb them. We address this sample bias by conducting a survey of public attitudes toward Syrians in Jordan, a developing country with one of the largest shares of refugees. Our analysis indicates that neither personal- nor community-level exposure to the economic impact of the refugee crisis is associated with antimigrant sentiments among natives. Furthermore, an embedded conjoint experiment validated with qualitative evidence demonstrates the relative importance of humanitarian and cultural concerns over economic ones. Taken together, our findings weaken the case for egocentric and sociotropic economic concerns as critical drivers of antimigrant attitudes and demonstrate how humanitarian motives can sustain support for refugees when host and migrant cultures are similar.

Teaching Computer Ethics

2020 · 39 citations

• Sociology
• Computer Science
• Political Science

We report on a curricular experiment at Stanford University focused on teaching computer ethics. After nearly a year of preparation, we launched a new course at the intersection of ethics, public policy, and technology that deeply marries the humanities, social sciences, and computer science. While the teaching of computer ethics courses dates back decades, such courses are often taught by a (single) CS faculty member without significant training in ethics, do not include a policy component, and are meant for CS students. By contrast, we take a deeply multidisciplinary approach, where three faculty instructors, from philosophy, political science, and CS, each bring their respective lens to four related course modules: algorithmic decision-making, data privacy and civil liberties, AI and autonomous systems, and the power of platform companies. Panels of guest speakers drawn from academia, industry, civil society, and government provide a practitioner's view of the topics addressed. Additionally, custom case studies were developed under the direction of the course staff. These materials (videos of the speaker panels and the case studies) are freely available for use by the broader community. We report on the details of the course structure, including how multiple disciplines are integrated throughout the course, including lectures, discussions, and assignments. We discuss aspects of the course that worked well as well as challenges in making the course broadly accessible (beyond just CS majors). Importantly, we also include a discussion of students' response to the course, showing that a deeply multidisciplinary approach resonates strongly with them.