Using Nonnaive Participants Can Reduce Effect Sizes

Jesse Chandler; Gabriele Paolacci; Eyal Peer; Pam Mueller; Kate A. Ratliff

doi:10.1177/0956797615585115

Using Nonnaive Participants Can Reduce Effect Sizes

¹Institute for Social Research, University of Michigan
²Mathematica Policy Research, Ann Arbor, MI
³Rotterdam School of Management, Erasmus University Rotterdam
⁴Graduate School of Business Administration, Bar-Ilan University
⁵Department of Psychology, Princeton University
⁶Department of Psychology, University of Florida

Jesse Chandler, University of Michigan, Institute for Social Research, 426 Thompson St., Ann Arbor, MI 48105 E-mail: jjchandl{at}umich.edu

Author Contributions J. Chandler developed the study concept. All authors contributed to the study design. Testing and data collection were performed by J. Chandler and P. Mueller. J. Chandler and E. Peer analyzed the data. J. Chandler, G. Paolacci, and E. Peer wrote the manuscript, and all authors provided critical revisions. All authors approved the final manuscript for submission.

Abstract

Although researchers often assume their participants are naive to experimental materials, this is not always the case. We investigated how prior exposure to a task affects subsequent experimental results. Participants in this study completed the same set of 12 experimental tasks at two points in time, first as a part of the Many Labs replication project and again a few days, a week, or a month later. Effect sizes were markedly lower in the second wave than in the first. The reduction was most pronounced when participants were assigned to a different condition in the second wave. We discuss the methodological implications of these findings.

Article Notes

Declaration of Conflicting Interests The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article.
Funding Wave 1 of this research was supported by Project Implicit.
Supplemental Material Additional supporting information can be found at http://pss.sagepub.com/content/by/supplemental-data
Open Practices

All data and materials have been made publicly available via Open Science Framework and can be accessed at https://osf.io/4vx3z/. The complete Open Practices Disclosure for this article can be found at http://pss.sagepub.com/content/by/supplemental-data. This article has received badges for Open Data and Open Materials. More information about the Open Practices badges can be found at https://osf.io/tvyxz/wiki/1.%20View%20the%20Badges/ and http://pss.sagepub.com/content/25/1/3.full.