'Critical Condition – Known arm within experiment' (AsPredicted #13289)
Author(s) Robert Mislavsky (Johns Hopkins University) - mislavsky@jhu.edu Berkeley Dietvorst (University of Chicago) - berkeley.dietvorst@chicagobooth.edu Uri Simonsohn (ESADE) - urisohn@gmail.com
Pre-registered on 2018/08/11 - 08:04 AM (PT)
1) Have any data been collected for this study already? No, no data have been collected for this study yet.
2) What's the main question being asked or hypothesis being tested in this study? Does knowing which arm of an experiment one was assigned to, influence the acceptability of the experiment as a whole?
3) Describe the key dependent variable(s) specifying how they will be measured. Is it okay for Amazon to do this? (1 = It’s really bad; 4 = It’s okay (neutral); 7 = It’s really good)
4) How many and which conditions will participants be assigned to? Participants read that Amazon is considering two changes to its product recommendation system and is (a) running an experiment with those policies or (b) chooses one of the two. They then read which policy (a) they were randomly assigned to in the experiment or (b) which policy Amazon chose, and rate Amazon’s actions. We also include a condition where participants read about an experiment, but do not learn which policy they were assigned to.
Therefore, there are 5 between-subject conditions: 2 (experiment vs. policy change) x 2 (receive “good” vs. “bad” policy) + 1 (experiment with no outcome information)
The policies are:
“Good” - Recommending most highly rated items
“Bad” - Recommending the most profitable items
5) Specify exactly which analyses you will conduct to examine the main question/hypothesis. For each policy (i.e., “good” and “bad”) we will compare responses in the experiment condition to those in the policy change condition, using t-tests. We will also compare the ratings of the “experiment/bad outcome” condition to the “experiment/no outcome information” condition, and the “policy/bad outcome” condition to the “experiment/no outcome information” condition
6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations. We will include all participant responses that are recorded in the survey. We include a captcha, attention check and a screening question (filtering out participants who fail to complete the captcha and attention check and are not regular Amazon customers, defined as making at least one purchase per month). We will not collect the dependent variable for participants who do not pass these screenings.
7) How many observations will be collected or what will determine sample size? No need to justify decision, but be precise about exactly how the number will be determined. 1,000 (200/cell)
8) Anything else you would like to pre-register? (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?) To describe the sample, we will collect age, gender, and whether participants are Amazon Prime members. We will compare responses for Prime and non-Prime members, but make no predictions.
