'mTurk hypothetical betting study October 2020'
(AsPredicted #50560)
Created: 10/26/2020 09:36 AM (PT)
This is an anonymized version of the pre-registration. It was created by the author(s) to use during peer-review.
A non-anonymized version (containing author names) should be made available by the authors when the work it supports is made public.
1) Have any data been collected for this study already?It's complicated. We have already collected some data but explain in Question 8 why readers may consider this a valid pre-registration nevertheless.
2) What's the main question being asked or hypothesis being tested in this study? We are interested to know to what extent a hypothetical betting manipulation affects participants' ratings of the likelihood of several events happening to them in an imagined fear-provoking situation. Self-identified Spider and Needle fearful participants will read a description of a situation involving either a spider or receiving a blood draw/injection. They then see 6 events that may or may not happen. 3 of these events describe an event hat might be described as 'objective' - i.e., something that will actually happen such as the spider running on their bare feet, or the injection stabbing them in the wrong place. 3 events describe an unpleasant subjective experience of participants, such as feeling so scared it's just horrible, or the feeling of the needle going into their arm being simply terrible.
Half of the participants rate the likelihood of the events happening to them if they are in the situation. The other half receive the extra instruction, to imagine whether they would bet on the event actually happening in reality, before they respond to each event. We predict ratings of objective events will be lowered more than subjective events by this manipulation.
Participants also indicate what they find worse when in such a situation - the objective threat posed, or how the situation makes them feel. We expect most find how they would feel worse than the objective threat posed.
3) Describe the key dependent variable(s) specifying how they will be measured. Rating of the likelihood of the event - from 0 (certainly will not happen) to 100 (certainly will happen), measured using a slider. We make inferences about these ratings by converting them to a 'probability of superiority' (PoS) for participants in the control vs. bet condition, in subjective vs. objective events.
Additional variable that is simply summarised: Participants select whether they find their feelings, or the objective danger posed by a situation, to be worse.
4) How many and which conditions will participants be assigned to? Participants are split into separate experiments/analyses according to fearing either Spiders or Needles. Participants are randomly assigned (evenly) to a control condition or the hypothetical bet condition. In the hypothetical bet condition, participants receive an additional instruction: that they should think whether they would bet on the event actually happening in reality before giving their response to each event.
5) Specify exactly which analyses you will conduct to examine the main question/hypothesis. Bayesian beta regression, formula: beta.prob ~ bet.control:belief + z + (1 | ppn)
beta.prob is the response rating transformed from 0-100 to between 0 and 1.
bet.control is Bet Condition vs. Control Condition
Belief is the event/belief the participant is responding to (e.g., 'The spider will run onto my bare feet')
z is z transformed Spider Phobia Questionnaire (15 item) or Injection Phobia Questionnaire scores
1|PPN indicates that observations are nested within ppts.
Mean responses to the different beliefs and the spread of the data (phi parameter) will be reported and presented. The key outcome variable is a conversion of these to probability of superiority ratings (PoS). We use the regression parameters to simulate hypothetical 'new' observations of 'participants' set at a SPQ/IPQ z score of 0 (average) for each belief, comparing whether randomly paired Bet or Control responses are higher. By doing this over a sample of the posterior we get a posterior of Bet vs. Control participants for each belief. We will show each PoS posterior for each belief, but the key comparison is the difference in PoS for the average of Objective vs. average of Subjective beliefs. Superiority of control ppts (higher scores than bet ppts) should be greater in Objective than Subjective responses. We evaluate this looking at the overall posterior distribution, but can make a basic decision based on the 95% highest density interval (HDI) of this comparison excluding 0 of no difference in PoS for Objective vs. Subjective items.
Priors:
Each belief: normal(m = 0 , sd = 1.4)
Questionnaire z score: student_t(nu = 5, m = 0 , sd = 1.25)
Phi: student_t(nu = 5, m = 0 , sd = 5)
6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations. Exclusions based on quality checks:
Making sense and being accurate when describing the task (0 or 1)
Correctly identifying the bet instruction (0 or 1).
Giving year of birth and age that correspond (0 or 1)
Identifying consistently as from the US (0 or 1)
Passing attention check questions about task features (0 - none, 1 - one right, 2 = 2 right)
Not completing all necessary tasks (0 or 1)
These are multiplied together to = 0 (total exclusion), 1 (not fully trustworthy but could be explored by others), 2 = passes all checks and will be included in proper analyses reported in main text.
7) How many observations will be collected or what will determine sample size?
No need to justify decision, but be precise about exactly how the number will be determined. Analysis will be conducted once we reach 50 ppts per cell. If the 95% HDI for the PoS comparison between subjective vs. objective events does not exclude 0, we may continue data collection up to at most 100 ppts per cell. We may also cease data collection if the results are quite clearly not showing the expected effect - e.g., the posterior is near-evenly spanning either side 0 of no differences in PoS for objective vs. subjective events, indicating that the effect is likely very small or that we may not have the resources to reach a sample size sufficient to resolve the precise difference. Exact numbers per cell depend on what fears people identify with.
If we continue after the first analysis, we would report each interim analysis in a supplement, and include a variable in our data that shows which interim analysis subjects were in. Below we specify a batch schedule to reduce the possibility that observing the results might change how we collect data and affect the data generating process.
Batches of 100 participants each day, Monday to Thursday, starting at approximately 6pm Amsterdam time. We first analyse data after 5 batches (hopefully 50 ppts per cell by then), and then may continue with individual batches to update the results.
8) Anything else you would like to pre-register?
(e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?) We collect timing for each set of questions. Possibility of exploratory analyses of timing - e.g., do the bet people take more time. Is time predictive of their responses, e.g., perhaps because it indicates greater reflection when answering the questions?
Along with the average probability of superiority, we will also be reporting the means, mean differences, phi parameter of the beta distribution. We will also show the consistency of effects for each event as well as the average.
We will run an additional analysis for spiders in which we simply update the results of a previous experiment with similar design, as this may allow for a more precise estimate of the effects. This previous experiment is the 'previously collected data' we refer to, and was only for spiders. The first batch will be run today.
