Author(s) Anne Schüler (Leibniz-Institut für Wissensmedien) - a.schueler@iwm-tuebingen.de Pauline Frick (Leibniz-Institut für Wissensmedien) - pauline.frick@uni-tuebingen.de
Pre-registered on 2021/08/24 01:27 (PT)
1) Have any data been collected for this study already? No, no data have been collected for this study yet.
2) What's the main question being asked or hypothesis being tested in this study? This study aims to investigate whether sentence-picture combinations are validated automatically.
Therefore, participants see either valid sentence-picture combinations (sentence describes the picture correctly) or invalid sentence-picture combinations (sentence describes the picture incorrectly). After that, participants have to react to the probe word "wrong" by pressing the key D or to the probe word "right" by pressing the key K. Importantly, participants are instructed to react only to the probe word, regardless of the previous sentence-picture combination.
We hypothesize that the reaction time after congruent stimuli (sentence-picture combination is valid, probe word is right or sentence-picture combination is invalid, probe word is wrong) is faster than after incongruent stimuli (sentence-picture combination is invalid, probe word is right or sentence picture combination is valid, probe word is wrong). This should result in a significant interaction between validity and probe word, meaning that for valid sentence-picture combinations faster reaction times regarding the reaction to the probe word "right" than to the probe word "wrong" are expected, whereas for invalid sentence-picture combinations faster reactions times for the probe word "wrong" than for the probe word "right" are expected.
Moreover, we hypothesize that the error rate after congruent stimuli (sentence-picture combination is valid, probe word is right or sentence-picture combination is invalid, probe word is wrong) is lower than after incongruent stimuli (sentence-picture combination is invalid, probe word is right or sentence picture combination is valid, probe word is wrong). This should result in a significant interaction between validity and probe word, meaning that for valid sentence-picture combinations less errors regarding the reaction to the probe word "right" than to the probe word "wrong" are expected, whereas for invalid sentence-picture combinations less errors for the probe word "wrong" than for the probe word "right" are expected.
3) Describe the key dependent variable(s) specifying how they will be measured. Reaction time: Time until a participant presses the key D or K after seeing the probe word measured in milliseconds.
Error rate: How many times a participant presses the key D after seeing the probe word right and presses the key K after seeing the probe word wrong.
4) How many and which conditions will participants be assigned to? There are four within-subjects conditions resulting from crossing the factor sentence-picture validity (valid vs. invalid) and the factor probe word (right vs. wrong).
We will use four counterbalanced list of items: Participants are randomly assigned to one of those 4 lists. Every list contains 80 sentence-picture combinations followed by the probe word (right or wrong). The number of valid and invalid sentence-picture combinations, the number of the probe words "right" and "wrong", and the number of congruent and incongruent trials are the same in all lists. Moreover, after 20 randomly selected trials participants have to answer if they have just seen this sentence or this picture. The list will be presented in two blocks à 40 trials each.
5) Specify exactly which analyses you will conduct to examine the main question/hypothesis. Reaction time: Linear mixed effects model with item and participant as random effects, probe word and validity as fixed effects will be conducted. If the interaction of validity * probe word is significant, pairwise single comparisons will be conducted. We will compare reaction times for probe words (wrong vs. right) for invalid as well as valid stimuli.
Error rate: General linear mixed effects model with item and participant as random effects, probe word and validity as fixed effects. If the interaction of validity * probe word is significant, pairwise single comparisons will be conducted. We will compare error rates for probe words (wrong vs. right) for invalid as well as valid stimuli.
In case that the interactions are not significant, we will run additional analyses to test whether the single comparisons reach significance.
6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations. Exclusions of participants: Participants who do not agree to have their data processed will be deleted. Participants have to speak German fluently and be right-handed. Participants who report serious technical issues during the study will be excluded as well. Moreover, participants with an error rate above 40% in the Stroop task or in the verification task will be excluded.
Exclusion of trials: Trials with a reaction time under 10ms or above 5s will be excluded. For reaction time analysis trials with incorrect responses will be excluded.
7) How many observations will be collected or what will determine sample size? No need to justify decision, but be precise about exactly how the number will be determined. We conducted several power simulations based on the reported data in Piest et al. 2018 in the first experiment. Please note that validity and probe word are contrast coded (invalid = 1, valid = -1; false = 1, correct = -1) and reaction time data is log transformed.
The simulations revealed that 150 participants lead to a sufficient power. Due to possible data loss, we aim for 160 participants.
The final simulations were conducted as follows:
Reaction time: The parameters were set as follows:
Fixed effects: estimate intercept = 6.273 estimate probe word = 0.009, estimate validity = 0.011, estimate probe word * Validity = -0.009
SD of random effects: SD participant = 0.14258, SD item = 0.01357, SD residual = 0.3
Note that we estimated a smaller interaction effect of probe word * validity and a higher residual variance compared to the results of Piest.
Based on 2000 simulations, 80 Items, and 150 participants the estimated power is 0.899
Error rate: The parameters were set as follows:
Fixed effects: estimate intercept = -4.472, estimate probe word = -0.224, estimate validity = 0.091, estimate probe word * Validity = -0.16
SD of random effects: SD participant = 0.7953, SD item = 0.4427
Based on 2000 simulations, 80 Items, and 150 participants the estimated power is 0.608
8) Anything else you would like to pre-register? (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?) Participants will be recruited via Prolific. Screening criteria will be language (first language = German, German as a fluent language, Germany as current place of residence), age (between 18 and 35), no reading related disorder, no participation in the conducted pre-studies and right-handedness.
Transformation of data: Since reaction time data are often not normally distributed the data will be log transformed. If this transformation is not adequate, another or no transformation will be chosen.
Additional analysis: The 80 Items are presented in two separate blocks (40 items each). If the interaction of validity * probe word * block reaches significance, the two blocks will be analysed separately. Moreover, as exploratory analysis we will compare reaction times/error rates for validity (valid vs. invalid) for the probe word right as well as the probe word wrong.
