'Multimedia EF in Computer-Based Testing: Effects on Emotion and Cognition'
(AsPredicted #114,387)
Download PDF

---

---

1) Have any data been collected for this study already?
No, no data have been collected for this study yet.

2) What's the main question being asked or hypothesis being tested in this study?
This experimental study examines the affective and cognitive effects of immediate elaborated feedback (EF) with different EF design variations (i.e., feedback message with representational picture [RP] and/or emotional design) provided during a computer-based assessment.

Effects of EF:
1. Emotions: After correct responses, we expect EF (across groups) as compared to no feedback to increase students' report of positive emotions and to decrease their report of negative emotions. After incorrect responses, we expect EF (across groups) as compared no feedback to decrease students' report of positive emotions and to increase their report of negative emotions.
2. Error Correction: We expect EF (across groups) as compared to no feedback to increase students' posttest performance for initially incorrectly answered items (i.e., error correction).
3. Time Spent on EF: We expect students to generally spend longer time on EF messages after incorrect responses than on confirmatory EF messages after correct responses.

Differential Multimedia EF Effects:
1. Emotions: Multimedia elements may have beneficial affective-motivational effects. However, we expect a very small affective benefit of multimedia EF messages in the current experiment at best, which may not statistically prevail. To the extent that we find an emotional benefit of multimedia EF compared to text-only feedback messages, it should be mainly detectable for EF with an RP (i.e., we expect a slight increase in positive emotions; slight decrease in negative emotions).
2. Error Correction: Given the equivalent feedback content in all four EF groups, we do not expect differences regarding students' error correction rates across the experimental groups with EF.
3. Time Spent on EF: We assume that multimedia elements function as a cognitive facilitator for information processing and that multimedia elements therefore reduce the time students require to process corrective EF messages. Thus, we expect students to spend less time on EF messages that include (1) an RP, (2) emotional design, (3) or both compared to text-only EF. Moreover, as the multimedia effects may be additive, students are expected to spend slightly less time on EF messages with both multimedia features (i.e., an RP and emotional design) compared to EF with only one multimedia feature (i.e., an RP or emotional design). In general, we expect these differential multimedia effects for time spent on EF messages to be only present after incorrect responses, while we expect short and similar viewing times across the EF conditions on confirmatory feedback messages (i.e., after correct responses).

3) Describe the key dependent variable(s) specifying how they will be measured.
We measure our dependent variables at the item level (i.e., after each geometry item; 12x in total), immediately after the feedback message (in the EF groups) or after submitting a response (in the no feedback group):
1. To assess their current positive and negative emotions, students rate six one-word statements (e.g., I am currently feeling ..) regarding their current emotions (i.e., positive emotions: happy, proud, confident; negative emotions: annoyed, angry, frustrated) on a 5-point Likert scale.
2. To assess error correction, we employ a randomized recall posttest (treatment test items) and analyze the feedback effects for items with initially incorrect responses.
3. To assess feedback time, we track the time spent on each feedback message in seconds. To adjust for extremely long times spent on feedback, we replace extreme values of M + 2SD with a constant (i.e., M + 2SD) at the item level and separately for feedback messages with (1) text-only, (2) text + RP, (3) text + emotional design, and (4) text + RP + emotional design. We will not adjust for short times as each message has a minimum display time of five seconds.

4) How many and which conditions will participants be assigned to?
The study follows a 1 x 5 between-subjects design. All students complete a computer-based self-paced test with 12 randomized geometric constructed-response items while receiving either no feedback or immediate EF messages in different design variations:
1) No-Feedback Control Group
2) Text-Only EF
3) Text + RP EF
4) Text + Emotional Design (with Signaling) EF
5) Text + RP + Emotional Design (with Signaling) EF

5) Specify exactly which analyses you will conduct to examine the main question/hypothesis.
We will conduct planned contrasts on moderated (generalized) linear mixed-effects models (i.e., GLMMs and LMMs). We define the no-feedback control group as intercept and add one fixed effect per EF group. Response correctness (i.e., 0 = incorrect response; 1 = correct response) will serve as a dichotomous moderator to display the EF effects after correct and incorrect responses.

6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations.
To test the hypotheses, we will exclude participants from the analysis that did not finish the treatment test (i.e., abortion) or showed clearly disengaged behavior (e.g., high amount [i.e., >25%] of rapid guessing or unrealistic response pattern).

7) How many observations will be collected or what will determine sample size?
No need to justify decision, but be precise about exactly how the number will be determined.
We estimated the required sample size in G*Power 3.1.9.4 (Faul et al., 2007) with a power analyses in a repeated-measures ANOVA framework. To detect small effects of f = .16 with a power of >95%, the analysis suggested a sample size of 400 participants. For the planned mixed-effects analyses, we crosschecked the power acquired by a sample size of 400 in additional a priori power simulations with the "simR" package (Green et al., 2019) for linear-mixed effects models (LMMs). Model coefficients were based on coefficients obtained in a pilot study. Based on 100 simulations, the moderated feedback effects after incorrect responses reached an approximate power of 99%.

On the basis of our power analysis, we plan to collect data from N = 410 German speaking undergraduate students that will be recruited from Prolific (www.prolific.co). The number of participants providing sufficient data will determine the final sample size.

8) Anything else you would like to pre-register?
(e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?)
This pre-registration is one set of two different sets of research questions with separate planned analyses that do not share a single dependent variable data point. The data for both parts will be collected in one experiment and published in separate papers.

1. We measure additional variables for sample statistics and supplemental/exploratory analyses:
- Before the treatment test, we collect information on gender, age, country of usual residence, average grade in mathematics, field of study, current semester and targeted degree, initial motivation/emotions, geometric/mathematical self-concept, and achievement goal orientations.
- Before and after the treatment test, students rate three items regarding their planned/invested effort and 13 items regarding their current test-taking motivation. The 13 items were adapted from the EVQ (Expectancy-Value Questionnaire; Trautwein et al., 2012) and the IMI (Intrinsic Motivation Inventory; Ryan & Deci, 2000) and cover students' test-related expectancy of success, intrinsic value, attainment value, and perceived cost. Students further estimate their amount of correct responses and they respond to three items regarding their achievement goal orientations (i.e., performance goal, mastery goal, performance avoidance goal) on a 5-point Likert scale.
- At the end of the study, we ask students to rate the animated pedagogical agent with 15 items extracted from the agent persona index (API; Ryu & Baylor, 2003) on a 5-point Likert scale.
2. Further outcomes will be assessed during and after the treatment test (i.e., self-efficacy, perceived usefulness, perceived facilitation for learning, preference). Effects regarding these outcomes will be reported in a separate paper (i.e., Preregistration Part 2).