'Acceptance and perceived performance of chat interactions (amendment)' (AsPredicted #65,353)
Author(s) Stefanie Ritz (Leibniz-Institut für Wissensmedien, Tübingen) - s.klein@iwm-tuebingen.de Sonja Utz (Leibniz-Institut für Wissensmedien, Tübingen) - s.utz@iwm-tuebingen.de
Pre-registered on 2021/05/07 05:51 (PT)
1) Have any data been collected for this study already? It's complicated. We have already collected some data but explain in Question 8 why readers may consider this a valid pre-registration nevertheless.
2) What's the main question being asked or hypothesis being tested in this study? Main question: To what extent do agent type and responsive communicative behavior influence the acceptance and the perceived performance of chat interactions in the student counseling context? Perceived performance is conceptualized in terms of satisfaction with the interaction. As acceptance is a heterogenous construct and this study is conducted as part of a larger research network, different facets of acceptance are used: Attitude towards the interaction (a), intention to use (b), perceived enjoyment (c), likeability (d), perceived intelligence (e), warmth (f), competence (g).
Hypotheses:
H1: Human identity cues have a positive effect on the acceptance outcomes (a-g) and on perceived performance (μhuman identity cues > μchatbot identity cues).
H2: Responsive verbal cues have a positive effect on the acceptance outcomes (a-g) and on perceived performance (μverbal responsive cues > μno verbal responsive cues).
H3: The effects of human identity cues on the acceptance outcomes (a-g) and on perceived performance are mediated by social presence.
H4: The effects of verbal responsive cues on the acceptance outcomes (a-g) and on perceived performance are mediated by social presence.
H5: The effects of verbal responsive cues on the acceptance outcomes (a-g) and on perceived performance are mediated by perceived dialog.
RQ1: Is there an interaction effect between agent type and responsiveness on the acceptance outcomes (a-g) and on perceived performance?
RQ2: Does feeling heard mediate the effects of responsiveness on the acceptance outcomes (a-g) and perceived performance?
RQ3: Do the effects hypothesized in H1 and H2 persist when controlling for the quality of previous chatbot experience?
3) Describe the key dependent variable(s) specifying how they will be measured. Dependent variables:
Attitude, adapted from Diers (2020), Schlohmann (2012)
Use intention, adapted from Diers (2020), Schlohmann (2012)
Perceived joy, adapted from Diers (2020), Schlohmann (2012)
Likeability and perceived intelligence, adapted from Bartneck et al. (2009), translated to German
Warmth and competence, adapted from Fiske (2018), translated to German
Satisfaction, adapted from de Ruyter & Wetzels (2000), Lagace et al. (1991), translated to German
Mediator variables:
Social presence, adapted from Gefen et al. (2004), translated to German
Perceived dialog, adapted from Sundar et al. (2016), translated to German
Feeling heard, adapted from Roos (in preparation), translated to German
All items will be measured on a scale from one "absolutely disagree" to seven "absolutely agree".
4) How many and which conditions will participants be assigned to? The study has a 2x2 between-subject design. Participants will be randomly assigned to four conditions. The conditions result from the two manipulated independent variables agent type (anthropomorphic chatbot vs. human) and responsive communicative behavior (absence vs. presence of responsive verbal cues).
5) Specify exactly which analyses you will conduct to examine the main question/hypothesis. All statistical analyses will be conducted in R version 4.0.3. Summary statistics and bivariate correlations between all constructs will be calculated for descriptive purposes. To test H1 and H2, a total of 16 independent t-tests will be carried out. To answer RQ1, a total of 8 2x2 ANOVAs will be calculated. P-values (H1, H2 and RQ1) will be adjusted using the Holm (1979) correction. Mediation hypotheses (H3, H4, H5) will be tested and RQ2 will be answered using the R mediation package (version 4.5.0). A total of 24 (3 hypotheses x 8 outcome variables) mediation models will be tested for H3, H4 and H5. A total of 8 mediation models will be tested for RQ2. A total of 16 linear regression models will be computed in order to answer RQ3 (8 outcome variables x 2 independent variables, controlled for the quality of previous chatbot experience). The linear regression models will also include a control variable indicating participants' confidence in being able to speak adequately with chatbots.
6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations. Participants who fail the agent type manipulation check ("If you think back to the chat you just saw: Who was Marc talking to? - to the student advisor Sophie - to the professor Sophie - to the chatbot Sophie - to the doctor Sophie - don't know") will be excluded from analysis.
7) How many observations will be collected or what will determine sample size? No need to justify decision, but be precise about exactly how the number will be determined. This study is a replication of a previous study we conducted with 253 participants. As the a-priori computed sample size could have been too low to detect significant effects in the previous study, we now aim to recruit 400 participants (total sample size based on 80% power, small to medium effect sizes, alpha error probability of p < 0.05) via the circular email offer of a university in a middle-sized town in the southwest of Germany. Based on the results of the previous study we expect to exclude at least 10% of participants from analysis due to our manipulation check exclusion rule. We therefore aim to recruit 440 participants in total.
8) Anything else you would like to pre-register? (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?) Exploratory factor analysis will be used in order to check internal consistency of the scales of the dependent and mediator variables. RQ1, RQ2 and RQ3 will be analyzed exploratively. In order to characterize our sample afterwards, gender and age are surveyed. For exploratory analyses, chatbot knowledge, use frequency, use contexts, use motivations and general attitudes towards AI (Schepman & Rodway, 2020) and algorithms are surveyed. An additional questionnaire part is made up of questions on the perceived moral agency of the chat agent (Banks, 2019), their use of and behavior in video calls and their smart speaker use. The data might be further analyzed for exploratory purposes (e. g. linear regression models including control variables, structural equation modeling).
Amendment/explanation regarding question no. 1: We have already collected some data via the above-mentioned way (question no. 7). Since response rates via the circular mail were unexpectedly low, we are now expanding our population to include students from German universities who are older than 18. We will recruit additional participants via circular mail offers from other German universities as well as via facebook groups with German university students.
