'Acceptance and perceived performance of chat interactions (pre-study)' (AsPredicted #58,037)
Author(s) Stefanie Ritz (Leibniz-Institut für Wissensmedien, Tübingen) - s.klein@iwm-tuebingen.de Sonja Utz (Leibniz-Institut für Wissensmedien, Tübingen) - s.utz@iwm-tuebingen.de
Pre-registered on 2021/02/11 04:51 (PT)
1) Have any data been collected for this study already? No, no data have been collected for this study yet.
2) What's the main question being asked or hypothesis being tested in this study? Main question: To what extent do agent type and responsive communicative behavior influence the acceptance and the perceived performance of chat interactions in the student counseling context?
Note: As acceptance is a heterogenous construct and this study is conducted as part of a larger research network, different aspects of acceptance are used.
We draw on the following facets of acceptance:
a) Attitude towards the interaction
b) Intention to use
c) Perceived enjoyment
d) Likeability
e) Perceived intelligence
f) Warmth
g) Competence
H1: Human identity cues have a positive effect on the acceptance outcomes (a-g) and on perceived performance (μ(human identity cues) > μ(chatbot identity cues)).
H2: Responsive verbal cues have a positive effect on the acceptance outcomes (a-g) and on perceived performance (μ(verbal responsive cues) > μ(no verbal responsive cues)).
H3: The effects of human identity cues on the acceptance outcomes (a-g) and on perceived performance are mediated by social presence.
H4: The effects of verbal responsive cues on the acceptance outcomes (a-g) and on perceived performance are mediated by social presence.
H5: The effects of verbal responsive cues on the acceptance outcomes (a-g) and on perceived performance are mediated by perceived dialogue.
RQ1: Is there an interaction effect between agent type and responsiveness on the acceptance outcomes (a-g) and on perceived performance?
RQ2: Does feeling heard mediate the effects of responsiveness on the acceptance outcomes (a-g) and perceived performance?
3) Describe the key dependent variable(s) specifying how they will be measured. - Attitude, adapted from Diers (2020) and Schlohmann (2012)
- Use intention, adapted from Diers (2020) and Schlohmann (2012)
- Perceived joy, adapted from Diers (2020) and Schlohmann (2012)
- Perceived intelligence and likeability, adapted from the Godspeed questionnaire (Bartneck et al., 2009), translated to German
- Warmth and competence, adapted from Fiske (2018), translated to German
- Satisfaction, adapted from Lagace et al. (1991), translated to German
- Social presence, adapted from Gefen et al. (2004), translated to German
- Perceived dialog, adapted from Sundar et al. (2016), translated to German
- Feeling heard, adapted from Roos (unpublished), translated to German
All items are measured on a scale from one “absolutely disagree” to seven “absolutely agree”.
4) How many and which conditions will participants be assigned to? Participants will be randomly assigned to four conditions. The conditions result from the two manipulated independent variables agent type (chatbot vs. human) and responsive communicative behavior (absence vs. presence of responsive verbal cues).
5) Specify exactly which analyses you will conduct to examine the main question/hypothesis. - All statistical analyses will be conducted in R.
- Summary statistics will be calculated for descriptive purposes.
- To test H1 and H2, independent t-tests will be carried out. A total of 16 t-tests will be conducted to test H1 and H2. p-values will be adjusted using the Holm (1979) correction.
- To answer RQ1, one 2x2 ANOVA will be calculated. p-values will be adjusted using the Holm (1979) correction.
- Mediation hypotheses (H3, H4, H5) will be tested and RQ2 will be answered using the mediation package in R. A total of 24 (3 x 8) mediation models will be tested to test H3-H5. A total of 8 mediation models will be tested to answer RQ2.
6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations. Participants who are under 18 years old, whose citizenship is not German and who do not have at least a High School degree (“Abitur” in Germany) are excluded via the prescreening option on Prolific.
Participants who need less than 3 minutes to complete the questionnaire and/or fail the agent type manipulation check (“If you think back to the chat you just saw: Who was Marc talking to? - to the student advisor Sophie - to the professor Sophie - to the chatbot Sophie - to the doctor Sophie - don’t know”) will be excluded from analysis.
7) How many observations will be collected or what will determine sample size? No need to justify decision, but be precise about exactly how the number will be determined. Based on an a-priori power analysis 280 participants from the database Prolific will be recruited (70 per condition). Participants are at least 18 years old, have German citizenship and have at least a High School degree (“Abitur” in Germany).
8) Anything else you would like to pre-register? (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?) - Exploratory factor analysis will be used in order to check internal consistency of the scales of the dependent variables.
- We want to investigate exploratively if there are interaction effects between agent type and responsiveness on the acceptance outcomes and on perceived performance (RQ1).
- We want to investigate exploratively if the “feeling heard” construct mediates the effect of responsiveness on the acceptance outcomes and on perceived performance (RQ2).
- Demographics like gender, age and study experience, control variables like chatbot knowledge, chatbot use and technology commitment and a manipulation check variable for the responsiveness condition are part of the questionnaire and may be included in the analysis for exploratory purposes.
- The data will maybe be further analyzed for exploratory purposes, this may include additional evaluation methods (for example computing a correlation matrix including the dependent variables or structural equation modelling) to explore the data for generating hypotheses to test in future studies.
