Author(s) This pre-registration is currently anonymous to enable blind peer-review. It has 2 authors.
Pre-registered on 2024/08/07 04:39 (PT)
1) Have any data been collected for this study already? No, no data have been collected for this study yet.
2) What's the main question being asked or hypothesis being tested in this study? Participants will read and evaluate an article written by an AI (LLM = Large Language Model; actually written by ChatGPT-4o) in a 2-factorial between-groups design. We manipulate the alleged authorship of an article and the AI framing, i.e., the information participants are provided with about LLMs before reading the article.
We will first test the following open research question for each of the dependent variables:
RQ1: Is there a difference in the No information framing condition compared to the basic information condition?
If these two conditions do not differ, we will merge them in the subsequent analysis.
Hypotheses regarding message and source credibility
- H1/4: We expect a main effect of authorship, with significantly higher ratings for articles allegedly co-authored compared to those authored solely by AI.
- H2/5: We expect a main effect of AI framing, with significantly higher ratings for articles in the strengths framing condition compared to the articles in other AI framing conditions.
- H3/6: We expect an interaction effect of authorship and AI framing. Specifically, ratings will be significantly lower for the AI author than for the Co-author in the limitations framing condition, compared to the other framing conditions.
Hypotheses regarding perceived intelligence and anthropomorphism
- H7/9: We expect a main effect of authorship, with significantly higher ratings for the Human-AI-Co-author compared to the AI-author.
- H8/10: We expect an interaction effect of authorship and AI framing. Specifically, ratings will be significantly lower for the AI author than for the Co-author in the limitations framing condition, compared to the other framing conditions.
Hypothesis regarding behavioral intentions
- H11: We expect a main effect of authorship, with significantly higher ratings in the Human-AI-Co-author conditions compared to the AI-author conditions.
We state no further hypotheses regarding behavioral intentions, intelligence, and anthropomorphism. As we always test the full model, these effects will be exploratively tested.
RQ2: Does the balanced information condition differ from the strengths and limitations conditions for each of the dependent variables?
3) Describe the key dependent variable(s) specifying how they will be measured. • Message credibility: 19 items (Appelman & Sundar, 2016; Sundar, 1999; 7-point Likert scale).
• Source credibility: 5 items (Flanagin & Metzger, 2000, 2007; 7-point bipolar scale).
• Anthropomorphism and intelligence: 10 items (Bartneck et al., 2009; 5-point bipolar scales).
• Behavioral intentions: 4 items (adapted from Venkatesh et al., 2003; 5-point Likert scale).
4) How many and which conditions will participants be assigned to? Participants will be randomly assigned to one of 10 between-group conditions resulting from the between-factors authorship and AI framing. The authorship will be manipulated by stating that the text was either written solely by an LLM or by a human with the help of an LLM (Human-AI-Co-Authorship). The AI framing conditions will differ as follows:
- No Information: no additional information regarding LLMs will be provided vs.
- Basic Information: a brief definition of LLMs will be provided vs.
- Strengths Information: information about the strengths of LLMs will be provided vs.
- Limitations Information: information about the limitations of LLMs will be provided vs.
- Balanced Information: information about both strengths and limitations will be provided
5) Specify exactly which analyses you will conduct to examine the main question/hypothesis. We will conduct linear regression models on each dependent variable to analyze the impact of AI framing and authorship. Therefore, we will first test a model without the interaction term against a model with interaction term (authorship x AI framing) and only proceed with the interaction model if it contributes significantly to the prediction. Custom contrasts will be set to compare Strengths against all other conditions, Limitations against all other conditions, and to exploratorily compare Balanced against Strengths and Limitation.
6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations. We will exclude participants who incorrectly answer the manipulation check for the two factors authorship or AI framing. Moreover, we will exclude participants who incorrectly answer the attention check related to the content of the article.
7) How many observations will be collected or what will determine sample size? No need to justify decision, but be precise about exactly how the number will be determined. A power-analysis for a small effect size of f = 0.14, an alpha error probability of .05, and a power of .80 revealed a total sample size of 614 participants. As we suspect that we will have to exclude participants due to incorrect answers to the manipulation checks or the attention check, we will collect observations from roughly 720 participants.
8) Anything else you would like to pre-register? (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?) We will also measure participants' experience with AI-generated texts (if they have ever heard about this technology or read an AI-generated text, and if they have ever used an LLM) as well as their prior attitudes toward LLMs by 5 items adapted from Venkatesh et al. (2003). Moreover, we will measure participants' prior self-assessed knowledge about AI on a 5-point single item from "no knowledge" to "extensive knowledge". Furthermore, participants will be asked how neutral they perceived the tone of the text by one five-point bipolar item from "absolutely neutral" to "absolutely evaluative".
For exploratory purposes, we will include the pre-attitudes toward LLMs and perceived intelligence as covariates in the analyses for message and source credibility.
