'ChatGPT in rational-analytical vs. emotional-supportive tasks'
(AsPredicted #168,016)
Download PDF

---

---

1) Have any data been collected for this study already?
No, no data have been collected for this study yet.

2) What's the main question being asked or hypothesis being tested in this study?
We aim to investigate if there are differences in the way people use ChatGPT in the writing process depending on whether a text has to be composed in a rational-analytical or an emotional-supportive task.
In this study, we answer the following research question (RQ) and test the respective hypotheses (H):
RQ: Do people use ChatGPT differently when writing rational-analytical texts than when writing emotional-supportive texts? People's use of ChatGPT will be captured in terms of their prompting behavior, adoption behavior, time spent on the writing task, and their self-rated task contributions.
• H1: Participants will send more prompts to ChatGPT when writing rational-analytical texts than when writing emotional-supportive texts.
• H2: Participants will show greater adoption of content from ChatGPT when writing rational-analytical texts than when writing emotional-supportive texts.
• H3: Participants will spend less time on the task when rational-analytical texts than when writing emotional-supportive texts.
• H4: Participants will rate their own contribution to the text lower in the rational-analytical task than in the emotional-supportive task.

3) Describe the key dependent variable(s) specifying how they will be measured.
Our dependent variables will be 1) the number of prompts to ChatGPT, 2) the amount of text adopted from ChatGPT, 3) time spent on the task, and 4) participants' self-rated contribution to the text.
After each of two different vignettes (in which people are described who find themselves in a psychologically difficult situation), participants will rate their own contribution to the text on a slider with endpoints of 0% and 100%. The proportion of adopted text will be quantified by an Originality score calculated using Turnitin.
The number of prompts to ChatGPT and the time of writing will be extracted from log files.

4) How many and which conditions will participants be assigned to?
The study will follow a between-subjects design with two experimental conditions (perspective of writing: rational-analytical vs. emotional-supportive). The participants will be randomly assigned to one of the two conditions.
There will be two different vignettes. All participants will be presented with the same vignettes, but the instructions will differ depending on the experimental condition.
In the emotional-supportive condition, participants will be presented with the following task:
"You may use ChatGPT to compose a rational and analytical counseling response for the individuals in the provided descriptions The goal is to develop solutions based on logical reasoning and facts. Consider the individual challenges and needs of the characters as well as potential courses of action that could help them address their problems."
In the rational-analytical condition, participants will be presented with the following task:
"You may use ChatGPT to compose an emotionally supportive counseling response for the individuals in the provided descriptions. Focus on conveying compassion and understanding for the characters' feelings and experiences. Offer comfort, encouragement, and empathy to assist them in coping with their challenges. Your counseling should be empathetic and uplifting, providing emotional support to the characters in difficult situations."
Within the two conditions, the order of the vignettes will be randomized.

5) Specify exactly which analyses you will conduct to examine the main question/hypothesis.
To test the hypotheses, t-tests will be conducted. For each of the dependent variables, one mean value will be calculated for each condition.

6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations.
We will only include participants who completed the experiment. We will exclude the complete dataset of participants who switched to another tab during the study more than three times. Participants will receive a warning when they attempt to switch tabs. If they switch tabs more than three times, they will be directed to an abort page, and the study will be terminated. Furthermore, we will delete the complete dataset of participants who decide to withdraw their data at the end of the study. After the instruction at the beginning of the study, and at the end of the study, participants will be presented with a multiple-choice question from which they should select their correct instruction for the writing task. Participants who answer this question incorrectly after the writing task will be excluded.

7) How many observations will be collected or what will determine sample size?
No need to justify decision, but be precise about exactly how the number will be determined.
Our participants need to be native German speakers and must be at least 18 years old. They will be recruited through the university's mailing lists by distributing a link to the study. To incentivize participation in the experiment, participants will have the opportunity to win gift certificates for an online shop. We calculated that we would require at least n = 102 participants to achieve a power of 80%.

8) Anything else you would like to pre-register?
(e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?)
Apart from demographics (age, gender, native language), we will also collect data on participants education: are they current or former psychology students? Additionally, we will gather information on whether they currently use ChatGPT.
To ensure reading of the instructions, the screen will be locked for 30 seconds. To ensure that the task has been understood, a multiple-choice question will be presented following the instructions. If this question is answered correctly, the study will proceed. If it is not answered correctly, participants will be redirected back to the instruction page. Afterwards the study will proceed.