#12974 | AsPredicted

As Predicted:Exploratory study of a website for collaborative argumentative discussion (#12974)


Created:       07/28/2018 03:53 PM (PT)

This is an anonymized version of the pre-registration.  It was created by the author(s) to use during peer-review.
A non-anonymized version (containing author names) should be made available by the authors when the work it supports is made public.

1) Have any data been collected for this study already?
No, no data have been collected for this study yet.

2) What's the main question being asked or hypothesis being tested in this study?
We evaluate the design of a website for collaborative argumentative discussion during a one-month long exploratory study. The main purpose is to evaluate the provided functionality to structure arguments, which we hypothesize can make it easier for laymen (unfamiliar with argumentation theory) to collaboratively integrate, evaluate, contest, and reason about information found online. Our main research question is: how to strike the right balance between introducing functionality for structured argumentation and usability in order to obtain more widespread adoption of argumentation technologies?

3) Describe the key dependent variable(s) specifying how they will be measured.
We will investigate how well our design succeeds in: (1) making arguments transparent (understood by others); (2) finding common ground; (3) being non-discriminatory, i.e., representing the opinion of minorities and repressed groups; (4) hosting conducive, noninflammatory, discussions; and (5) inspiring critical thinking.

This will be assessed based on:
- free form feedback provided by users on a subreddit dedicated to discussing the website
- responses to a survey sent out to all users (that are not part of the project) after the site has been in private beta for one month
- semi-structured interviews with different types of users, depending on how they interact or interacted with the site (post hoc analysis). For example: top contributors; users with interesting interaction patterns (e.g., users that only use one feature); and users who leave feedback, but indicate they will no longer contribute to the site.

The survey will ask for:
- the user's age, sex, and profession
- the user's engagement in offline argumentative discussion prior to private beta (average reported hours per week)
- the user's engagement in argumentative discussion on other social network sites prior to private beta (average reported hours per week)
- the user's engagement in argumentative discussion on our website during private beta (average reported hours per week)
- the user's preferred website for argumentative discussion prior to and after private beta
- the user's experience with argumentation theory and/or informal logic prior to private beta (5 point-Likert scale)
- a 5-point Likert scale for each of the 5 dependent variables investigated, including an option to provide free-form feedback to clarify the score given
- free form feedback on what would encourage the user to contribute more on the site

4) How many and which conditions will participants be assigned to?
Only our website. Minor features addressing usability concerns impertinent to the primary design of the site might be added while the study is running.

5) Specify exactly which analyses you will conduct to examine the main question/hypothesis.
Our analysis will be purely qualitative. However, we will provide descriptive statistics of the quantifiable data in the survey.

6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations.
We won't report on bugs or trivial usability issues. Instead, the primary focus of our analysis is on the core functionality of the website and how users envision it could be improved and evolve over time.

7) How many observations will be collected or what will determine sample size?
No need to justify decision, but be precise about exactly how the number will be determined.

We will announce the site publicly through personal networks and on various social network sites, trying to reach as many users as possible. Users who register do not immediately get access to the site. We will only gradually give access to users while we ensure the site keeps operating as intended, achieved by monitoring performance times and contributed content. Thus, sample size will be determined based on how many users we attract, how much time it takes to moderate contributed content, and how well the site performs with increased traffic.

8) Anything else you would like to pre-register?
(e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?)