'MAPPinfo, mapping quality of health information – a validation study'
(AsPredicted #22546)


Author(s)
Jürgen Kasper (Norway's Arctic University, Tromsø) - k@sper.info
Julia Lühnen (University of Hamburg, Germany) - julia.luehnen@uni-hamburg.de
Jana Hinneburg (Martin-Luther-University Halle-Wittenberg) - jana.hinneburg@medizin.uni-halle.de
Jan Keppler (Christian Albrechts University, Kiel, Germany) - kepplerjan@googlemail.com
Anke Steckelberg (Martin-Luther-University Halle-Wittenberg, Germany) - anke.steckelberg@uk-halle.de
Pre-registered on
04/23/2019 06:23 AM (PT)

1) Have any data been collected for this study already?
It's complicated. We have already collected some data but explain in Question 8 why readers may consider this a valid pre-registration nevertheless.

2) What's the main question being asked or hypothesis being tested in this study?
The new instrument, MAPPinfo is reliably and validly assessing the quality of health information materials. The instrument is a checklist supposed to be used by people without a specific training in evidence based medicine or a rater training. MAPPinfo aims at measuring the extent to which criteria for quality of health information are met within specific health information materials for patients or other lay people. The set of criteria is based on the recently published guideline for evidence based health information.

3) Describe the key dependent variable(s) specifying how they will be measured.
Reliability in the case of an observation based instrument refers to inter-rater-agreement. This will be measured pairwise on itemlevel by use of the T-coefficient. T is working like Cohen's kappa, however, instead of empirically given observations using equal probabilities as marginal distributions. Validity is assessed in terms of correlations of judgements meade by users of MAPPinfo with expert judgements (construct validity 1). Additionally, validity is approached to by calculating inter-rater-reliability between judgements made by MAPPinfo and by MAPPinfo plus MAPPinfo supplement (construct validity 2). The supplement comprises expert judgements based on information provided by developers of health information materials. Moreover, divergent validity is assessed as a correlation of MAPPinfo judgements with judgements by another instrument (EQUIP).

4) How many and which conditions will participants be assigned to?
This is not an intervention study.

5) Specify exactly which analyses you will conduct to examine the main question/hypothesis.
T-coefficients and percentage agreements will be calculated pairwise for each item. T-coefficients equal or higher than .50 and percentage agreements equal or higher than .75 are assumed sufficient. Spearman's correlation coefficients for assessment of validity (construct 1 and 2) are assumed satisfactory, if they are equal or higher .55.

6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations.
Given, a health information is considered accessible with the MAPPinfo criteria set, no observations can be excluded. Health information materials are accessible if they are written for lay people and supposed to inform medical (health related) decisions between alternative strategies (one of which can be to choose "no action").

7) How many observations will be collected or what will determine sample size?
No need to justify decision, but be precise about exactly how the number will be determined.

50 health information websites will be used to test reliability. At minimum two independent working raters are needed to determine inter-rater-agreement. The same number of health information websites is aspired to be assessed also in the validation study using MAPPinfo supplement (information provided by developers). However, with regard to practical constrains, a minimum number of 25 websites is considered enough to answer this question.

8) Anything else you would like to pre-register?
(e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?)

In case reliability does not prove satisfactory in the planned pretest, a new test will be conducted with new raters of the same material after the instrument has passed a qualitative process purposing on identifying need for revision.

This validation study comprises five consecutive steps and the inherent possibility to revise the instrument and repeat validation steps until properties are convincing. We have alread begun to draft the instrument and went through the first qualitative steps. We do not consider this a problem as long as we carefully document the entire process. Quantitative data have not been collected yet.

Version of AsPredicted Questions: 2.00