'Explaining DA to Participants: Experimental Analysis'
(AsPredicted #140,409)
Download PDF

---

---

1) Have any data been collected for this study already?
It's complicated. We have already collected some data but explain in Question 8 why readers may consider this a valid pre-registration nevertheless.

2) What's the main question being asked or hypothesis being tested in this study?
Briefly, we are interested in the question: How effective are different descriptions of the deferred-acceptance algorithm (DA)?

By "different descriptions", we mean the five conditions presented in 4).

By "effective" we primarily mean how effective these descriptions are in facilitating two things: (a) an understanding of DA's mechanics (i.e., how DA works), and (b) an understanding of DA's strategyproofness. Secondarily, we are interested in seeing how the different descriptions influence (c) straightforward (SF) play. We view our primary questions (a and b) as positive: these are concrete factual questions about the mechanism or its strategyproofness, which the descriptions aim to convey (regardless of how participants use this information). We view secondary interest in SF play (c) as normative, because whether or not playing straightforwardly is a desirable outcome depends on assumptions, e.g., about preferences.

We investigate three main outcome variables: (a) understanding the description/treatment itself (Descr-understanding), (b) understanding of strategyproofness (SP-understanding), and (c) straightforward play (SF-play) in incentivized rounds of playing DA. We measure Descr-understanding using integrated comprehension questions, tailored to the specific treatment, throughout the initial part of the experiment, before the incentivized rounds begin. We measure SP-understanding with a multi-part test on the concept of strategyproofness (conducted at the end of the experiment), which is the same across treatments. We are also interested in additional outcome variables such as earnings and the time spent in different parts of the experiment, as well as analyzing different sub-variables of the main three (e.g., shares of straightforward play in different subsets of rounds, specific sets of comprehension questions regarding a specific part of the description, etc).

We hypothesize that compared with the traditional description, our new menu description, which is longer and more complicated, would make it more difficult to understand the mechanics of DA; however, it would at the same time make it easier to understand that DA is strategyproof.
Of course, we also think that some participants who understand strategyproofness well will play SF in order to maximize their earnings. However, we have no strong hypothesis regarding straightforward play across the two types of descriptions, since all such hypotheses rely on assumptions about participants' preferences, and our main hypothesis relates only to participants' understanding. We also hypothesize that the correlation between understanding (the mechanics of) DA and DA's strategyproofness (as measured by our SP-understanding tests) would be higher in the menu description than in the traditional description.

3) Describe the key dependent variable(s) specifying how they will be measured.
We have two sets of main questions (positive and normative), three actual main questions (re understanding of mechanics/description, understanding of strategyproofness, and playing straightforwardly), and three main outcome variables (one long battery of questions on understanding of mechanics/descriptions, a multi-part test on understanding of strategyproofness, and simple count of straightforward play). However, we also have lots of additional outcome variables, for example subsets of the mechanics-understanding measures (including, for example, how well people did in actual allocation/matching tasks in different rounds), the time spent in different parts, attention to video explanations, specific SP-undestanding components, shares of straightforward play in different rounds and in all rounds, etc.

4) How many and which conditions will participants be assigned to?
Our five conditions correspond to five different descriptions of DA:
1. The traditional description of DA;
2. A new "menu" description of (one agent's match in) DA, based on the theoretical result in Gonczarowski, Heffetz, & Thomas (2023), under which strategyproofness holds by a one-sentence proof; to compare with #1;
3. A stripped down version of #2, where we only keep the parts of the new menu description of DA that are crucial for the one-sentence proof of strategyproofness to hold; to compare with #2;
4. A stripped down description of DA that does not describe its mechanics but states that it is its strategyproofness (according to common, textbook-esque definitions of strategyproofness that we adapted to describe to real participants); to compare with #4;
5. A null treatment that describes very little about the allocation process of DA; to compare with each of #1-4.

5) Specify exactly which analyses you will conduct to examine the main question/hypothesis.
Our first main hypothesis is that understanding of strategyproofness will be higher in Menu (treatment #2) than in Traditional (treatment #1). Our second main hypothesis is that (while Menu may be somewhat harder to understand), understanding the mechanics/description will correlate more strongly with straightforward play in Menu than Traditional. Such a correlation may or may not hold in Traditional, but we hypothesize that the correlation will be positive and stronger in Menu. Since we conjecture that strategyproofness will be made clear only when one understands certain aspects of how Menu works, we will also consider it a significant result if Menu only outperforms Traditional for those participants who understand it well (according to some reasonable metric defined by our experiment, and comparing the same percentiles of the best-understanding participants across treatments).

6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations.
We will exclude any participant who does not finish the study. Other than that, we will not exclude anyone. For robustness, we will also re-analyze the data while excluding, within each treatment, the top and bottom 2.5% of participants based on completion time.

7) How many observations will be collected or what will determine sample size?
No need to justify decision, but be precise about exactly how the number will be determined.
We plan to collect 100 observations X 5 treatments. 250 on Prolific and 250 Cornell students.

8) Anything else you would like to pre-register?
(e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?)
Last week we collected 38 observations across the five treatments to verify no big bugs in the code. We only found small bugs and fixed them. We will include these 38 observations towards our target of 500.