'Facial signals are associated with social actions of questions' (AsPredicted #83223)
Author(s) Naomi Nota (MPI, Donders Institute) - naomi.nota@ed.ac.uk James Trujillo (Max Planck Institute for Psycholinguistics) - james.trujillo@mpi.nl Judith Holler (Max Planck Institute for Psycholinguistics) - judith.holler@mpi.nl
Pre-registered on 2021/12/15 - 06:49 AM (PT)
1) Have any data been collected for this study already? It's complicated. We have already collected some data but explain in Question 8 why readers may consider this a valid pre-registration nevertheless.
2) What's the main question being asked or hypothesis being tested in this study? As a follow-up of Nota et al. (2021), we will investigate a more fine-grained categorization of different social actions of questions in the same corpus of 34 dyadic face-to-face Dutch face-to-face conversations. These social actions consist of 8 categories: information requests (InfReq), understanding checks (UndCheck), self-directed questions (SelfDir), stance or sentiment questions (StanSem), other-initiated repairs (OIR), active participation questions (ActPart), questions intended for structuring, initiating or maintaining conversation (SIMCo), and plans and actions questions (PlanAct).
Our main research questions are:
(1) What is the distribution of facial signals across social actions, and are there social-action-specific clusters of facial signals?
(2) What are the timings of facial signals within social actions?
(3) What is the temporal organization of facial signals with regard to one another across the different social actions?
We hypothesize that social actions will differ with respect to the facial signals they are associated with. In line with Nota et al. (2021), we further hypothesize that most facial signals will occur around the start of the utterance (i.e., eyebrow movements such as frowns, raises, frown raises, eye widenings, squints, blinks, gaze shifts, nose wrinkles). Additionally, we expect that some facial signals will occur predominantly late in the utterance (i.e., mouth movements such as pressed lips, mouth corners down, and smiles). Due to the lack of systematic large-scale analyses of facial signals and social actions, we do not make further predictions.
3) Describe the key dependent variable(s) specifying how they will be measured. We will examine annotations made in a corpus of 34 dyadic face-to-face Dutch face-to-face conversations (CoAct corpus, ERC project led by JH).
4) How many and which conditions will participants be assigned to? NA
5) Specify exactly which analyses you will conduct to examine the main question/hypothesis. Analyses per research question:
(1)
a. Quantify the proportion of each facial signal that has an overlap with a social action out of the total amount of overlaps of that facial signal with all social actions. If the durations of utterances are different across social actions, we will standardize the amount of overlap to utterance length.
b. Study whether there is a general effect of social actions on facial signal distribution using generalized linear mixed-effect models (GLMM) in R (R Core Team, 2019) with RStudio (RStudio Team, 2019). The models will be run with the lme4 package for R. If there are convergence issues with these models, we will use mixed-effect models instead. The fixed effects will be social action and the utterance count per social action. We will use the maximal random effects structure justified by the experimental design and for which convergence will be reached. We will use an ANOVA to compare the maximal model to a null model without social action as a predictor, to see whether the null model can be rejected. We will apply a Bonferonni correction to counteract the problem of multiple comparisons, and perform a post-hoc analysis among groups after fitting the models using the emmeans package for R.
We specify the model as follows:
M1 <- glmer(Facial signal count ~ social action + utterance count + (1 | File) )
M0 <- glmer(Facial signal count ~ utterance count + (1 | File) )
c. Perform a statistical analysis consisting of Decision Tree (DT) models (Loh, 2011) to find out whether the social actions are distinguishable based on the set of facial signals that accompany them.
(2)
a. Standardize all social action utterances from 0 (onset) to 1 (offset), and plot facial signal onsets relative to the utterance duration for each social action.
(3)
a. Determine how many unique sequences there are per social action, and look at the proportion of each of these unique sequences out of the total amount of that specific sequence per social action.
b. Explore possible sequences, or patterns of facial signals, in social actions using Markov chains (Abbott, 1995; Bakeman & Quera, 2011). We will wrangle the data to form sequences of facial signals in order of appearance per utterance for each social action category, to create a dataframe with a sequence per row. We will determine the transitional probability between each pair of facial signals over all sequences within each social action. The Markov chains are our prime focus; however, it may be that wrangling the data to a desired input for this analysis is not possible, due to the format of the annotations and the vast amount of possible combinations of facial signal sequences for all utterances. If this is the case, we will use an alternative approach to capture the sequential patterns of facial signals in social actions.
6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations. Any facial signal annotation that started or ended within 80 ms (two frames) of an unrelated speech boundary will be excluded from the analysis.
7) How many observations will be collected or what will determine sample size? No need to justify decision, but be precise about exactly how the number will be determined. NA
8) Anything else you would like to pre-register? (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?) This study is based on already collected corpus data (in the form of annotated videos); however, the analyses have not yet been performed. Thus, the study can be considered a valid pre-registration.
Hypothetical results for the three main research questions:
(1)
a. Eight bar plots with the proportional distribution of each facial signal per social action category.
b. Mixed effect results showing that facial signal count can statistically predict social action and utterance count.
c. One DT model plot showing that it is statistically possible to distinguish between the social actions based on facial signals.
(2)
a. Eight density plots (one per social action) with all facial signals, showing facial signal onsets relative to utterance duration standardized from 0 to 1 (including facial signal onsets that occur before 0 or after 1).
(3)
a. Eight tables (one per social action) with a count and proportional distribution for all unique facial signal sequences per social action.
b. Eight transitional diagrams (one per social action) showing different likelihoods between facial actions for each of the social actions. Different counts and proportional distributions of facial signal sequences for each of the eight social actions.
