Petra Wagner
Bielefeld University
Petra Wagner
Bielefeld University
Germany
Multimodal prosody – an overview with a focus on multimodal prominence
Speaking is an intrinsically multimodal activity, and prosody shows a particularly strong cross-modal link, as made visible by cross-modal similarities in the temporal distribution and form of prosodic events.
Multimodal expression has been found to contribute to various key functions of prosody, e.g., phonological phrasing, floor management and prominence expression. However, many open questions remain with respect to the exact interplay of the modalities involved.
Besides giving a broad introduction to the general topic of multimodal prosody, my talk will focus on the multimodal expression of prosodic prominence. Here, I will particularly concentrate on various aspects of cross-modal integration. Based on recent and ongoing studies, I will investigate the link between manual co-speech movements and their anchoring to structural and signal-level prosodic events. Besides, I will discuss the variability in the temporal coordination across modalities in prosodic expression.
Lastly, I will discuss my findings in the context of cross-modal integration in speech processing.
Background Literature
Wagner, P., Cwiek, A., Samlowski, B. 2019. Exploiting the speech-gesture link to capture fine-grained prosodic prominence impressions and listening strategies. Journal of Phonetics, 76, UNSP 100911. doi:10.1016/j.wocn.2019.07.001
Wagner, P., Bryhadyr, N. 2017. Mutual Visibility and Information Structure Enhance Synchrony between Speech and Co-Speech Movements. Journal of Multimodal Communication Studies, 4(1-2), 69-74. doi:10.14746/jmcs
Malisz, Z., Wlodarczak, M., Buschmeier, H., Skubisz, J., Kopp, S., Wagner, P. 2016. The ALICO Corpus: Analysing the Active Listener. Language Resources and Evaluation, 50(2), 411–442. doi:10.1007/s10579-016-9355-6
Wagner, P., Malisz, Z., Kopp, S. 2014. Gesture and Speech in Interaction: An Overview. Speech Communication, 57, 209-232. doi:10.1016/j.specom.2013.09.008
