New York:

Petra Wagner

Petra Wagner is Professor of Phonetics at Bielefeld University. Her work is dedicated to the communicative functions of a wide range of non-verbal facets of speech, and to model these insights in human-machine interaction. Her research topics include the multimodal expression of prosodic prominence, the interactive functions of disfluencies, the multimodal expression of conversational feedback and the prosodic expression and function of attitudinal speech.

She is particularly interested in studying data beyond highly controlled laboratory settings, and in finding ways of gaining insights from more spontaneous, interactive settings both between humans and between humans and machines. 

Multimodal prosody – an overview with a focus on multimodal prominence

Speaking is an intrinsically multimodal activity, and prosody shows a particularly strong cross-modal link, as made visible by cross-modal similarities in the temporal distribution and form of prosodic events.

Multimodal expression has been found to contribute to various key functions of prosody, e.g., phonological phrasing, floor management and prominence expression. However, many open questions remain with respect to the exact interplay of the modalities involved.

Besides giving a broad introduction to the general topic of multimodal prosody, my talk will focus on the multimodal expression of prosodic prominence. Here, I will particularly concentrate on various aspects of cross-modal integration. Based on recent and ongoing studies, I will investigate the link between manual co-speech movements and their anchoring to structural and signal-level prosodic events. Besides, I will discuss the variability in the temporal coordination across modalities in prosodic expression.

Lastly, I will discuss my findings in the context of cross-modal integration in speech processing.


Background Literature

Wagner, P., Cwiek, A., Samlowski, B. 2019. Exploiting the speech-gesture link to capture fine-grained prosodic prominence impressions and listening strategies. Journal of Phonetics, 76, UNSP 100911. doi:10.1016/j.wocn.2019.07.001

Wagner, P., Bryhadyr, N. 2017. Mutual Visibility and Information Structure Enhance Synchrony between Speech and Co-Speech Movements. Journal of Multimodal Communication Studies, 4(1-2), 69-74. doi:10.14746/jmcs

Malisz, Z., Wlodarczak, M., Buschmeier, H., Skubisz, J., Kopp, S., Wagner, P. 2016. The ALICO Corpus: Analysing the Active Listener. Language Resources and Evaluation, 50(2), 411–442. doi:10.1007/s10579-016-9355-6

Wagner, P., Malisz, Z., Kopp, S. 2014. Gesture and Speech in Interaction: An Overview. Speech Communication, 57, 209-232. doi:10.1016/j.specom.2013.09.008

More in this category: Edward Gibson »