Potential Field “Multimodal Communication”

Live sports reporting provides a unique opportunity to study the joint effect of linguistic features and visual information on the choice of word order and referential expressions. Multimodality differs for speaker and hearer between radio and TV reports because the speaker always relies on visual input whereas only TV reports provide the hearer with access to the visual input. In preparation of a larger research project, we plan a pilot experiment requiring participants to describe picture stories showing sport events to determine how linguistic prominence and visual salience interact during spontaneous language production.

Keywords: Linguistic prominence, visual salience, language production, sports reporting, TV and radio.

This project investigates the limits of linguistic iconicity. Segmental and prosodic properties of ideophones are systematically manipulated, and participants are asked to assess whether the resulting forms allow for an iconic or non-iconic interpretation. The aim is to determine at what point a modification causes the iconic meaning to be lost.

Keywords: Iconicity, ideophones, perception.

The central question is how gestures in a language—in this case, Totoli—align with prosodic landmarks when there are no word- or sentence-level prosodic prominence markers such as lexical stress, lexical tone, or pitch accents. Gesture–prosody coordination has primarily been studied in languages with phrasal prominence, where tones and accents have been established as anchor points for gestures. In Totoli, we aim to investigate how gesture–prosody coordination interacts with pragmatic prominence, specifically focus and the givenness of discourse referents.

Keywords: Prosody–gesture coordination, prominence, prosodic and gestural landmarks.

This short-term collaboration focuses on transferring a research approach from linguistics to the study of the production and perception of theatre and performance art. It starts from the question of whether the partly conscious and partly unconscious direction of visual and auditory attention in the performing arts can be described and analyzed more precisely using approaches from linguistic research on prominence. To this end, we plan to apply a range of concepts and analytical tools from linguistic prominence research to this new domain.

Keywords: Prominence, multimodal artistic performances.

Semantic satiation refers to the observation that repeated exposure to a meaningful item can temporarily reduce access to its meaning. While this effect has been studied mainly with spoken and written words, the proposed project will, for the first time, adapt the semantic satiation paradigm to gesture comprehension by repeatedly presenting gesture videos and measuring subsequent semantic processing in a classical behavioural paradigm such as, for example a semantic categorization task. By comparing different gesture types, such as emblematic and iconic gestures, the study will provide a first test of whether meaning “fatigue” extends to visual-manual communication and to which kinds of visual expressions. Additionally, our collaboration, bringing together perspectives from linguistics, psychology, as well as from literary and media studies, provides an interdisciplinary resource for future research on repetition and semantic processing.

Keywords: Semantic satiation, gesture studies, language processing, multimodality, psycholinguistics.

The proposed project extends the sentence recall paradigm to multimodal communication in order to investigate how gestural information is retained and reproduced. Two pilot studies will evaluate the feasibility of the method and test whether recall behavior is sensitive to theoretically motivated differences between gesture types and gesture positions. The project lays the groundwork for a novel production-based approach to studying the semantics and pragmatics of gesture.

Keywords: Multimodal language production, gesture semantics, psycholinguistics, methodological advancement.

“Semantic satiation” refers to the empirical observation that a large number of repetitions of a word leads to the loss of its meaning in the speaker, reader, or listener. While this effect has been studied with visual and auditory word presentation, it has so far not been examined in a multimodal communication setting. Here, we propose to use EEG for a more direct test of semantic satiation than in previous work, and apply for funding to support a pilot EEG study with visual word presentation to test this novel approach. This work will provide a basis for studying semantic satiation in written, auditory (spoken words), visual (gestures and signs), and multimodal (verbal plus gestural) communication settings.

Keywords: Semantic satiation, word processing, repetition, EEG, machine learning regression, encoding model.

Imaginary Multimodal Machines: A Participatory Museum will develop a beta online catalogue of existing fictional, speculative, metaphorical, and unrealized devices for producing, organizing, and communicating knowledge. Drawing on literature, cinema, philosophy, scientific speculation, media theory, and technological history, the catalogue will examine how imagined machines combine text, images, numbers, diagrams, sound, gesture, and spatial organization. Invited members of the MultiCom network will be able to propose additional entries through a moderated, token-protected submission form.

Keywords: Multimodal communication, imaginary media, epistemic machines, media archaeology, knowledge infrastructures.

The workshop brings together researchers from MultiCom and authors of the Multimodal Appreciation Toolkit developed at the HU Berlin. The workshop will create a space for valuing and assessing multimodal works within interdisciplinary settings and will lay the groundwork for future collaboration within the network.

Keywords: Evaluation, multimodal appreciation, methodological challenges, immersion.

The proposed project investigates how visual and textual information interact in the comprehension of visual narratives, employing an interdisciplinary approach that merges formal-semantic analysis with psycholinguistic experimentation. To explore the roles each modality plays and how their combination might facilitate (or hinder) language comprehension, we will conduct a pilot study comparing comprehension across unimodal and multimodal conditions. Results will lay the foundation for a planned eye-tracking study investigating multimodal language processing and comprehension in visual narratives.

Keywords: Visual narratives, multimodal meaning construction, semantics, psycholinguistics.

Research on European languages using Latin script has demonstrated recalibration effects, in which decoding of linguistic information (e.g., identifying a presented pseudoword as either /aba/ or /ada/) is biased by information from another sensory modality (e.g., vision of written text). Two unusual features of Thai make it an insightful language for studying recalibration phenomena from a new perspective: (a) Thai phonology exhibits a three-way distinction between phonemic categories on a voice-onset-time continuum, where previous research has relied on languages with only a two-way distinction, and (b) this three-way distinction on a phonetic continuum is reflected by corresponding similarity on a graphetic (visual character similarity) continuum. We propose a study adapting the standard recalibration study design to study recalibration effects in Thai, providing a foundation for a new perspective from which to study multimodal interactions in visual and auditory word recognition.

Keywords: Thai, perception, recalibration, phonology, orthography.