Multimodality Research at Goethe University and RMU

ViCom investigates the special features and linguistic significance of visual communication. This comprises sign languages as fully developed natural languages which exclusively rely on the visual channel for communication, but also visual means that enhance spoken language such as gestures. It aims at disclosing the specific characteristics of the visual modality as a communication channel and its interaction with other channels (especially the auditory channel) to develop a comprehensive theoretical linguistic model of human communication and its cognitive foundations.

Visit the official ViCom website →

The Potential Field Multimodal Communication strengthens research on multimodality at Goethe University and across the Rhine–Main Universities by bringing together researchers from different disciplines, subfields, and methodological traditions.

Its activities include an interdisciplinary network of associated researchers, workshops and opportunities for structured exchange, and a Short-Term Collaboration Program supporting eleven collaborative projects on multimodal communication.

Visit the MultiCom website →

This project uses experimental methods to investigate the at-issue status of different forms of modification. We compare modification in the visual domain (e.g., speech-accompanying gestures) and in the domain of speech (e.g., adverbials, verbs, ideophones). We focus on the question of how these different information types affect entry into and retrieval from the common ground. Based on the experimental data, the project will contribute to a formal semantic model of the common ground that takes modality, iconicity, and at-issueness into account.

View the official project page →

How do speech and gestures combine to create meaning? CoSGrIn-VR develops computational models of speech-gesture integration and tests them in immersive virtual reality environments. Using avatars, motion tracking, gaze data, and AI-supported annotation, the project examines how gestures relate fully, partially, or indirectly to spoken language. The findings will advance formal semantics, multimodal communication research, and innovative VR-based experimentation.

View the official project page →

How do spoken and sign languages use iconic expressions to convey meaning? This project investigates ideophones in spoken languages and idiomatic signs in sign languages as examples of lexicalized demonstrations—forms that combine conventional meaning with expressive depiction. Using linguistic databases, experiments, and semantic analysis across German, Akan, German Sign Language, and Ghanaian Sign Language, the research develops a cross-modal theory of how iconic meaning works across modalities. The findings will advance language theory while creating valuable resources for Deaf communities and the wider public.

View the official project page →

How do speakers highlight important information using voice and gesture together? This project investigates how information structure—especially focus—is expressed multimodally through prosody and gesture in Catalan and German. Combining production, perception, and cognition studies, the research explores how acoustic and visual cues help structure discourse and guide understanding. The findings will advance models of multimodal language processing and the interaction between pragmatics, gesture, and prosody.

View the official project page →

How do language, gestures, and images express perspective together—or differently? This project investigates how meaning is shaped across spoken language, gesture, emojis, memes, and other visual demonstrations in multimodal communication. Combining formal semantics with experimental methods, the research examines which visual elements carry meaning and how they express stance, emotion, or viewpoint. The findings will advance our understanding of perspective taking and contribute to the study of visual communication across modalities.

View the official project page →

This project builds a VR-based multimodal corpus of natural dialogue to investigate how gestures, facial expressions, and speech interact in communication. It combines innovative recording and machine-learning methods to analyze visual and non-visual meaning, supports formal approaches to multimodal meaning composition, and provides an open resource for research on gesture, dialogue, and human-machine interaction.

View the official project page →

While the lexical items of human spoken languages are to a large part conventionalized and hence to a large extend non-iconic, there are pockets of iconicity in human communication. We will investigate three kinds of iconicity: speech-accompanying manual gestures, ideophones as e.g. helterskelter, and prosodic modulations as in boooring. The central research issue is the pragmatic status of such iconic meanings: Do they contribute to what is said (i.e., are they at-issue), and under which conditions can they become at-issue? If they are not at issue, are they a type of conventional implicature, or rather a type of presupposition? Existing proposals for iconic gestures that they are supplements like appositive relative clauses, or that they are cosuppositions will be tested experimentally for gestures and for the other two types of iconic meaning. We will investigate the precise alignment of iconic manual gestures and the oral expressions that they affect, the connections of gestures with ideophones and prosodic modulations, and the way how iconic meaning components contribute to the overall meaning, e.g., whether they project under semantic operators, or whether and how they can be addressed directly in communication. We will investigate the multimodal aspects of communication by means of existing corpora, production experiments with synchronized acoustic, video and motion capture data as well as audiovisual perception experiments and behavioral measure of cognitive processing (reaction times). The goal is to gain a better understanding of the often neglected iconic aspects of meaning in human communication, and of the interplay of various types of meanings.

View the official project page →

This is a multimodal corpus on Catalan and German semi-spontaneous speech collected in the frame of the DFG-funded ViCom project MultIS.

Access the corpus →

The Working Group Media Anthropology supports exchange and collaboration among anthropologists working on media in culturally and socially situated contexts. Its activities include a continuing series of workshops and panels on multimodal digital engagements, curating, publishing, and related approaches to multimodal anthropological research.

The network is connected to MultiCom through associated researcher Simone Pfeifer, one of the working group’s co-convenors.

Visit the working group’s website →

Read about multimodal digital curating →

This kit provides a set of ­practical exercises and a framework for valuing and assessing multimodal works. It recognizes their unique capacity to weave together and activate diverse media forms, collaborative practices, and public engagement while expanding the traditional boundaries of anthropological knowledge production.

This kit is intentionally designed as an open-ended, dynamic resource. We are certain that there are dimensions we have overlooked and anticipate that new dimensions will emerge through its use. This is why you’ll find blank cards, expandable lists, adaptable templates, and blank pages throughout the kit – they are invitations for you to contribute your own insights, document new approaches, and expand the kit based on your experiences in appreciating and evaluating multimodal works. Your engagement with the kit actively shapes its evolution, making it a lively and collectively enriched resource rather than a fixed set of instructions.

The external resource is connected to MultiCom through the funded short-term collaboration Workshop on the Multimodal Appreciation Toolkit.

Explore the toolkit →