How Semiotic Modes Work Together in Multimodal Texts: Defining and Representing Intermodal Relations
Journal Article by Martin Siefkes (Chemnitz)
Abstract
This article investigates an area of multimodality research that has received relatively little attention: relations between semiotic modes in multimodal texts. It sketches an approach towards intermodal relations with the aim of categorising them, axiomatising them, and representing them as part of a formal discourse analysis.
This article proposes a model that formalises and represents intermodal relations, and applies it to the analysis of a range of examples. It builds on Segmented Discourse Representation Theory (SDRT), a formalised theory that allows the representation of discourse segments and the inference of relations between them. The proposed model for text and discourse representation considers three strata: (1) the stratum of expression or form; (2) the stratum of semantics (including discourse relations); (3) the stratum of style.
On the basis of this extended discourse representation model, different relations between modes can be represented. The approach defines a number of Intermodal Relation Types (IRTs), such as Disambiguation, Typification, and Intermodal Predication, gives axioms for their inference, and shows how they can be included into a formal discourse representation.
An important innovation is the possibility to represent how expression properties (e.g. the adjacency of two pictures or similar or different font sizes of headlines) interact with content properties (e.g. who is depicted on the images or what is mentioned in the headlines), and with stylistic properties (e.g. the use of traditional or contemporary layout).
The Contribution
The Author
Martin Siefkes is Research Assistant at the Technical University Chemnitz (Institute for German Language and Communication). His research focuses on multimodal linguistics, discourse analysis, stylistics, and visual aesthetics. In his master’s thesis (2006), he investigated discourses surrounding German colonialism and the Nama-Herero-uprising. In 2009, he was a visiting scholar at the Institut Jean Nicod in Paris. His doctoral dissertation on the semiotics of style was completed in 2010, and awarded the Young Researcher’s Award of the German Semiotics Society. From 2011 to 2013, he was a Research Fellow at the IUAV in Venice. In 2014, a research project on the analysis of multimodal texts brought him to the University of Bremen. Since 2015, he co-edits the Zeitschrift für Semiotik. Current research and publications can be found under http://siefkes.de.