PAB & Paper #2

Multimodality in the field of linguistics: An Interview with Dr. Defibaugh (paper #2)

My first in-depth exposure to the imperatives put in place with multimodality was by attending DMAC—the Digital Media and Composition 10-day institute hosted by Drs. Cindy Selfe and DeWitt, at The Ohio State University. I went to this institute mainly because I could intuit the imperative of multimodality, and the affordances set into motion by digitality, but I was afraid of all that might mean and the ways it would change my teaching and my approach to the discipline. We certainly engaged in the technological affordances of blogs and coding and video-editing, and captioning, etc… but more importantly, we discussed inclusion and the changing definitions of text.

So, I’m not completely new to the concepts of mulitmodality, especially as they pertain to pedagogy (my primary focus professionally). However, where I needed more clarity was in the theoretical underpinnings of multimodality, its history, its disciplinary trajectory, and its definitions from the field that most naturally aligns with semiotics: linguistics. The more I read about the topic, the more I find myself centered in the sub-discipline (or discipline) of linguistics, particularly sociolinguistics, with scholars like Kress, Bezemer, VanLeeuwen, Halliday, etc… To better understand multimodality within linguistics scholarship, I chose to interview Dr. Defibaugh for more information specifically reagrding how linguists encounter multimodality and how that experience takes shape in the scholarship.

Dr. Defibaugh’s research is not centered on sociolinguistics, per say, but I find her work regarding nurse practitioner-patient interactions and consumer pharmaceutical ads as part of a similar effort to challenge to the borders of traditional academic inclusivity, into the extra-academic semiotic territories. Her perspective, consequently, attracted my attention.

Here’s what I asked of her:

1.  In what ways/contexts have you encountered the term multimodalityin the scholarship within your own field of linguistics?
2. How would you (or how have others who identify as linguists) define(d) multimodality?
3. Have you encountered any key pieces of scholarship/key scholars that tie multimodality and linguistics/sociolinguistics together that you’d recommend? Any key questions being asked among scholars in this field? [As far as I can tell, the only link is the methodology of Multimodal Discourse Analysis. Am I missing something?]
4. What imperative (if any) do you see on sociolinguists to study discourse that concerns the messages designed in multiple modes, using the full range of semiotic resources?

For Dr. Defibaugh, from a linguist’s purview, multimodality was introduced through the scholarship, namely in Lorenza Mondada’s work: “Multimodal resources for turn-taking: pointing and the emergence of possible next speakers” about gestures in discourse. She goes to say that:

Since then, I’ve seen people use the term to say that all language is multi-modal (c.f. the edited collected by Levine & Scollon); perhaps this is a push back against the scholars that want to use multi-modality as a new theoretical concept. They are right in that language is inherently multi-modal but the truth is that we don’t always study it as such. (Defibaugh).

Further, it’s interesting to hear Dr. Defibaugh note the disciplinary boundary issues that we associate with rhetorics or communication. Aspects of the field align in several different disciplinary “homes.” She says, “Sociolinguistics is a pretty varied field- some of which aligns more with the CA folks, some with the LA folks, and some do things that are really socio-phonetics, studying the sounds of a language variety” and she claims that, as far as she knows, “these scholars [don’t] consider multimodality (or anything along these lines in their research) at all.” So, while I keep coming across multimodality in terms of social semiotics, rooted primarily in the field of linguistics, Dr. Defibaugh doesn’t see the field taking it up universally as the imperative that I think Rhetoric-Composition is beginning to.

When asked about the imperatives of multimodal approaches to scholarship directly, she replied:

My opinion always goes back to the question of what is the object of inquiry. Depending on the answer to that question, including multi-modal linguistic resources may be 1) completely necessary or 2) it may be simply enlightening, adding to what you already found by studying the verbal aspects of communication or 3) it may be outside the purview of the particular study (such as the socio-phonetic work). (Defibaugh)

Dr. Defibaugh sees no inherent need to study multimodality. I find that Dr. Defibaugh’s relatively non-committal response to the imperative of multimodality to be in contrast to Kress’ assertion that higher education must move to be inclusive. Kress, in an interview with Anders Bjorkvall at the opening of Multimodality in Higher Education, likened higher education to tilling a field. He used the word Umbruch to describe what happens when the earth turns over as you plough and compares that moment to what he believes academia is experiencing– a “period of ‘Umbrach’” (21). This Umbrach marks a period where the ways that knowledge is enacted and produced and consumed is under heavy examination. We no longer prepare students for a seemingly secure place in the economy, and this changing social experience is accompanied by new questions of purpose. The question of academia’s privileging of the alphabetic text is one such imperative question, according to Kress:

We have been taught, or told, or have accepted, that language, and maybe writing, is the finest achievement of humanity, so therefore it is only when something is represented in the linguistic form and in written form it can count as knowledge to be taken seriously. Well, I think that is nonsense. (Archer and Breuer, 23)

For Kress, the social semiotic study of knowledge production and consumption is one potential answer to the question of where academia needs to look to secure its future.

Finally, when asked about the research that’s been done around multimodality in the field of linguistics, Defibaugh reminded me that terminology is key here. For example, in her field, the term semiotics is likely more ubiquitous that the term multimodality. That makes sense. I repeatedly come across the term semiotics and think it captures a larger canvas of design and meaning-making, from a communications point of view. To that end, what strikes Dr. Defibaugh as most relevant is the MDA (Multimodal Discourse Analysis) technique that is a trend in linguistics and she provided a list of source to check out:

Mondada, L. (2016), Challenges of multimodality: Language and the body in social interaction. Journal of Sociolinguistics, 20: 336–366. doi:10.1111/josl.1_12177

Mondada, L. (2007). Multimodal resources for turn-taking: pointing and the emergence of possible next speakers: pointing and the emergence of possible next speakers. Discourse Studies, 9 (2): 194-225.

LeVine, P. & Scollon, R. (2004). Discourse and Technology: Multimodal Discourse Analysis Georgetown University Press.

Manning, P. (2012). The Semiotics of Drink and Drinking. London: Continuum.

References:

Archer, Arlene and Esther Odilia Breuer (Eds.). Multimodality in Higher Education. In Fidalgo, Raquel and Thierry Olive (Eds.) Studies in Writing Series, Vol. 33, 2016.

Defibaugh, Stephanie. Personal Interview. 22 September, 2017.

**********

PAB Source #1: Werner, Courtney. “How Rhetoric and Composition Described and Defined New Media at the Start of the Twenty-First Century.” College Composition and Communication 68.4 (2017), 713-741.

In my search for research related to multimodalities, I come across new media often. The connection between the two is ambiguous to me, however, and presumably ambiguous to others (Anderson et al.). So, my search began in the latest edition of CCC, where I found a helpful article by Courtney Werner, attempting to parse out the difference in the historical use of the terms new media and multimodal.

In fact, the terms are often conflated, as exemplified by Moje when she calls for research that explicates several necessary distinctions in the study of new and multiple media and literacies” (qtd. in Werner, 715). New media and modes of composition are used almost synonymously.

This exploration matters in all kinds of ways, but one major reason is to help with the definition of our field (or at least aim in the direction of cohesive definition, even if that goal is not desirable or achievable) and its boundaries (again, if any such boundaries are necessary).

After analyzing 70 peer-reviewed articles for scholar’s use of and framing of the term new media (between 2000-2010), Werner finds 68 definitions, coded in six ways (726):

When new media is defined as multimodal, argues Werner, scholars challenge to traditional notions of writing and text serve to redefine composition. She concludes by explaining that new media has no stable meaning but is instead conflated with its prevalent characteristics (including multimodality) (735).

While Werner calls for discussion specific to new media, the implications include conversations regarding the use of the term multimodal, as well, to avoid unnecessary subfields or schisms within the field. Werner is essentially naming one of the problems with defining our field as a matter of not agreeing on the objects of research. That lack of definitional discussion leads, then, to schisms and scholars who prefer to make centrifugal motions away from the center. This is otherwise known as North’s separatist approach to fixing the disparate ideas within the field.

************************

PAB Source #2: Bezemer, Jeff and Gunther Kress. “Writing in Multimodal Texts: A Social Semiotic Account of Designs for Learning.” Written Communication 25.2 (2008), 166-95.

Because Bezemer and Kress are cited in just about every work I come across regarding multimodality and semiotics, I decided to directly engage their scholarship.

Bezemer and Kress, from earlier readings, tell me that multimodality means “finding, constructing and using terms that apply to encompass the characteristics of and potentials of all modes” (“Recognition” 6). So, they relate multimodality with modes most notably, which makes sense. To understand this assertion more fully, I explored their “Writing in Multimodal Texts: A Social Semiotic Account of Designs for Learning” to understand the “ensemble of semiotic features” used when writing in multimodal texts (168)– where the authors look closely at all the terms relevant to the semiotic situation, no matter its context.

I want to understand the components of the semiotic situation to compare those components to those we traditionally associate with the rhetorical situation. Most notably, Bezemer and Kress explore: mode, medium, design, and affordances.

Mode: Kress notes that a mode is any “socially and culturally shaped resources for making meaning” (171)– anything really that a “society decides to regard and use as a mode” (87) clearly marking mode within a social semiotic framework. The term mode helps us get away from the language-centric ways of imaging how we make meaning. The confusion in scholarship and pedagogy often lives here—in the distinction between mode and media.

In fact, Lauer’s “Contending with Terms” asserts mode over media, marking the difficulty in distinction often experienced with the scholarship and pedagogy. Mode is about process of design while media is about the product of design (instrument of distribution).

While Werner, for example, examines modality in a different way. She is specifically looking to see how “new media” has been taken up by those in the Rhet-Comp field, but names modality as one of three “prevalent characteristics” of new media, alongside digitality and interactivity (Werner 718-19).

Medium: Bezemer and Kress say it like this: a medium is both social and material, material in that it is the “substance in and through which meaning is instantiated/realized and through which meaning becomes available to others” and social in that it is the result of “semiotic, sociocultural, and technological practices” (172).

I immediately think of designing as relatively synonymous with composing, except that designing has broader connotations (maybe because we’ve come to name our lingo-centric English courses Composition), allowing for applications well beyond the alphabetic text, so I turn to Bezemer and Kress for help with how design has been positioned with composing. They seem to mirror my original hunch, that composition marks “competence in a specific practice…of a mode— writing, say” while design marks practice with making signs-as-texts, where modes, media, frames, and sites of display, as well as rhetorical purposes, characteristics of audience—all are brought into “coherence with each other” (172). So, designing is more broadly conceived that composing. I’d already chosen to shift my own language from writing to composing, aiming to more broadly capture what I think students are doing as they negotiate the rhetorical situations in which they’ve chosen to engage, but it seems like I now need to consider shifting from composing to designing, if indeed I see my role in the classroom as engendering appropriate semiotic resources for rhetorical purpose.

Bezemer and Kress also discuss affordances as a component of the semiotic situation, describing them as modal resources that “can used to do different kinds of semiotic work” (171). I can see why affordances would be more likely to be used with students, but I appreciate another way to refer to these as modal resources. In fact, I like this term better because it speaks directly to the relationship between mode and affordances.

 

If terminology and definitions of the work we do, and the work we ask our students to do, limit or expand the world as Wittgenstein claims in his famous quote, then using the terminology that invites all potential modes for consuming or designing rhetorical products, then Bezemer and Kress have given us a wealth of new, more inclusive, terms to introduce to our students.

For an interview with Gunther Kress, see “What is Multimodality” (also below).

Advertisement