Reaching beneath the tip of the iceberg: A guide to the Freiburg Multimodal Interaction Corpus
Most corpora tacitly subscribe to a speech-only view filtering out anything that is not a ‘word’ and transcribing the spoken language merely orthographically despite the fact that the “speech-only view on language is fundamentally incomplete” (Kok 2017, 2) due to the deep intertwining of the verbal,...
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
De Gruyter
2023-11-01
|
| Series: | Open Linguistics |
| Subjects: | |
| Online Access: | https://doi.org/10.1515/opli-2022-0245 |
| _version_ | 1797561943913922560 |
|---|---|
| author | Rühlemann Christoph Ptak Alexander |
| author_facet | Rühlemann Christoph Ptak Alexander |
| author_sort | Rühlemann Christoph |
| collection | DOAJ |
| description | Most corpora tacitly subscribe to a speech-only view filtering out anything that is not a ‘word’ and transcribing the spoken language merely orthographically despite the fact that the “speech-only view on language is fundamentally incomplete” (Kok 2017, 2) due to the deep intertwining of the verbal, vocal, and kinesic modalities (Levinson and Holler 2014). This article introduces the Freiburg Multimodal Interaction Corpus (FreMIC), a multimodal and interactional corpus of unscripted conversation in English currently under construction. At the time of writing, FreMIC comprises (i) c. 29 h of video-recordings transcribed and annotated in detail and (ii) automatically (and manually) generated multimodal data. All conversations are transcribed in ELAN both orthographically and using Jeffersonian conventions to render verbal content and interactionally relevant details of sequencing (e.g. overlap, latching), temporal aspects (pauses, acceleration/deceleration), phonological aspects (e.g. intensity, pitch, stretching, truncation, voice quality), and laughter. Moreover, the orthographic transcriptions are exhaustively PoS-tagged using the CLAWS web tagger (Garside and Smith 1997). ELAN-based transcriptions also provide exhaustive annotations of re-enactments (also referred to as (free) direct speech, constructed dialogue, etc.) as well as silent gestures (meaningful gestures that occur without accompanying speech). The multimodal data are derived from psychophysiological measurements and eye tracking. The psychophysiological measurements include, inter alia, electrodermal activity or GSR, which is indicative of emotional arousal (e.g. Peräkylä et al. 2015). Eye tracking produces data of two kinds: gaze direction and pupil size. In FreMIC, gazes are automatically recorded using the area-of-interest technology. Gaze direction is interactionally key, for example, in turn-taking (e.g. Auer 2021) and re-enactments (e.g. Pfeiffer and Weiss 2022), while changes in pupil size provide a window onto cognitive intensity (e.g. Barthel and Sauppe 2019). To demonstrate what opportunities FreMIC’s (combination of) transcriptions, annotations, and multimodal data open up for research in Interactional (Corpus) Linguistics, this article reports on interim results derived from work-in-progress. |
| first_indexed | 2024-03-10T18:21:47Z |
| format | Article |
| id | doaj.art-5e2f7e57797e41fc892aa24ff6545f5f |
| institution | Directory Open Access Journal |
| issn | 2300-9969 |
| language | English |
| last_indexed | 2024-03-10T18:21:47Z |
| publishDate | 2023-11-01 |
| publisher | De Gruyter |
| record_format | Article |
| series | Open Linguistics |
| spelling | doaj.art-5e2f7e57797e41fc892aa24ff6545f5f2023-11-20T07:16:44ZengDe GruyterOpen Linguistics2300-99692023-11-0191p. 19723110.1515/opli-2022-0245Reaching beneath the tip of the iceberg: A guide to the Freiburg Multimodal Interaction CorpusRühlemann Christoph0Ptak Alexander1Germanistische Linguistik, Freiburg University, Freiburg, GermanyGermanistische Linguistik, Freiburg University, Freiburg, GermanyMost corpora tacitly subscribe to a speech-only view filtering out anything that is not a ‘word’ and transcribing the spoken language merely orthographically despite the fact that the “speech-only view on language is fundamentally incomplete” (Kok 2017, 2) due to the deep intertwining of the verbal, vocal, and kinesic modalities (Levinson and Holler 2014). This article introduces the Freiburg Multimodal Interaction Corpus (FreMIC), a multimodal and interactional corpus of unscripted conversation in English currently under construction. At the time of writing, FreMIC comprises (i) c. 29 h of video-recordings transcribed and annotated in detail and (ii) automatically (and manually) generated multimodal data. All conversations are transcribed in ELAN both orthographically and using Jeffersonian conventions to render verbal content and interactionally relevant details of sequencing (e.g. overlap, latching), temporal aspects (pauses, acceleration/deceleration), phonological aspects (e.g. intensity, pitch, stretching, truncation, voice quality), and laughter. Moreover, the orthographic transcriptions are exhaustively PoS-tagged using the CLAWS web tagger (Garside and Smith 1997). ELAN-based transcriptions also provide exhaustive annotations of re-enactments (also referred to as (free) direct speech, constructed dialogue, etc.) as well as silent gestures (meaningful gestures that occur without accompanying speech). The multimodal data are derived from psychophysiological measurements and eye tracking. The psychophysiological measurements include, inter alia, electrodermal activity or GSR, which is indicative of emotional arousal (e.g. Peräkylä et al. 2015). Eye tracking produces data of two kinds: gaze direction and pupil size. In FreMIC, gazes are automatically recorded using the area-of-interest technology. Gaze direction is interactionally key, for example, in turn-taking (e.g. Auer 2021) and re-enactments (e.g. Pfeiffer and Weiss 2022), while changes in pupil size provide a window onto cognitive intensity (e.g. Barthel and Sauppe 2019). To demonstrate what opportunities FreMIC’s (combination of) transcriptions, annotations, and multimodal data open up for research in Interactional (Corpus) Linguistics, this article reports on interim results derived from work-in-progress.https://doi.org/10.1515/opli-2022-0245talk-in-interactionmultimodalitygestureeye trackingcorpus construction |
| spellingShingle | Rühlemann Christoph Ptak Alexander Reaching beneath the tip of the iceberg: A guide to the Freiburg Multimodal Interaction Corpus Open Linguistics talk-in-interaction multimodality gesture eye tracking corpus construction |
| title | Reaching beneath the tip of the iceberg: A guide to the Freiburg Multimodal Interaction Corpus |
| title_full | Reaching beneath the tip of the iceberg: A guide to the Freiburg Multimodal Interaction Corpus |
| title_fullStr | Reaching beneath the tip of the iceberg: A guide to the Freiburg Multimodal Interaction Corpus |
| title_full_unstemmed | Reaching beneath the tip of the iceberg: A guide to the Freiburg Multimodal Interaction Corpus |
| title_short | Reaching beneath the tip of the iceberg: A guide to the Freiburg Multimodal Interaction Corpus |
| title_sort | reaching beneath the tip of the iceberg a guide to the freiburg multimodal interaction corpus |
| topic | talk-in-interaction multimodality gesture eye tracking corpus construction |
| url | https://doi.org/10.1515/opli-2022-0245 |
| work_keys_str_mv | AT ruhlemannchristoph reachingbeneaththetipoftheicebergaguidetothefreiburgmultimodalinteractioncorpus AT ptakalexander reachingbeneaththetipoftheicebergaguidetothefreiburgmultimodalinteractioncorpus |