huSync - A Model and System for the Measure of Synchronization in Small Groups: A Case Study on Musical Joint Action
Human communication entails subtle non-verbal modes of expression, which can be analyzed quantitatively using computational approaches and thus support human sciences. In this paper we present huSync, a computational framework and system that utilizes trajectory information extracted using pose esti...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2022-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9869836/ |
| _version_ | 1811210234343981056 |
|---|---|
| author | Sanket Rajeev Sabharwal Manuel Varlet Matthew Breaden Gualtiero Volpe Antonio Camurri Peter E. Keller |
| author_facet | Sanket Rajeev Sabharwal Manuel Varlet Matthew Breaden Gualtiero Volpe Antonio Camurri Peter E. Keller |
| author_sort | Sanket Rajeev Sabharwal |
| collection | DOAJ |
| description | Human communication entails subtle non-verbal modes of expression, which can be analyzed quantitatively using computational approaches and thus support human sciences. In this paper we present huSync, a computational framework and system that utilizes trajectory information extracted using pose estimation algorithms from video sequences to quantify synchronization between individuals in small groups. The system is exploited to study interpersonal coordination in musical ensembles. Musicians communicate with each other through sounds and gestures, providing nonverbal cues that regulate interpersonal coordination. huSync was applied to recordings of concert performances by a professional instrumental ensemble playing two musical pieces. We examined effects of different aspects of musical structure (texture and phrase position) on interpersonal synchronization, which was quantified by computing phase locking values of head motion for all possible within-group pairs. Results indicate that interpersonal coupling was stronger for polyphonic textures (ambiguous leadership) than homophonic textures (clear melodic leader), and this difference was greater in early portions of phrases than endings (where coordination demands are highest). Results were cross-validated against an analysis of audio features, showing links between phase locking values and event density. This research produced a system, huSync, that can quantify synchronization in small groups and is sensitive to dynamic modulations of interpersonal coupling related to ambiguity in leadership and coordination demands, in standard video recordings of naturalistic human group interaction. huSync enabled a better understanding of the relationship between interpersonal coupling and musical structure, thus enhancing collaborations between human and computer scientists. |
| first_indexed | 2024-04-12T04:52:21Z |
| format | Article |
| id | doaj.art-4d615d690cff463fb7a69c479ae22a5e |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-04-12T04:52:21Z |
| publishDate | 2022-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-4d615d690cff463fb7a69c479ae22a5e2022-12-22T03:47:16ZengIEEEIEEE Access2169-35362022-01-0110923579237210.1109/ACCESS.2022.32029599869836huSync - A Model and System for the Measure of Synchronization in Small Groups: A Case Study on Musical Joint ActionSanket Rajeev Sabharwal0https://orcid.org/0000-0003-2712-8453Manuel Varlet1Matthew Breaden2https://orcid.org/0000-0002-9510-363XGualtiero Volpe3https://orcid.org/0000-0003-0760-4627Antonio Camurri4https://orcid.org/0000-0003-3378-8685Peter E. Keller5DIBRIS, University of Genoa, Genova, ItalyMARCS Institute for Brain, Behaviour and Development, Western Sydney University, Penrith, NSW, AustraliaMARCS Institute for Brain, Behaviour and Development, Western Sydney University, Penrith, NSW, AustraliaDIBRIS, University of Genoa, Genova, ItalyDIBRIS, University of Genoa, Genova, ItalyMARCS Institute for Brain, Behaviour and Development, Western Sydney University, Penrith, NSW, AustraliaHuman communication entails subtle non-verbal modes of expression, which can be analyzed quantitatively using computational approaches and thus support human sciences. In this paper we present huSync, a computational framework and system that utilizes trajectory information extracted using pose estimation algorithms from video sequences to quantify synchronization between individuals in small groups. The system is exploited to study interpersonal coordination in musical ensembles. Musicians communicate with each other through sounds and gestures, providing nonverbal cues that regulate interpersonal coordination. huSync was applied to recordings of concert performances by a professional instrumental ensemble playing two musical pieces. We examined effects of different aspects of musical structure (texture and phrase position) on interpersonal synchronization, which was quantified by computing phase locking values of head motion for all possible within-group pairs. Results indicate that interpersonal coupling was stronger for polyphonic textures (ambiguous leadership) than homophonic textures (clear melodic leader), and this difference was greater in early portions of phrases than endings (where coordination demands are highest). Results were cross-validated against an analysis of audio features, showing links between phase locking values and event density. This research produced a system, huSync, that can quantify synchronization in small groups and is sensitive to dynamic modulations of interpersonal coupling related to ambiguity in leadership and coordination demands, in standard video recordings of naturalistic human group interaction. huSync enabled a better understanding of the relationship between interpersonal coupling and musical structure, thus enhancing collaborations between human and computer scientists.https://ieeexplore.ieee.org/document/9869836/Entrainmentinterpersonal synchronizationjoint actionspose estimationmusical ensemble performancesocial interaction |
| spellingShingle | Sanket Rajeev Sabharwal Manuel Varlet Matthew Breaden Gualtiero Volpe Antonio Camurri Peter E. Keller huSync - A Model and System for the Measure of Synchronization in Small Groups: A Case Study on Musical Joint Action IEEE Access Entrainment interpersonal synchronization joint actions pose estimation musical ensemble performance social interaction |
| title | huSync - A Model and System for the Measure of Synchronization in Small Groups: A Case Study on Musical Joint Action |
| title_full | huSync - A Model and System for the Measure of Synchronization in Small Groups: A Case Study on Musical Joint Action |
| title_fullStr | huSync - A Model and System for the Measure of Synchronization in Small Groups: A Case Study on Musical Joint Action |
| title_full_unstemmed | huSync - A Model and System for the Measure of Synchronization in Small Groups: A Case Study on Musical Joint Action |
| title_short | huSync - A Model and System for the Measure of Synchronization in Small Groups: A Case Study on Musical Joint Action |
| title_sort | husync a model and system for the measure of synchronization in small groups a case study on musical joint action |
| topic | Entrainment interpersonal synchronization joint actions pose estimation musical ensemble performance social interaction |
| url | https://ieeexplore.ieee.org/document/9869836/ |
| work_keys_str_mv | AT sanketrajeevsabharwal husyncamodelandsystemforthemeasureofsynchronizationinsmallgroupsacasestudyonmusicaljointaction AT manuelvarlet husyncamodelandsystemforthemeasureofsynchronizationinsmallgroupsacasestudyonmusicaljointaction AT matthewbreaden husyncamodelandsystemforthemeasureofsynchronizationinsmallgroupsacasestudyonmusicaljointaction AT gualtierovolpe husyncamodelandsystemforthemeasureofsynchronizationinsmallgroupsacasestudyonmusicaljointaction AT antoniocamurri husyncamodelandsystemforthemeasureofsynchronizationinsmallgroupsacasestudyonmusicaljointaction AT peterekeller husyncamodelandsystemforthemeasureofsynchronizationinsmallgroupsacasestudyonmusicaljointaction |