SIEMPRE - Social Interaction Entertainment Using Music Performance

Description

The goal of the SIEMPRE project is to develop novel research theoretical and methodological frameworks, computational models, and algorithms for the analysis of creative communication within groups of people.

Research in SIEMPRE is focused on ensemble musical performance and audience experience, an ideal test-bed for the development of models and techniques for measuring creative social interaction in an ecologically valid framework.

In particular, the focus is on exploring interpersonal interaction in (i) musician-musician, (ii) conductor-musicians, and (iii) musician-listener scenarios. The overarching strategy of the project is to study the interpersonal processes that make live performance and listening unique phenomena.

We focus on three aspects of the phenomenon - entrainment (which creates physical alignment between the individuals); emotional contagion (which creates emotional bonds between them); and co-creation, by which both performers and audience contribute to shaping the overall event. We study each at two levels - between performers, and between performer and audience.

Entrainment, emotional contagion, and co-creation will be investigated both during an explicit communication process, such as that occurring between the orchestra conductor (or the group leader) and the other musicians, and during implicit synchronisation in emotionally intense experiences (joint music performance such as in a string quartet; audience live experience).

Real-time extracted multimodal features on movement, audio, and physiological data will be used to analyse expressive qualities, e.g., starting from research on multimodal expressive gesture (Camurri et al., 2005) and quantitative emotional estimation (Knapp et al., 2009): we refer to such multi-level features as expressive Movement, Audio, Physiological (eMAP) features. eMAP features are the inputs to the theoretical and computational models that will be studied in the project, capable to explain the processes underlying interpersonal creative communication. eMAP features will be extracted from participants (musicians, listeners, conductor) using real-time, synchronized, multi-modal feature extraction techniques.

The challenges and objectives of SIEMPRE are ambitious, and require cross-fertilisation among several disciplines: human-computer interaction, computer science, neurophysiology, psychology, cognitive musicology, artistic and aesthetic theories. Research challenges address the following questions:

  1. What are the key factors driving the interpersonal synchronization among subjects (e.g., visual and auditory expressive cues, rules and conventions) and how these factors can be measured in ecological environments? In what ways do these vary according to the physical, emotional and social contexts?
  2. How can specific roles inside the group be identified (e.g., leadership, hierarchy) and measured? Can general principles be discerned concerning the influence of some individuals over others?
  3. What are the factors that determine feelings of group cohesion or a sense of shared meaning? Can the validity of such reports by participants be supported or confirmed by measures of expressive gesture, auditory cues, physiological signals?
  4. How does social context affect individual intrapersonal synchronization of expressive movement, auditory, physiological signals and vice-versa? In what ways does the emotion of the individual affect the collaborative creative product?
  5. Which are the neurophysiological foundations of creative group communication?
  6. Can performing arts and humanistic theories contribute to advances in scientific research on non-verbal social signal processing?
  7. Which impact of the research on future ICT systems: e.g., user-centric media, future media internet, future social media?