Development and evaluation of a mixed reality music visualization for a live performance based on music information retrieval
- The present study explores the development and evaluation of a mixed reality music visualization for a live music performance. Real-time audio analysis and crossmodal correspondences were used as design guidelines for creating the visualization, which was presented through a head-mounted-display. To assess the impact of the music visualization on the audience’s aesthetic experience, a baseline visualization was designed, featuring the same visual elements but with random changes of color and movement. The audience’s aesthetic experience of the two conditions (i.e., listening to the same song with different visualizations) was assessed using the Aesthetic Emotions Scale (AESTHEMOS) questionnaire. Additionally, participants answered questions regarding the perceived audiovisual congruence of the stimuli and questionnaires about individual musicality and aesthetic receptivity. The results show that the visualization controlled by real-time audio analysis was associated with a slightlyThe present study explores the development and evaluation of a mixed reality music visualization for a live music performance. Real-time audio analysis and crossmodal correspondences were used as design guidelines for creating the visualization, which was presented through a head-mounted-display. To assess the impact of the music visualization on the audience’s aesthetic experience, a baseline visualization was designed, featuring the same visual elements but with random changes of color and movement. The audience’s aesthetic experience of the two conditions (i.e., listening to the same song with different visualizations) was assessed using the Aesthetic Emotions Scale (AESTHEMOS) questionnaire. Additionally, participants answered questions regarding the perceived audiovisual congruence of the stimuli and questionnaires about individual musicality and aesthetic receptivity. The results show that the visualization controlled by real-time audio analysis was associated with a slightly enhanced aesthetic experience of the audiovisual composition compared to the randomized visualization, thereby supporting similar findings reported in the literature. Furthermore, the tested personal characteristics of the participants did not significantly affect aesthetic experience. Significant correlations between these characteristics and the aesthetic experience were observed only when the ratings were averaged across conditions. An open interview provided deeper insights into the participants’ overall experiences of the live music performance. The results of the study offer insights into the development of real-time music visualization in mixed reality, examines how the specific audiovisual stimuli employed influence the aesthetic experience, and provides potential technical guidelines for creating new concert formats.…


| Author: | Matthias ErdmannORCiD, Markus von BergORCiD, Jochen SteffensORCiD |
|---|---|
| Qualitätssicherung: | peer reviewed |
| open access: | Gold - Erstveröffentlichung mit Lizenzhinweis |
| Institutes: | Institute of Sound and Vibration Engineering (ISAVE) |
| Research fields: | Informationstechnologie / Internet der Dinge |
| Kognition und Wissen / Innovation | |
| Kultur / Kreativität und Darbietung | |
| Kultur / Medien | |
| Fachbereich/Einrichtung: | Hochschule Düsseldorf / Fachbereich - Medien |
| Document Type: | Article |
| Year of Completion: | 2025 |
| Language of Publication: | English |
| Publisher: | Frontiers |
| Parent Title (English): | Frontiers in Virtual Reality |
| Volume: | 6 |
| Article Number: | 1552321 |
| Page Number: | 18 |
| URN: | urn:nbn:de:hbz:due62-opus-51110 |
| DOI: | https://doi.org/10.3389/frvir.2025.1552321 |
| ISSN: | 2673-4192 |
| Tag: | DFG Publikationskosten; HSD Publikationsfonds aesthetic experience; concert formats; mixed reality; music information retrieval; music visualization; real-time audio analysis |
| GND Keyword: | Virtuelle RealitätGND; Mixed RealityGND; AudiovisualisierungGND; Information RetrievalGND; Ästhetische WahrnehmungGND; MusikwahrnehmungGND; Head-mounted DisplayGND |
| Funding institution: | DFG / zentraler HSD-Publikationsfonds |
| Corresponding Author: | Matthias Erdmann |
| Information on the Research Data: | The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author. |
| Dewey Decimal Classification: | 0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 000 Informatik, Informationswissenschaft, allgemeine Werke |
| Licence (German): | Creative Commons - CC BY - Namensnennung 4.0 International |
| Release Date: | 2025/03/25 |
| Note: | Funding: Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 532148125 and supported by the central publication fund of Hochschule Düsseldorf University of Applied Sciences. |
