Development and evaluation of a mixed reality music visualization for a live performance based on music information retrieval

  • The present study explores the development and evaluation of a mixed reality music visualization for a live music performance. Real-time audio analysis and crossmodal correspondences were used as design guidelines for creating the visualization, which was presented through a head-mounted-display. To assess the impact of the music visualization on the audience’s aesthetic experience, a baseline visualization was designed, featuring the same visual elements but with random changes of color and movement. The audience’s aesthetic experience of the two conditions (i.e., listening to the same song with different visualizations) was assessed using the Aesthetic Emotions Scale (AESTHEMOS) questionnaire. Additionally, participants answered questions regarding the perceived audiovisual congruence of the stimuli and questionnaires about individual musicality and aesthetic receptivity. The results show that the visualization controlled by real-time audio analysis was associated with a slightlyThe present study explores the development and evaluation of a mixed reality music visualization for a live music performance. Real-time audio analysis and crossmodal correspondences were used as design guidelines for creating the visualization, which was presented through a head-mounted-display. To assess the impact of the music visualization on the audience’s aesthetic experience, a baseline visualization was designed, featuring the same visual elements but with random changes of color and movement. The audience’s aesthetic experience of the two conditions (i.e., listening to the same song with different visualizations) was assessed using the Aesthetic Emotions Scale (AESTHEMOS) questionnaire. Additionally, participants answered questions regarding the perceived audiovisual congruence of the stimuli and questionnaires about individual musicality and aesthetic receptivity. The results show that the visualization controlled by real-time audio analysis was associated with a slightly enhanced aesthetic experience of the audiovisual composition compared to the randomized visualization, thereby supporting similar findings reported in the literature. Furthermore, the tested personal characteristics of the participants did not significantly affect aesthetic experience. Significant correlations between these characteristics and the aesthetic experience were observed only when the ratings were averaged across conditions. An open interview provided deeper insights into the participants’ overall experiences of the live music performance. The results of the study offer insights into the development of real-time music visualization in mixed reality, examines how the specific audiovisual stimuli employed influence the aesthetic experience, and provides potential technical guidelines for creating new concert formats.show moreshow less

Download full text files

Export metadata

Metadaten
Author:Matthias ErdmannORCiD, Markus von BergORCiD, Jochen SteffensORCiD
Qualitätssicherung:peer reviewed
open access:Gold - Erstveröffentlichung mit Lizenzhinweis
Institutes:Institute of Sound and Vibration Engineering (ISAVE)
Research fields:Informationstechnologie / Internet der Dinge
Kognition und Wissen / Innovation
Kultur / Kreativität und Darbietung
Kultur / Medien
Fachbereich/Einrichtung:Hochschule Düsseldorf / Fachbereich - Medien
Document Type:Article
Year of Completion:2025
Language of Publication:English
Publisher:Frontiers
Parent Title (English):Frontiers in Virtual Reality
Volume:6
Article Number:1552321
Page Number:18
URN:urn:nbn:de:hbz:due62-opus-51110
DOI:https://doi.org/10.3389/frvir.2025.1552321
ISSN:2673-4192
Tag:DFG Publikationskosten; HSD Publikationsfonds
aesthetic experience; concert formats; mixed reality; music information retrieval; music visualization; real-time audio analysis
GND Keyword:Virtuelle RealitätGND; Mixed RealityGND; AudiovisualisierungGND; Information RetrievalGND; Ästhetische WahrnehmungGND; MusikwahrnehmungGND; Head-mounted DisplayGND
Funding institution:DFG / zentraler HSD-Publikationsfonds
Corresponding Author:Matthias Erdmann
Information on the Research Data:The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.
Dewey Decimal Classification:0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 000 Informatik, Informationswissenschaft, allgemeine Werke
Licence (German):Creative Commons - CC BY - Namensnennung 4.0 International
Release Date:2025/03/25
Note:
Funding:
Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 532148125 and supported by the central publication fund of Hochschule Düsseldorf University of Applied Sciences.
Diese Webseite verwendet technisch erforderliche Session-Cookies. Durch die weitere Nutzung der Webseite stimmen Sie diesem zu. Unsere Datenschutzerklärung finden Sie hier.