TY - JOUR A1 - Herder, Jens A1 - Cohen, Michael T1 - The Helical Keyboard: Perspectives for Spatial Auditory Displays and Visual Music JF - Journal of New Music Research N2 - Auditory displays with the ability to dynamically spatialize virtual sound sources under real-time conditions enable advanced applications for art and music. A listener can be deeply immersed while interacting and participating in the experience. We review some of those applications while focusing on the Helical Keyboard project and discussing the required technology. Inspired by the cyclical nature of octaves and helical structure of a scale, a model of a piano-style keyboard was prepared, which was then geometrically warped into a helicoidal configuration, one octave/revolution, pitch mapped to height and chroma. It can be driven by MIDI events, real-time or sequenced, which stream is both synthesized and spatialized by a spatial sound display. The sound of the respective notes is spatialized with respect to sinks, avatars of the human user, by default in the tube of the helix. Alternative coloring schemes can be applied, including a color map compatible with chromastereoptic eyewear. The graphical display animates polygons, interpolating between the notes of a chord across the tube of the helix. Recognition of simple chords allows directionalization of all the notes of a major triad from the position of its musical root. The system is designed to allow, for instance, separate audition of harmony and melody, commonly played by the left and right hands, respectively, on a normal keyboard. Perhaps the most exotic feature of the interface is the ability to fork oneís presence, replicating subject instead of object by installing multiple sinks at arbitrary places around a virtual scene so that, for example, harmony and melody can be separately spatialized, using two heads to normalize the octave; such a technique effectively doubles the helix from the perspective of a single listener. Rather than a symmetric arrangement of the individual helices, they are perceptually superimposed in-phase, co-extensively, so that corresponding notes in different registers are at the same azimuth. KW - 3D audio KW - virtual reality KW - computer music KW - spatialization KW - spatial media KW - visual music KW - VSVR Y1 - 2002 VL - 31 IS - 3 SP - 269 EP - 281 ER - TY - JOUR A1 - Amano, Katsumi A1 - Matsushita, Fumio A1 - Yanagawa, Hirofumi A1 - Cohen, Michael A1 - Herder, Jens A1 - Martens, William A1 - Koba, Yoshiharu A1 - Tohyama, Mikio T1 - A Virtual Reality Sound System Using Room-Related Transfer Functions Delivered Through a Multispeaker Array: the PSFC at the University of Aizu Multimedia Center JF - TVRSJ N2 - The PSFC, or Pioneer Sound Field Controller, is a DSP-driven hemispherical loudspeaker array, installed at the University of Aizu Multimedia Center. The PSFC features realtime manipulation of the primary components of sound spatialization for each of two audio sources located in a virtual environment, including the content (apparent direction and distance) and context (room characteristics: reverberation level, room size and liveness). In an alternate mode, it can also direct the destination of the two separate input signals across 14 loudspeakers, manipulating the direction of the virtual sound sources with no control over apparent distance other than that afforded by source loudness (including no simulated environmental reflections or reverberation). The PSFC speaker dome is about 10 m in diameter, accommodating about fifty simultaneous users, including about twenty users comfortably standing or sitting near its ``sweet spot,'' the area in which the illusions of sound spatialization are most vivid. Collocated with a large screen rear-projection stereographic display, the PSFC is intended for advanced multimedia and virtual reality applications. KW - audio signal processing, audio telecommunications KW - auralization KW - calm technology KW - directional mixing console KW - multichannel sound reproduction KW - room-related transfer functions KW - roomware KW - sound localization KW - virtual conferencing environment Y1 - 1998 U6 - https://doi.org/10.18974/tvrsj.3.1_1 VL - 3 IS - 1 SP - 1 EP - 12 PB - J-STAGE ER - TY - JOUR A1 - Cohen, Michael A1 - Herder, Jens A1 - L. Martens, William T1 - Cyberspatial Audio Technology T1 - available in Japanese as well - Acoustical Society of Japan, Vol. 55, No. 10, pp. 730-731 JF - The Journal of the Acoustical Society of Japan (E) N2 - Cyberspatial audio applications are distinguished from the broad range of spatial audio applications in a number of important ways that help to focus this review. Most significant is that cyberspatial audio is most often designed to be responsive to user inputs. In contrast to non-interactive auditory displays, cyberspatial auditory displays typically allow active exploration of the virtual environment in which users find themselves. Thus, at least some portion of the audio presented in a cyberspatial environment must be selected, processed, or otherwise rendered with minimum delay relative to user input. Besides the technological demands associated with realtime delivery of spatialized sound, the type and quality of auditory experiences supported are also very different from those associated with displays that support stationary sound localization. Y1 - 1999 U6 - https://doi.org/10.1250/ast.20.389 N1 - available in Japanese as well - Acoustical Society of Japan, Vol. 55, No. 10, pp. 730-731 VL - 20 IS - 6 SP - 389 EP - 395 ER - TY - JOUR A1 - Ryskeldiev, Bektur A1 - Cohen, Michael A1 - Herder, Jens T1 - StreamSpace: Pervasive Mixed Reality Telepresence for Remote Collaboration on Mobile Devices JF - Journal of Information Processing N2 - We present a system that exploits mobile rotational tracking and photospherical imagery to allow users to share their environment with remotely connected peers “on the go.” We surveyed related interfaces and developed a unique groupware application that shares a mixed reality space with spatially-oriented live video feeds. Users can collaborate through realtime audio, video, and drawings in a virtual space. The developed system was tested in a preliminary user study, which confirmed an increase in spatial and situational awareness among viewers as well as reduction in cognitive workload. Believing that our system provides a novel style of collaboration in mixed reality environments, we discuss future applications and extensions of our prototype. KW - Ubiquitous Computing KW - Telepresence KW - Remote Collaboration KW - Mixed Reality KW - Live Video Streaming Y1 - 2018 U6 - https://doi.org/10.2197/ipsjjip.26.177 VL - 26 SP - 177 EP - 185 PB - J-STAGE ER -