Extended reality “XR” is an umbrella term that encloses augmented reality (AR), virtual reality (VR), and mixed reality (MR) (Chua, 2018). XR has a long history, dating back to 1960, when Ivan Sutherland, a computer scientist, invented the first head-mounted display. Myron Krueger eventually invented the Videoplace in 1975, which did not require headsets or gloves. A simulated reality was created using a mixture of projectors and video cameras. VPL Research, a young technology company, was the first to bring this to the market in 1989. Technology has undergone a number of changes over time that have increased its usability. Many large companies are investing in XR technology, including Apple, Samsung, HTC, and Facebook (CHRP-INDIA, 2018).
Virtual Reality, Augmented Reality, Mixed Reality There is a controversy around the origin of the term VR, even so several authors associate it to Jaron Lanier (1985). VR refers to the ability to perform a task in a computer-generated environment. To create this illusion, the device would “deceive the brain” by supplying it content that is similar to what the brain will see in the natural world. VR typically works via a person using a head-mounted display with different controls that are connected to a computer. On the other hand, the goal of AR is to enrich the perception and knowledge of a real environment by adding digital information relating to this environment, using the technology as an ‘overlay’ to the real world. Images are captured by a camera and displayed on a screen with augmented data overlaid on the objects. This information is most often visual, sometimes auditory and is rarely haptic. Even though they share algorithms and technologies, VR and AR can be clearly distinguished from each other. The main difference is that in VR the tasks executed remain virtual, whereas in AR they are real. MR, also known as hybrid reality, is the most recent development in the XR technologies that combines the best parts of both AR and VR. Developments in this field have been based on position and orientation sensors as well as restitution devices. (Arnaldi, Guitton, Moreau, 2018).
Extended Reality in Music and Art
EX and Music
EX has become a popular medium for musicians in the creation and in the performance of their music. Countless different apps are already being developed to make creating music a more shareable and immersive experience. EX can allow users to interactively control and modulate electronic musical performance (Poupyrev, Berry, Billinghurst, Kurumisawa, 2001) and to present their concert hall and their orchestra in new ways (Thielen, Letellier, Sieck, Thoma, 2018). Furthermore, XR technologies add depth to the standard experience of listening to music.
When the Walkman was first released, it attracted a lot of attention to the notion that people might use music and headphones to reshape reality, according to Lych (2019). The “Walkman Effect'' is elevated to new heights and complexities with XR, which creates virtual sensory environments that simulate the way sound waves communicate with physical space. Audio Augmented Reality is an instance of AR, whereby experiences and actions in the real world are accompanied by additional layers of sound. Use of audio augmentation can be both entertaining and engaging. The use of audio layers can provide additional information and affordances to a user or audience. Furthermore, the use of augmented sound overcomes the cognitive and physical constraints encountered when using display devices (Lawton, Cunningham, Convery, 2020). For example, the Icelandic band Sigur Rós and the company Magic Leap have been working together for five years, exploring and expanding the frontiers of musical creation within MR. As a result, they have created an entirely new MR listening experience called “Tònandi” (Hogan, 2017). This new experience lets individuals move around a given room, where spatial sounds are visible all around, and ready to respond to every interaction (Sigur Rós, 2018). Tònandi enables viewers to explore new pieces of the band's composition in ways that no other AR software or virtual reality headset can (Hermann, 2018).
EX and Art EX has also become a popular medium for museums. Cultural heritage has been inspiring the AR community. During the last several years, numerous museums and cultural institutions have felt the need to modernise their image, attract new visitors, and above all offer stronger and more engaging experiences with cultural heritage, in order to "live" rather than just "learning" cultural heritage. Digital tools, especially XR technologies, provide a new way to interact with cultural heritage. The full surrounding environment can be transformed into an interaction surface (Damala, Cubaud, Bationo, Houlier, Marchal, 2008). There are three ways in which AR can be used as a tool for museums: screen-based, which is seen on a monitor, projector or mobile phone, and using head-mounted displays (HMDs) (Kolstee and Wim van Eck, 2011). The first goal of these experiences is to provide heritage with context, which can mean several things. It can be about restoring the context of fabrication of the object, of archeological discovery, the context of use etc. It can also mean the real-time display of additional information about the object, or the restitution of a former state. The first aim of these experiences is to provide an explanation for heritage, which can signify a variety of things. It may be about restoring the object's fabrication context, archaeological excavation context, or usage context, for example. It may also refer to the showing of new details about an object in real time, or the restoration of a previous state. Some locations or historical sites are difficult to reach. Physical barriers to entry may also be a problem. This may include situations such as archaeological site closures due to environmental concerns, disability, or isolated areas such as mountain tops (Minsar.app, 2021).
Immersion Drawing on the engaging features of VR, AR and MR, is possible to refer to these three concepts collectively as “immersive media” (Kaplan-Rakowski and Meseberg, 2018). Immersion is a vital feature of music and art that EX technologies can enhance. It is described as a state of intense mental engagement in which the subject can become disassociated from physical awareness as a result of a change in their attentional state (Agreval et al., 2019). It is commonly used among artists to reach that special state called "flow". This is the feeling outside of time that artists use to dive into the creative process (Amores and Lanier, 2017). VR allows fans to experience their favourite artists and music videos in a completely new and immersive way. Music lovers who cannot physically be inside the venue can still experience watching their favourite artist. AR has the potential to provide an interesting alternative for navigation, interaction and orientation in the museum setting. Despite the advantages described above, XR could minimise social interaction. EX delivers a range of viewing choices that can totally engulf human brains and remove the need for touch (Shaptunova, 2018). Since EX science is only in its early stages, there are not many reports on contraindications. It will take more time and analysis to prove this, but AR, will undoubtedly change how people see music, and navigate museums in the future (Mbryonic, 2018).
For my upcoming blog, I will explore two fascinating instances of extended reality creation in the realms of music and art, specifically focusing on Björk's album "Vulnicura" and the Modigliani exhibition.