Abstract
This case study presents two Mixed Reality simulation technologies which can be used for cultural heritage simulations. These technologies are Augmented Reality (AR) and Holographic Mixed Reality (HMR). Augmented Reality in our case study employs both MetaioSDK and the recent ARKit while Holographic Mixed Reality uses Microsoft Hololens. Specifically, we compare the immersive experience, interaction modality, field of view and level of presence that each of these technologies provide. Our results have been first demonstrated at the Asinou church, a UNESCO Cultural Heritage monument located in Cyprus and are currently being deployed at the Greek Ministry of Culture. The priest of Asinou church acts as our virtual curator which gives a virtual tour in the church and supports a range of different capabilities like performing gestures, speech and lip synchronization. Finally, in order for the user experience to be improved with higher realism and believability, a recent attempt is described that upgrades the AR tracking technology for mobile Augmented Reality from MetaioSDK to Apple’s ARKit.
Keywords
Holographic Reality, Augmented Reality, cultural heritage simulation, Microsoft Hololens, MetaioSDK, ARKit
In this work, we present and compare the usage of latest Augmented Reality Head Mounted displays and related devices. These include Microsoft Hololens [1] and mobile AR. Each one of these technologies provides different immersion experience, interaction system, field of view and different level of presence.
In this application we employ our reconstructed virtual curator to provide a tour to the end-user about historical information concerning the history and critical events that took place at the monument and particularly important information regarding the unique frescos of the church. Also, the release of ARKit by Apple made us want to try and improve the user experience and level of realism by upgrading the mobile AR technology that we used from MetaioSDK [2] to ARKit [3].
The first results were indeed very inspiring and are shown and discussed below.
Comparison of Holographic and AR technologies - State of the art
In this case study we compare the usage of latest Holographic and AR technologies which are Microsoft Hololens and mobile AR, respectively. Microsoft Hololens Head Mounted display was used for the Holographic technology and MetaioSDK was used for the mobile AR technology. The immersion experience, the interaction system, the field of view and the level of presence of each technology was compared. Apple’s ARKit was used later, in order to upgrade our mobile AR technology from MetaioSDK by using Unity.
Holographic Technology - Microsoft Hololens
In Microsoft Hololens the user interacts with the virtual curator through gestures and voice commands. Via specific words the user can select the type of transformation (translation, rotation, scale) that would like to apply to the virtual curator. After selecting the type of transformation via speech commands the user performs the manipulate gesture to apply the selected transformation to the holographic priest. Microsoft Hololens provides full body tracking and it is partially immersive since it combines virtual and real world. One drawback of Microsoft Hololens that affects the user experience is the narrow field of view.
Mobile AR - MetaioSDK
In mobile AR we have used the MetaioSDK for the camera tracking. We have used the toolbox application to crate the 3d map file which includes the features of the scene, then pass this file to our application bundle and the MetaioSDK is able to compute the view matrix in order to place our virtual curator on a stable position in the real environment. The field of view depends on the camera of the device and it is partially immersive since it combines virtual and real world. The interaction system in mobile AR is the Graphical User Interface (GUI). Through the buttons of the GUI the user is able to transform the virtual curator and start the virtual tour in the church. Figure 2 shows the reconstructed priest in the church from two different points of view in mobile AR.
Mobile AR - ARKit
With the release of Apple’s ARKit, we decided that it would be a very good idea to upgrade the already existing mobile AR software that we used (MetaioSDK) to the latest ARKit, so that we can offer even better and more realistic results to the user. Also, the easy use of ARKit and its availability to the user’s phones and tablets makes it accessible to even more people. We made the upgrade attempt with the use of Unity software. This upgrade is still in early stages but the results that we have are looking very promising and inspiring.
Conclusions
In this case study, we presented two different technologies that can be used for cultural heritage simulation, the Holographic technology (with Microsoft Hololens) and the mobile AR technology (with MetaioSDK and ARKit). Both technologies, each with their advantages and disadvantages succeed in simulating cultural heritage monuments (priest of the Asinou church and the Asinou church in this case) which captures the user attention and makes historical information more attractive to the users. This level of attraction and realism is expected to be risen by the use of ARKit which brings mobile AR even closer to the people.
References
Margarita Papaefthymiou, Steve Kateros, Stylianos Georgiou, Nikos Lydatakis, Paul Zikas, Vasileios Bachlitzanakis, and George Papagiannakis. Gamified AR/VR Character Rendering and Animation-Enabling Technologies, pages 333-357. Springer International Publishing, Cham, 2017.
Luca Vacchetti, Vincent Lepetit, Michal Ponder, George Papagiannakis, Pascal Fua, Daniel Thalmann, and Nadia Magnenat Thalmann. A stable Real-time AR Framework for Training and Planning in Industrial Environments, pages 129-145. Springer London, London, 2004.
George Papagiannakis, Sébastien Schertenleib, Brian O’Kennedy, Marlene Arevalo-Poizat, Nadia Magnenat-Thalmann, Andrew Stoddart, and Daniel Thalmann. Mixing virtual and real scenes in the site of ancient pompeii: Research articles. Comput. Animat. Virtual Worlds, 16(1):11-24, February 2005.
Links
Microsoft Hololens. (www.microsoft.com/en-us/hololens)
MetaioSDK. (www.metaio.com)
ARKit. (developer.apple.com/arkit/)
Author
Geronikolakis Efstratios, Papagiannakis George - FORTH - Thematic Area 4