This case study presents a comparison of latest software and hardware for rapid reconstruction of real humans using as an input RGB or RGB-D images, and based on this comparison is introducing the pipeline that produces high realistic reconstructions in a reasonable amount of time, suitable for real-time Virtual Reality (VR) simulations in virtual museums.
Rapid reconstruction, Virtual Reality (VR), Agisoft Photoscan software, Fast Avatar Capture application, Occipital Structure Sensor
Virtual characters play a fundamental role for attaining high level of believability in Mixed-reality environments and they are the key-element for transferring knowledge and presenting scenarios in different Cultural Heritage applications. In this work, we populate our Mixed-reality applications with Virtual curators which play a vital role in the presentation of Cultural Heritage by giving instructions to the users and transfer knowledge about the history of the Cultural Heritage monument. Such Virtual curators are reconstructed out of real humans and have the ability for verbal as well as nonverbal communication skills. In this work, we compare latest 3D reconstruction methodologies of realistic Virtual characters by capturing real human geometry from photographs.
Rapid reconstruction - State of the art
The 3D rapid reconstruction process was done with the help of different software and hardware methods for reconstructing virtual characters suitable for Mixed-reality environments. These methods are Agisoft Photoscan software , Occipital Structure Sensor  and Fast Avatar Capture  that utilises the capabilities of Kinect v1 sensor . Firstly, we tested all the methods on a subject and then based on our findings we reconstructed the virtual priest of Asinou church.
Agisoft Photoscan software
Agisoft Photoscan is a software that uses photogrammetry techniques and computer vision methods for reconstructing 3D objects given as input a set of photographs.
Click here to learn more
Fast Avatar Capture application
The second software that we have tested was Fast Avatar Capture desktop application . Fast Avatar Capture is not suitable for face reconstruction because it produces low resolution texture. This occurs because Fast Avatar Capture utilises Kinect v1 for the reconstruction and captures images with resolution 640x480. The software captures the subject from 4 different angles and each capture last approximately 15 seconds. For each capture, the subject rotates 90° around himself and hist posture must be as stable as possible to achieve higher quality 3D model. Figure 3 shows the system used for avatar creation. The Kinect was placed one meter distance from the subject that stood on the base that we constructed for rotating the human during the capture process in order to be as stable as possible. After the scanning process, the software requires approximately 2 minutes for reconstructing the character.
Occipital Structure Sensor
The third reconstruction method that we have tested is Occipital Structure Sensor, illustrated on Figure 4. We achieved the reconstruction via the “Scanner” application running on iPad Air 2 which requires the Occipital Structure Sensor to scan the objects. Structure Sensor provides rapid reconstruction of the object and the “Scanner” application produces and illustrates the 3D mesh while scanning, as a result the user can stop the scanning only when he achieves high quality 3D mesh. On Figure 5 the 3D mesh captured with Occipital Structure Sensor is shown.
Comparison of reconstruction methods
Fast Avatar Capture and Occipital Structure Sensor are suitable for reconstructing the body and Agisoft Photoscan software is suitable for reconstructing the face because it produces detailed geometry and high resolution texture without noise.
Click here to learn more
Virtual priest reconstruction
Based on our results, we decided to use the Occipital Structure Sensor to reconstruct the priest of Asinou church. The 3D mesh that was produced needed further improvement in order to have the desirable result. We have improved the body and the face of the virtual curator using 3ds Max modelling software. To avoid a lot of manual work, one alternative option is to reconstruct the face of the priest using Agisoft Photoscan software.
Virtual Tour in Asinou Church in Virtual Reality (VR)
Oculus Rift provides fully immersive virtual environment since the user does not have any access in the real world. Oculus rift provides rotational and positional tracking but the movements are restricted to a more limited space in front of the sensors. The user interacts with the virtual environment using the touch controllers. Through Oculus Rift the user is able to navigate in the exterior and interior part of the asinou church as shown in Figure 8 below.
In this case study, we have presented and compared different hardware and software methods for reconstructing virtual characters out of real humans, suitable for realtime Virtual Reality (VR) and proposed the best method for reconstructing virtual characters. We have demonstrated our results at Asinou church, by creating a virtual tour in the church in Mixed-Reality environments. The tour in the church is provided by the reconstructed virtual priest. By employing our reconstructed interactive virtual priest in our Mixed-Reality environments we have achieved a very interesting virtual tour that attracts the user attention and further provokes the interest of the users for learning historical information about the church.
1. Mixamo. (www.mixamo.com)
2. Agisoft Photoscan. (www.agisoft.com)
3. Fast Avatar Capture. (smartbody.ict.usc.edu/fast-avatar-capture-software-download)
4. Occipital Structure Sensor. (structure.io)
5. Microsoft Kinect. (developer.microsoft.com/el-gr/windows/kinect)
6. Papaefthymiou, M., Kanakis, M., Geronikolakis, E., Nohos, A., Papagiannakis, G., “Rapid reconstruction and simulation of virtual characters in Mixed Reality environments”, ITN-DCH final conference, Olimje, Slovenia, May 2017
7. Stavros Kateros, Stylianos Georgiou, Margarita Papaefthymiou, George Papagiannakis, and Michalis Tsioumas. A comparison of gamified, immersive vr curation methods for enhanced presence and human-computer interaction in digital humanities. International Journal of Heritage in the Digital Era, 4(2):221-233, 2015. also presented in “The 1st International Workshop on ICT for the Preservation and Transmission of Intangible Cultural Heritage”, EUROMED2014