Abstract

This case study presents a comparison of latest software and hardware for rapid reconstruction of real humans using as an input RGB or RGB-D images, and based on this comparison is introducing the pipeline that produces high realistic reconstructions in a reasonable amount of time, suitable for real-time Virtual Reality (VR) simulations in virtual museums.

Keywords

Rapid reconstruction, Virtual Reality (VR), Agisoft Photoscan software, Fast Avatar Capture application, Occipital Structure Sensor

Virtual characters play a fundamental role for attaining high level of believability in Mixed-reality environments and they are the key-element for transferring knowledge and presenting scenarios in different Cultural Heritage applications. In this work, we populate our Mixed-reality applications with Virtual curators which play a vital role in the presentation of Cultural Heritage by giving instructions to the users and transfer knowledge about the history of the Cultural Heritage monument. Such Virtual curators are reconstructed out of real humans and have the ability for verbal as well as nonverbal communication skills. In this work, we compare latest 3D reconstruction methodologies of realistic Virtual characters by capturing real human geometry from photographs.

Rapid reconstruction - State of the art

The 3D rapid reconstruction process was done with the help of different software and hardware methods for reconstructing virtual characters suitable for Mixed-reality environments. These methods are Agisoft Photoscan software [2], Occipital Structure Sensor [4] and Fast Avatar Capture [3] that utilises the capabilities of Kinect v1 sensor [5]. Firstly, we tested all the methods on a subject and then based on our findings we reconstructed the virtual priest of Asinou church.

Agisoft Photoscan software

Agisoft Photoscan is a software that uses photogrammetry techniques and computer vision methods for reconstructing 3D objects given as input a set of photographs.

Click here to learn more

Photoscan is the slowest method for reconstructing a mesh compared to the Occipital Structure Sensor and Fast Avatar Capture, so we decided to use it for reconstructing only the face. Moreover, we selected this method because the face is the part of the human that we would like to give more focus and maintain the details. We gave to the software as an input a set of 120 photographs. We took photographs around the face from 3 different heights in order to cover the whole surface of the face. On the first one we were focusing on the lower part of the face, on the second one on the middle part and the third one on the upper part. Figure 1 shows the position and the rotation of the photographs captured around the face, that were given as an input to Agisoft Photoscan software. Firstly, we align the photographs and then build the dense cloud (Figure 2 (left)). We build the cloud with medium resolution because if large number of points are generated, the shape of the 3D model is not so smooth. Then, we had to manually clear the cloud using the delete tool of Photoscan in order to remove the point data of other objects in the environment, and leave only the points of the face, to achieve faster 3D mesh reconstruction. Then we build the 3D mesh and the texture of the avatar (Figure 2 (right)).

Figure 1: The position and the rotation of the photographs captured around the face that were given as an input to Agisoft Photoscan software.

Figure 2: The dense cloud data (left) and the 3Dmesh (right) produced from the photographs taken around the face, using the Agisoft Photoscan software.

Figure 2: Final 3D mesh and the texture of the avatar (Figure 2 (right)).

 

 

 

 

 

 

 

 

 

Fast Avatar Capture application

The second software that we have tested was Fast Avatar Capture desktop application [3]. Fast Avatar Capture is not suitable for face reconstruction because it produces low resolution texture. This occurs because Fast Avatar Capture utilises Kinect v1 for the reconstruction and captures images with resolution 640x480. The software captures the subject from 4 different angles and each capture last approximately 15 seconds. For each capture, the subject rotates 90° around himself and hist posture must be as stable as possible to achieve higher quality 3D model. Figure 3 shows the system used for avatar creation. The Kinect was placed one meter distance from the subject that stood on the base that we constructed for rotating the human during the capture process in order to be as stable as possible. After the scanning process, the software requires approximately 2 minutes for reconstructing the character.

Figure 3: The system used for reconstructing the subject with Fast Avatar Capture software

Figure 3: The character produced by Fast Avatar Capture software

 

 

 

 

 

 

 

 

 

Occipital Structure Sensor

The third reconstruction method that we have tested is Occipital Structure Sensor, illustrated on Figure 4. We achieved the reconstruction via the “Scanner” application running on iPad Air 2 which requires the Occipital Structure Sensor to scan the objects. Structure Sensor provides rapid reconstruction of the object and the “Scanner” application produces and illustrates the 3D mesh while scanning, as a result the user can stop the scanning only when he achieves high quality 3D mesh. On Figure 5 the 3D mesh captured with Occipital Structure Sensor is shown.

Figure 4: The Occipital structure sensor used to reconstruct the virtual priest of Asinou church.

Figure 5: Our subject (left) and the priest of Asinou church (right) reconstructed by the Occipital Structure Sensor.

 

 

 

 

 

 

 

 

 

Comparison of reconstruction methods

Fast Avatar Capture and Occipital Structure Sensor are suitable for reconstructing the body and Agisoft Photoscan software is suitable for reconstructing the face because it produces detailed geometry and high resolution texture without noise.

Click here to learn more

We have decided to use Occipital Structure Sensor to reconstruct the priest of Asinou church, because it is easier compared to Fast Avatar Capture. While capturing with Fast Avatar Capture the subject must rotate, as a result the posture is quite difficult to be exactly the same to all captures, thus the reconstruction in some cases is not possible. On the other hand, with the Occipital Structure Sensor the subject is stable, so it is more potential to produce higher quality reconstruction. Moreover, Fast Avatar Capture captures images from only 4 different points of view from a certain distance in contrast to Structure Sensor that the user can capture data from any point of view and from any distance around the subject. This allows producing high quality textures and more accurate geometry.

Virtual priest reconstruction

Based on our results, we decided to use the Occipital Structure Sensor to reconstruct the priest of Asinou church. The 3D mesh that was produced needed further improvement in order to have the desirable result. We have improved the body and the face of the virtual curator using 3ds Max modelling software. To avoid a lot of manual work, one alternative option is to reconstruct the face of the priest using Agisoft Photoscan software.

Figure 6: Digitisation of the priest of Asinou church using Structure Sensor.

 

Figure 7: Some of the Blend shapes created on the face of the virtual character that are used for lip synchronization.

Virtual Tour in Asinou Church in Virtual Reality (VR)

Oculus Rift provides fully immersive virtual environment since the user does not have any access in the real world. Oculus rift provides rotational and positional tracking but the movements are restricted to a more limited space in front of the sensors. The user interacts with the virtual environment using the touch controllers. Through Oculus Rift the user is able to navigate in the exterior and interior part of the asinou church as shown in Figure 8 below.

Figure 8: The virtual tour in Asinou church running on Oculus Rift.

Conclusions

In this case study, we have presented and compared different hardware and software methods for reconstructing virtual characters out of real humans, suitable for realtime Virtual Reality (VR) and proposed the best method for reconstructing virtual characters. We have demonstrated our results at Asinou church, by creating a virtual tour in the church in Mixed-Reality environments. The tour in the church is provided by the reconstructed virtual priest. By employing our reconstructed interactive virtual priest in our Mixed-Reality environments we have achieved a very interesting virtual tour that attracts the user attention and further provokes the interest of the users for learning historical information about the church.

References

1. Mixamo. (www.mixamo.com)

2. Agisoft Photoscan. (www.agisoft.com)

3. Fast Avatar Capture. (smartbody.ict.usc.edu/fast-avatar-capture-software-download)

4. Occipital Structure Sensor. (structure.io)

5. Microsoft Kinect. (developer.microsoft.com/el-gr/windows/kinect)

6. Papaefthymiou, M., Kanakis, M., Geronikolakis, E., Nohos, A., Papagiannakis, G., “Rapid reconstruction and simulation of virtual characters in Mixed Reality environments”, ITN-DCH final conference, Olimje, Slovenia, May 2017

7. Stavros Kateros, Stylianos Georgiou, Margarita Papaefthymiou, George Papagiannakis, and Michalis Tsioumas. A comparison of gamified, immersive vr curation methods for enhanced presence and human-computer interaction in digital humanities. International Journal of Heritage in the Digital Era, 4(2):221-233, 2015. also presented in “The 1st International Workshop on ICT for the Preservation and Transmission of Intangible Cultural Heritage”, EUROMED2014

Download PDF