Output of the working groups

How the work was carried out

We first use our network to connect with a list of experts that are relevant to the thematic area.

After getting in contact with experts, we have been through different steps to extract the needed information:

• First their precise status as expert in the field and their research focus
• The relationship with our thematic area in the context of the subsequent work package
• Few case studies related to cultural heritage where the expert had to bring his participation under the form of an active role and contribution.
• Summary of the aforementioned involvement of our expert and what we can take as a general contribution for CH research, that we can integrate into the defined categories (see above)

Then from all experts and our state of the art, we could finalize the five topics that brought us to the overview of tools and technologies that are required for 3D data handling and metadata management within our context of CH.

In order to enhance the role of TA6, a one-day event, as a physical meeting, was organized and 12 participants from international and national institutions, related to Digital Cultural Heritage, were present, discussing about the needs, the added values, the technologies and the assessment of Virtual Multimodal Museums. The event was organized at MIRALab, at University of Geneva,  in three sessions based on the Working Groups and consisted of keynote speeches and presentations devoted to the topics related to them. Experts were categorized based on their research interest and their up to date contribution to the field. Minutely, the first session (WG1: Digitization) was chaired by Prof Nadia Magnenat-Thalmann from Switzerland and hosted by Prof. Frank Boochs from Germany, Lisa Chen from Switzerland and Prof. Martha Vassiliadi from Greece. The second session (WG2: Metadata and Semantics) was chaired by Prof. Daniel Thalmann from Switzerland and hosted Prof. Gilles Falquet from Switzerland, Yacine Bensamour from Switzerland, and Dr Rui Filipe Antunes from Portugal. The last session (WG3: Cooperation/Standardization) was chaired by Prof. George Papagiannakis from Greece and hosted by Asst Prof. Manolis Wallace from Greece, Dr Dimitris Protopsaltou from Greece and ? Victor-Jan Vos from Netherlands. At the end of the three sessions, a discussion/conclusion session was led concerning possible future needs for the enhancement of Virtual Museums. The minutes of the event, as well as the presentations were uploaded to a space within the project’s working space on Basecamp.

This event allowed every participant to be aware of the state of the art in Cultural heritage. . Moreover, we all could analyze the best practices, standardization and available tools and technologies required in the field of Cultural Heritage.

The aim of the workshop was to extract information from experts through their experience as case studies in various European or national projects. This collection of information can be sorted out in defined categories:

1. Data provenance (Museums, society, association)
2. Format (computer format, translations, conversion)
3. Storage (databases, centralization, modularity, cloud-based)
4. Architecture of data (hierarchical, tabular)
5. Technologies (software programming languages, hardware, etc.)

After getting in contact with experts, we have been through different steps to extract the needed information:

• First their precise status as expert in the field and their research focus
• The relationship with our thematic area in the context of the subsequent work package
• Few case studies related to cultural heritage where the expert had to bring his/her participation under the form of an active role and contribution.
• Summary of the aforementioned involvement of our expert and what we can take as a general contribution for CH research, that we can integrate into the defined categories (see above)

Then from all experts and our state of the art, we could finalize the five topics that brought us to the overview of tools and technologies required by 3D data handling and metadata management within our context of CH.

Summary of the results and impact from each TA

WG1: Digitization

Expert 1: Professor Nadia Magnenat-Thalmann

Nadia Magnenat-Thalmann is the Founder and Director of the MIRALab, an interdisciplinary lab in Human Computer Animation, University of Geneva, Switzerland. She is also Director of the Institute for Media Innovation in NTU, Singapore. Her research domains are Virtual Humans, Social Robots, mixed realities and medical simulation.

Case study: Nadine, social companion demonstrated at ArtScience Museum in Singapore.

Prof. Nadia Thalmann explained that robots were imagined since the Antiquity, as for example in Greece and the idea of automaton is in the mind of the people since a long time. Later, Leonardo da Vinci produced several mechanical automatons as the robotic knight, which was capable of independent motion. Later on, as an example, during the 18^th century, Jaquet-Droz from Switzerland produced several marvelous automatons, one for example that was able to write various texts.

Today, we can model in 3D, robots or robot’s parts or scan them. 3D digitization allows a next step to 3D fabricate the models. For example, a hand of a robot can be scanned and 3D modelled, then 3D fabricated. The scanning and 3D fabrication will allow to produce cheaper robots in the future. Then of course, the behavioral aspect should be developed adding several specific software.

As a valuable example, we can indicate social robot “Nadine”, held in a special exhibition at ArtScience Museum in Singapore. More than 100’000 visitors have met her during the 6 months exhibition and the visitors’ reaction has proven that such kind of social robot could perfectly fit as effective personalized interactive guide in a museum.

Expert 2: Professor Frank Boochs

Frank Boochs is a Professor for applied informatics, at the University of applied sciences, in Mainz in Germany. His research topics are the use of images and point clouds for the generation of different types of knowledge concerning the objects captured.

Case study: Understanding of the digitization of CH objects

His presentation focused on various questions we have raised in advance. He tried to answer most of them. What is important when digitizing / documenting an object? First of all, we need an optimal digitization and documentation of its spatial shape and its surface structure. Moreover, documentation of its visual appearance, referring to real color and texture. And finally the physico-chemical composition.

What implies optimal digitization? An optimal digitization can be characterized by the instruments we are using, the position and the level of the resolution. Moreover, different specters are needed to define the level of detail. The selection of the right instrument, adapted to our needs, is of a great challenge.

But, finally how we can model a good approach? First of all, we need to analyze the process to understand each step. We need to evaluate and assess the object, the application, the environment, the technology and the data. Secondly, we need to identify the relevant characteristics. Last, ontology is taking place, where we need to structure the collected content.

We can conclude that digitization is indeed a complex process, which needs to be adapted to the needs of an application. Semantic technology though opens the door to transform such common understanding.

Expert 3: Lisa Chen

Lisa Chen holds a geodetic science and surveying degree and is currently the technical communication manager at Pix4D.

Case study: Digital photogrammetry for 3D cultural heritage archives.

Lisa Chen presented several cases studies. Examples of projects that have already worked with their new software are the rayCloud which constitutes an application to be used for the 3D visualization of photogrammetry (e.g it can visualize the different elements of the reconstruction and it can verify or improve the accuracy of the reconstruction of the model) and the digiArt which concerns the 3D reconstruction of CH. Moreover, places that have been reconstructed based on geo-informatics are a colonial Era church ruins, preserving American history, Tank canyon case (India) and the ancient city of Paquime (Mexico).

The outcome of this presentation showed the importance of defining a semantic model in order to have an accurate 3D model. The link between both is essential at the design phase.

WG2: 3D data handling and metadata

We reach multiple expert in this working group that could provide us the needed information, allowing to build up our state-of-the art based on the very broad view of the thematic area.

Expert 1: Professor Daniel Thalmann

Prof. Daniel Thalmann is Honorary Professor at EPFL and Director of Research & Development at MIRALab Sarl. He is a pioneer in research on Virtual Humans. Daniel Thalmann has been the Founder of The Virtual Reality Lab (VRlab) at EPFL, Switzerland.

Case study: Real time crowd simulation

Daniel worked for years in the domain of Virtual humans’ agent behavior modelling and crowd simulation. A virtual human agent corresponds to several components, among them the 3D graphics body representation, the attributes (psychological states), and sets of complex behavior (look at behavior, bakery behavior).

Beside the parameter of each agent as individual, we need to develop several methods: Region of interest (ROI) for motion planning; Emergent behavior (line formation); Avoiding behavior in the case of car going into it.

A case study was shown as the simulation of people in Pompeii having different behavior according to those living in poor or rich area.

The contribution of this talk was to demonstrate the needs of the semantic of the motion or the animation. Indeed, digitizing places in 3D is not enough, we need to bring life in them by adding social human agents that have adequate behaviors, with intentions and goals. This is part of the needs for intangible cultural heritage management.

Expert 2: Yacine Benmansour

Specialized in virtual reality and multimedia concrete applications, he is an experienced Research Project manager with a demonstrated history of working with industry in the higher education level. He is mainly dedicated to applied research in Geo data science: Geo-information, Geographic Information Systems (GIS), Building Information Modelling (BIM), BIM-GIS convergence, Geo data treatment process optimization, Project life cycle Management.

Case study: GEO-BIM

Yacine Benmansour demonstrated a commercial software GEO-BIM. GEO-BIM is a framework allowing to merge geo system information with building information modelling. Some examples were shown as the 3D model of Geneva with public transport, a traffic simulation in Geneva, Solar exposures, mobility graphs, and aerodynamic simulation of a place in Geneva, named Paquis, neighborhood printed in 3D.

BIM embed these following information: Spatial object information, descriptive information, physic properties, time information, project information, access information, Real Time information, dynamic 3D model, accurate geometry.

Different versions of BIM can be used, open or closed. To use this software with many various partners, a BIM system manager is needed.

To build a place or building, it is required a strong cooperation between the different actors: 3D architect, museum curator, gas technician, etc…

The drawback of this kind of platform is the compatibility between various software as several partners may have different software for specific application. The interest of this presentation is to show that we are going to use in the future this kind of platform in CH, not only to reconstruct building but to add all functional information.

Expert 3: Professor Gilles Falquet

Associate professor, University of Geneva, Faculty of economics and social sciences, department Hautes études commerciales. The general theme of his research activities is “accessing knowledge”, which includes: Semantic hyper books and semantic digital libraries, virtual documents, ontology management systems, point of views in ontologies, knowledge-based indexing and information retrieval, integrating knowledge visualization in 3D interfaces.

Case study: Manuscript from Ferdinand de Saussure (1857-1913)

In this case study, Gilles and his team were working on the forgotten manuscripts de F. de Saussure that were difficult to decipher. More than 40000 handwritten texts were left unpublished, waiting for scientist to understand them. The chaotic aspect of these texts leads to a primary need for classification, and dating issues. His team investigated a way to extract scientific content, historical and bibliographic context and terminology context. They used a knowledge graph (DBpedia) constituted by entity class and relationship in order to propose an implementation with the web semantic interface.

His role was to depict hand written notes using different methods. The links between CH objects was also presented with semantics dimensions. His examples showed the capacity to reassemble some various located handwritten documents and to interpret them.

Expert 4: Dr Rui Philippe Antunes

Academic, researcher and visual artist in new media. He was a Marie Sklodowska-Curie research fellow of MIRALab, at the University of Geneva and currently a senior researcher at the Faculty of Sciences from the University of Lisboa.

Case study: Simulation of populated Mertola and Sylves / Street of Xelb

His case studies concern inhabited ancient sites, which means we do not know anything about the population except what remains as a ruin. We only have the documentation accessible through cultural heritage repositories in libraries or Museums.

The aim is to allow curators in museums to simulate the ancient population and its behavior. To allow the curators to be able to do the work by themselves, the behavior for all human agents has been hard coded, then it needs to provide parameters for simulating the population in the crowd.

The Key features of the application are developed in Unity and are the reconstruction of the space (define, reconstruct), clothes interpretation from historians and videos of historical recreations, population generator (Point of interest, shape, etc.).

To conclude, Rui’s contribution corresponds to allow the definition of a populated scene to the to museum curator (and not to programmers). A specific interface was proposed for CH applications.

WG3: Cooperation

Expert 1: Professor Manolis Wallace

Manolis Wallace is currently an Assistant Professor at UOP’s Department of Informatics and Telecommunications and the director of the knowledge and Uncertainty Research Laboratory.

Case study: CrossCult EU project

Manolis Wallace works on the european project “CrossCult”, which empowers the reuse of digital heritage in context-aware crosscuts of European history, aiming to alter the way Europeans citizens appraise history. Crosscult’s role is not to publish new digital items but to use the already published ones in order to develop novel cultural experiences. These experiences will be created based on the exchange of visitors’ previous acquired experiences and by challenging the thinking process. Thus, diverse sources are combined in such a way so that the generation of the content that is delivered to the used should be seen as a cooperative interactive publication of digital content. For this content, already publically available content will be used, such as Europeana collections, the Tate collections, the National Gallery collections and so on. Using freely available content means there are no major security, privacy and IP issues to consider, but heterogeneity is still an important concern. In order to address this, most of the processing is done at a semantic level.

For the extraction and correlation of semantic information, they utilize proprietary software that is developed within the project. It is possible (but still depends on the business and exploitation plan that is still under development) that some of these software components will be made freely available later in the lifetime of the project.

Thus, the contribution to this WG consists of an explanation and description of how the elaboration of the potential of automatic contextualization will be done and how it can be linked with cooperative environment. The proposed idea concerns a new way for a more interactive and effective visit in the museums. This will be based not only on the common audio guidance but on a more customized path created by shared experience of people being in the museum. The idea is for the people to be connected and share their experience – contextualization

Expert 2: Dr Dimitris Protopsaltou

Dimitris Protopsaltou is CEO at the Stavros Niarchos Foundation Cultural Center. Co-founder of Future Library, a non-profit organisation with the aim to develop a network of public libraries.

Case study: Stavros Niarchos Foundation Cultural Center (SNFCC)

The SNFCC is the first private-public partnership of its type in Greece, and one of the most important civic (cultural/educational) projects ever undertaken in the country. The plan envisioned including the National Library of Greece, the Greek National Opera, and an educational and cultural park in one site, and thus bequeathing to the Greek people a great civic, cultural, educational and environmentally responsible landmark of international stature.

SNFCC organizes a lot of campaigns in order to build a big network with schools and local libraries across Greece. Many campaigns have been organized with libraries for promoting reading in children. There have been created, moreover, brain storming areas, customized libraries and musical studio with the possibility to rent the musical instruments.

Although SNFCC is almost new, it attracts over 2 million visitors per year. This raises more and more big needs. These needs are related to the kinds of technology that can be used in order to enhance the indoor experience of the user and to the cost so that the solutions are affordable from this large audience. In SNFCC, they have managed to gather a lot of visual, audio and other data but they cannot ensure their online accessibility. Thus, they proceed to collaborate with other institutes, like Europeana, in order to publish their collections (example of archive of the World War I, hosted by Europeana).

Expert 3: Victor – Van Jos

Case study: Europeana

Europeana is a digital platform which allows cross domain, cross topic and multilingual access to digital CH. There has been around 10 years since its activation and it is funded by European Union, serving five domains, such as CH institutions, European citizens, research, education and creative industries. Its work is divided in steps in order to make sure that the services offered are standardized and that CH is used. The main role of Europeana is the interconnection between countries and institutes to have the optimal way to find the objects they need by guarantying the access. They ensure the cooperation process through the interoperability (example: International Image Interoperability Framework (iiif) which is not located in Europeana and it scans with a high-resolution analysis manuscript, providing the possibility of a clear and high-speed zoom).

In the framework of cooperation process, they organize a lot of campaigns in order to have people engaged. As an example, we can mention the recent collaboration between Stavros Niarchos Foundation and Europeana concerning memorial for the World War I. Through these campaigns, they want to make sure that things are feasible and to ensure the quality of the content they want to provide.

Expert 4: Professor George Papagiannakis

George Papagiannakis is a computer scientist specialized in computer graphics and virtual-augmented reality. He is Associate Professor of Computer Graphics at the Computer Science department of the University of Crete, Greece and Affiliated Research Fellow at the Computer Vision and Robotics Laboratory in the Institute of Computer Science of the Foundation for Research and Technology Hellas, Heraklion, Greece.

Case study: “X” reality applications for cultural heritage

Georges Papagiannakis worked in several projects implying Virtual reality and Mixed reality. From the EU project LifePlus to the most recent ITN-DCH, his research was focused on geometric algebra applied to virtual reality. His work highlights the concept of “X” reality, that cover all domain of mixed, augmented, mobile and virtual reality. (From unmodeled world to fully modeled one). In the ITN project, they used a virtual priest and a curator into a real environment (Augmented mobile reality and holographic reality) and they provide on-site information to the public as well as illustration of intangible heritage (ceremony).

A specific mathematic algebra had to be developed in order to optimize the efficiency of XR applications. They also used gamified environment and virtual reality (Cardboard or HTC Vive) for education. The difficulty was to acquire the 3D data. Different interaction is possible with the XR application, as using hands or gamepad. The conclusion is how to improve the presence of user within CH environment and define methods to adapt storytelling.

Questions raised concerned ways with which we can determine the interpretation of a reconstruction or interpretation of some CH objects, considering the user’s feedback. Also, the economy of such technology, like cardboard for schools, or in the other way around has been remotely onsite to discover without moving.

At last, in the framework of contextualization, now we have the system which is able to bring together the afferent documents of information that have been artificially isolated, bringing them virtually back.

Expert 4: Professor Martha Vassiliadi

Martha Vassiliadi is currently Assistant Professor of Philology at the Aristotle University of Thessaloniki, Greece. Prior to this post, she was a Lecturer at the same department and before that a Lecturer of Modern Greek at the University of Geneva, Switzerland.

Case study: Archaeologies of the future

There are some refunding patterns when talking about CH: (destruction Pompeii and construction Athens). Comparing the MR reality experience with storytelling. Pompeii, they are trying to produce the storytelling using images and sounds, without humans. (sensory experience of time and space). These are heterotopias because they are the product of the digital culture. In Acropolis, there is the cult of representation. This is also a sensory experience (existential), feeling of “derealization” from Freud. An example shown is a museum film without words. We need to talk directly to the emotions. Material reality and fantasy animation. Needs to put some words to give the literature a challenge.

What we know now and what questions arise

Based on the up-to-date contribution of our experts here is what we already know.

WG1:

• Better techniques of 3D scanning are used, providing a cleaner and quicker 3D model that can be integrated to any kind of robot than can serve the digital heritage context.
• The need of semantics has been highlighted. The simulation of the shape is not enough. In order to make it more valuable there is the need of connection between data and representations and not only geometrical content but also semantic content.
• Semantics needs to be linked as well with motion capture. Emerging technologies like Mobile Laser have been reconstructing the 3D view in real time, enabling semantic acquisition (geo-informatics)

WG2:

• What is the use of semantic layer in the crowd simulation? It is not enough to model one person. Each person has to have the awareness of the external stimuli. We need to take into account the intention awareness of the surrounding.
• Concerning simulation of buildings, we are now in the position to have a simulation of all their aspects. The question is therefore about the compatibility of information structure across the different platform. This issue is addressed partially by the IFC standard.
• What is the importance of emotional semantics and analytics semantics?

WG3:

• There are institutes (Europeana for example) that are taking care of the build of standards in order for everyone to have access to any kind of information concerning Cultural Heritage. This constitutes a depository of what it exists up-to-date in Europe.
• Institutes like SNFCC provide multimodal interactive libraries that present any kind of information concerning Cultural Heritage and can enhance the learning of its aspects.
• In terms of contextualization we are now in the position to build on a new idea, connected to social networks, concerning a new way of museum guidance based on sharing experiences among people. Moreover, we have the technology to bring virtually back information that have been artificially isolated or forgotten.W

What should be carried forward to the EU roadmapping debate

TA6 working groups have produced over the past months a good fundament for further discussions on the discovery aspects of virtual cultural heritage. Especially in the debates among our experts interesting and highly relevant questions were addressed that are also worth to be carried forward to the EU roadmapping debate as they show that across Europe as well as across different academic fields there is a need to clarify certain aspects connected to the digitization of virtual and 3D entities, metadata collection and cooperation prospects.

The following ideas/questions arised in the framework of TA6 are considered worthwhile to be considered to the roadmapping debate:

1. How can we further improve the techniques of 3D scanning in order to have the optimal model that can be integrated to any kind of robot?
2. Semantics have to be taken into account. The importance of emotional and analytics semantics have been highlighted and it needs further examination. Data and representations have to be connected and not only geometrical content but also semantic content.
3. How can we ensure the awareness of external stimuli, as well as the intention awareness in a crowd simulation?
4. The importance for an efficient Virtual Museum is not just to describe the item itself but to digitize things and to find the optimal way to present them.
5. There is a need to find affordable solutions that can be applied to their big audience. The cost of a solution is important for cultural organization.
6. We need to support a focus on the learning process, evaluating what is the learning impact on a visitor as the aim is for him/her to learn something.
7. How can the technologies used so far enhance the inside experience without missing what is around the participant?

The most important thing we have to remember is that everything we make is about visitors’ experience and the change of their needs.