Periodic Reporting for period 2 - iMARECULTURE (Advanced VR, iMmersive serious games and Augmented REality as tools to raise awareness and access to European underwater CULTURal heritagE.)
Période du rapport: 2017-11-01 au 2020-01-31
Project’s scope is to raise public’s awareness on European identity by focusing in maritime CH, which by default bridges different civilizations. Scope of the project is to design, analyse, develop and validate pioneer applications and systems in the context of Virtual Museums through collaborative and innovative research from a diverse group of scientists, researchers, archaeologists, experts and museums.
Underwater CH assets are widely spread into the Mediterranean. However, unlike land archaeological sites, submerged settlements, submerged ancient ports and especially shipwrecks, are not accessible to the general public nor even to all experts, due to their environment and depth. Photos and surfaced finds exhibited in maritime museums provide fragmented aspects of such sites but are the only one’s visitors can see. Thus, Virtual Museums provide a unique opportunity for digital accessibility to both scholars and general public, interested in having a better grasp of underwater sites and maritime archaeology. Such digital encounters should provide virtual visit opportunities for children, elderly or people with mobility problems, and enhance the actual underwater visit with augmented digital content.
Despite the recent developments in Augmented Reality devices, tools for CH, are limited to dry conditions. The main reason is the demanding environment in terms of 3D data acquisition and poor imaging conditions. Therefore, AR tools, which could enhance the digital experience of visitors in situ, do not work underwater. The Project will address this using underwater tablets, provided to visitors.
The main tools provided by the Project are (a) VR dry visits in underwater CH, in a variety of devices (smartphones, Daydream and HTC Vive HMDs, Holographic screens), (b) an AR tablet with housing for divers/visitors in underwater CH sites, to be used both as guides as well as for AR superimposition of conceptual 3D drawings of the visited area, (c) a Seafaring serious game, (d) a VR Excavation serious game and (e) physical and VR 3D puzzles of ships and amphorae fragments. All of these are available to public except for (b), which requires dedicated acoustic positioning modems and corresponding antennas. Apart from these main deliverables, several other by products were delivered such as software for underwater image dehazing and colour correction, web based classical ship route calculation according to dynamic weather data, VR training for underwater documentation and excavation, ontology based web tools for Xlendi shipwreck and an ontology editor.
During project's duration, apart from the technical aspect, two challenges were faced: IPR and finding the equilibrium among education vs fun aspect in the applications. Before any commercialization of results and/or applications, the developer must go through a meticulous process of identifying rights and applying permission of usage. This extents from the local Ministry and excavation Director to the photographers and 3D reconstruction experts. Usually this is the main constrain in most commercialization efforts in CH through Mixed Reality applications.
Underwater CH assets are widely spread into the Mediterranean. However, unlike land archaeological sites, submerged settlements, submerged ancient ports and especially shipwrecks, are not accessible to the general public nor even to all experts, due to their environment and depth. Photos and surfaced finds exhibited in maritime museums provide fragmented aspects of such sites but are the only one’s visitors can see. Thus, Virtual Museums provide a unique opportunity for digital accessibility to both scholars and general public, interested in having a better grasp of underwater sites and maritime archaeology. Such digital encounters should provide virtual visit opportunities for children, elderly or people with mobility problems, and enhance the actual underwater visit with augmented digital content.
Despite the recent developments in Augmented Reality devices, tools for CH, are limited to dry conditions. The main reason is the demanding environment in terms of 3D data acquisition and poor imaging conditions. Therefore, AR tools, which could enhance the digital experience of visitors in situ, do not work underwater. The Project will address this using underwater tablets, provided to visitors.
The main tools provided by the Project are (a) VR dry visits in underwater CH, in a variety of devices (smartphones, Daydream and HTC Vive HMDs, Holographic screens), (b) an AR tablet with housing for divers/visitors in underwater CH sites, to be used both as guides as well as for AR superimposition of conceptual 3D drawings of the visited area, (c) a Seafaring serious game, (d) a VR Excavation serious game and (e) physical and VR 3D puzzles of ships and amphorae fragments. All of these are available to public except for (b), which requires dedicated acoustic positioning modems and corresponding antennas. Apart from these main deliverables, several other by products were delivered such as software for underwater image dehazing and colour correction, web based classical ship route calculation according to dynamic weather data, VR training for underwater documentation and excavation, ontology based web tools for Xlendi shipwreck and an ontology editor.
During project's duration, apart from the technical aspect, two challenges were faced: IPR and finding the equilibrium among education vs fun aspect in the applications. Before any commercialization of results and/or applications, the developer must go through a meticulous process of identifying rights and applying permission of usage. This extents from the local Ministry and excavation Director to the photographers and 3D reconstruction experts. Usually this is the main constrain in most commercialization efforts in CH through Mixed Reality applications.
During the first period, the work performed was focusing on data retrieval and gathering. Data gathered varied from historic and archaeological resources, to geodata for the seafaring game and 3D models of classical ships. Amphorae 3D models were also gathered, and a database of amphorae types was formed. The ontologies of the two 3D libraries, of ships and amphorae, was also completed. In the same time, significant work was performed towards serious games development. A pilot hyper storytelling application has been also produced to act as a base for further improvements and additions. Following the evaluation, storytellers drafted narratives and cut scenes for video production and animations.
Reaching the second period of the project, the Seafarers serious game undergone two major revisions based on internal evaluations. The VR Excavation serious game has completed the digging module and the collaborative (telepresence) training module. A working version of the VR 3D puzzles has been developed. Also, detailed 3D models of the three test sites have been prepared for the dry visits. Texture has been processed to be compatible with VR devices. The underwater AR tablet was focusing on precise localization, using hybrid acoustic and imaging techniques. Several tests have taken place, in a variety of conditions. On these tests, real time colour correction has been evaluated along with dehazing algorithms. Image enhancement tests also involved improved QR coded target recognition. AR tools have been evaluated for positioning in the underwater environment.
The final period of the project was devoted in applications’ finalizing, evaluation and communication activities. This last period was also devoted in elaborating business and exploitation plan. Several internal evaluations helped improving UX and initial concept. Storytelling was completed and integrated into the Dry Visit applications and serious games. Most of the releases of the project undergone evaluation by expert and non-expert.
Publicity of the project increased with demonstrations in many events around Europe. The exhibition of the project in the Thalassa Museum, sealed this final phase by offering to the visitors the promised enhanced experience. During this six-week exhibition with Holografika screens and HTC Vive HMDs, public evaluated through questionnaires the UX, providing feedback. Results of the project are in time with the opening of many underwater archaeological parks to the visitors in the Mediterranean Sea. The research carried out during the project, attracted a lot of attention towards that direction, offering possibilities for immediate exploitation of the results.
Reaching the second period of the project, the Seafarers serious game undergone two major revisions based on internal evaluations. The VR Excavation serious game has completed the digging module and the collaborative (telepresence) training module. A working version of the VR 3D puzzles has been developed. Also, detailed 3D models of the three test sites have been prepared for the dry visits. Texture has been processed to be compatible with VR devices. The underwater AR tablet was focusing on precise localization, using hybrid acoustic and imaging techniques. Several tests have taken place, in a variety of conditions. On these tests, real time colour correction has been evaluated along with dehazing algorithms. Image enhancement tests also involved improved QR coded target recognition. AR tools have been evaluated for positioning in the underwater environment.
The final period of the project was devoted in applications’ finalizing, evaluation and communication activities. This last period was also devoted in elaborating business and exploitation plan. Several internal evaluations helped improving UX and initial concept. Storytelling was completed and integrated into the Dry Visit applications and serious games. Most of the releases of the project undergone evaluation by expert and non-expert.
Publicity of the project increased with demonstrations in many events around Europe. The exhibition of the project in the Thalassa Museum, sealed this final phase by offering to the visitors the promised enhanced experience. During this six-week exhibition with Holografika screens and HTC Vive HMDs, public evaluated through questionnaires the UX, providing feedback. Results of the project are in time with the opening of many underwater archaeological parks to the visitors in the Mediterranean Sea. The research carried out during the project, attracted a lot of attention towards that direction, offering possibilities for immediate exploitation of the results.
Project’s scientific contribution so far, includes:
- Automatic amphorae detection using CNN
- Ontology based web tools for 2D/3D/4D data extraction for the Xlendi shipwreck dataset
- Advanced hyper-storytelling approach with 360 degrees videos with actors
- Improvements on underwater virtual environment.
- New dehazing technique for improved underwater target recognition
- Classical ship route calculator web-based platform
- Advanced combined acoustic and visual underwater positioning for the AR tablet.
Project’s impact is also realized through the extensive dissemination and publication list on conferences and respectable journals. Results, as derived from extensive evaluations, show that VR dry visits are a main tool for attracting museum visitors, either by free apps in smartphones, or by dedicated immersive VR hardware in museums. Most of the young visitors are attracted to Mixed Reality technologies and willing to devote more time going through all the info points to learn as much as possible for the test sites. Storytelling is a crucial module of the dry visits. Data acquisition, storyboards, filming and final implementation play a crucial role in the overall experience. In a similar way, experienced divers were fascinated by the possibility to use tablets in underwater housing to guide them in their dive and provide AR superimposition of the villa in its original glory. The possibility of using such devices underwater significantly upgraded their experience during underwater visits. The serious games were appreciated for their realism and potential of being used as training tools.
- Automatic amphorae detection using CNN
- Ontology based web tools for 2D/3D/4D data extraction for the Xlendi shipwreck dataset
- Advanced hyper-storytelling approach with 360 degrees videos with actors
- Improvements on underwater virtual environment.
- New dehazing technique for improved underwater target recognition
- Classical ship route calculator web-based platform
- Advanced combined acoustic and visual underwater positioning for the AR tablet.
Project’s impact is also realized through the extensive dissemination and publication list on conferences and respectable journals. Results, as derived from extensive evaluations, show that VR dry visits are a main tool for attracting museum visitors, either by free apps in smartphones, or by dedicated immersive VR hardware in museums. Most of the young visitors are attracted to Mixed Reality technologies and willing to devote more time going through all the info points to learn as much as possible for the test sites. Storytelling is a crucial module of the dry visits. Data acquisition, storyboards, filming and final implementation play a crucial role in the overall experience. In a similar way, experienced divers were fascinated by the possibility to use tablets in underwater housing to guide them in their dive and provide AR superimposition of the villa in its original glory. The possibility of using such devices underwater significantly upgraded their experience during underwater visits. The serious games were appreciated for their realism and potential of being used as training tools.