Final Report Summary - 4D-CH-WORLD (Four Dimensional Cultural Heritage World)
4D-CH-Word is a Marie Curie project funded by European Union with the main purpose of designing, analyzing, researching and developing computer vision and machine learning algorithms for a cost-effective creation of 4D digital maps in cultural heritage. 4D digital modelling implies the creation of precise time-varying 3D reconstructions of cultural heritage objects to capture temporal geometric variations/distortions, i.e. a spatio-temporal assessment.
The key research challenge for 4D modelling, was the data collection over heterogeneous unstructured web resources. Such “in the wild” data include outliers and significant noise, since they have not been created for 3D modelling and reconstruction purposes. In addition, GPS and geo-information is limited or non-existent. However, such data allow for a massive reconstruction of the content even for monuments that have been destroyed due to natural phenomena or humans’ interventions.
The project exploits digital information from both unstructured data, available over distributed multimedia repositories and social networks, and from in-situ data acquisition procedures. The 4D digital modelling methods were testing in Calw, a town in the middle of Baden-Württemberg, located 40 km southwest of Stuttgart. Other demonstration scenarios refer to the digital reconstruction of the monument of Porta Nigra in Germany and Padrão dos Descobrimentos in Portugal.
The first step involved the content-based retrieval of visual information from distributed multimedia databases, like Picasa, Flickr and Twitter. Machine learning methods, exploiting unsupervised clustering algorithms, were used to isolate data outliers and group together similar data with respect to their visual properties. Additionally, textual analysis and processing take place to assign semantic metadata (tags) to the visual collected data.
As far as the in-situ data collection process in concerned, terrestrial laser scanning, close-range photographs and aerial imagery have been used. All these information are appropriately fused, using computer vision algorithms, machine learning methods and photogrammetry tools for a cost-effective, automated and precise time varying 3D modelling.
4D-CH-World supported various capturing procedures, including generation of dense 3D point clouds and terrestrial and aerial overlapping images. In the context of data analysis, novel computer vision tools have been adopted for fusing the heterogeneous collected data to improve the precision and accuracy of modelling towards an automated framework. Finally, synthetic data and computer graphics methods are exploited to fill out missing information from the data acquisition process.
To implement the time dimension of the modelling, data from different time instances have been collected. It is clear that for content referring to past periods, only RGB image data are available. These data are collected through developed search engine, retrieving images over distributed databases and the social media. More specifically, for the Calw case our data go back to 1850s and 1860s, when the first glass-plate photographs could be collected.
In order to understand and classify spatio-temporal regions in historic urban cities, deep learning strategies are employed. The term deep learning refer to a variety of techniques that can understand complicated patterns by automatically extracting the appropriate features for the task. In the context of 4D-CH-Word, we have focused on convolutional neural networks (CNNs) for two different scenarios; a) timber house identification and b) urban development monitoring. The deep learning approaches supported the 4D reconstruction; i.e. understanding the time period of the reconstructed areas by either, separating new from old buildings, or newly developed areas. In the end, deep learning and advanced filtering approaches allowed the creation of image clusters according to the time period.
The 4D-CH-World outcomes are presented via 4D visualization toolkits. The visualization methods have been developed in a Web-based interface. A user is able to freely navigate within the reconstructed environment and interact with it. Texturized detailed models depict accurate the current place, providing further information when requested by the user. Furthermore, for specific areas, user can experience the evolution through the years; see the current monument, how it was the previous decades and view the raw images that supported the 3D reconstruction. Finally, by selecting a year as a reference (e.g. 2000), we can see (if available) the difference in a specific area just by activating the time filter. That way, the 4D-CH-word team, successfully combined 3D models of a scene associated to different time instances, developing a system for 4D reconstruction and visualization of the results.
The final research achievement of 4D-CH-Word deal with recommendation system able to filter content according to both user preferences and visual attributes. The recommendation system empowers the user to interact with the platform and selects CH objects of interest displayed to him/her through suitable profiles. Users’ profiles are created by measuring the feedback of the users to the system through the activation of recursive machine learning algorithms that on-line update the system response to fit user’s preferences.
The produced software was designed for tablets and phones. It is also possible to run the platform in a standard desktop PC. However, touch screen is strongly suggested. The user is able to navigate freely to the 3D world either as pedestrian (road view) either from above. At specific areas, indicated on the map, we are able to see the actual image that led to the creation of the 3D object.
The tools developed are currently exploited for the digitalization of intangible cultural heritage such as dances through a collaboration with the TERPSICHORE and iTreasures EU funded projects. The innovative tools are also exploited under the framework of the international project being currently in progress regarding the restoration of the Holy Aedicule of the Holy Sepulcher Church in Jerusalem.
The key research challenge for 4D modelling, was the data collection over heterogeneous unstructured web resources. Such “in the wild” data include outliers and significant noise, since they have not been created for 3D modelling and reconstruction purposes. In addition, GPS and geo-information is limited or non-existent. However, such data allow for a massive reconstruction of the content even for monuments that have been destroyed due to natural phenomena or humans’ interventions.
The project exploits digital information from both unstructured data, available over distributed multimedia repositories and social networks, and from in-situ data acquisition procedures. The 4D digital modelling methods were testing in Calw, a town in the middle of Baden-Württemberg, located 40 km southwest of Stuttgart. Other demonstration scenarios refer to the digital reconstruction of the monument of Porta Nigra in Germany and Padrão dos Descobrimentos in Portugal.
The first step involved the content-based retrieval of visual information from distributed multimedia databases, like Picasa, Flickr and Twitter. Machine learning methods, exploiting unsupervised clustering algorithms, were used to isolate data outliers and group together similar data with respect to their visual properties. Additionally, textual analysis and processing take place to assign semantic metadata (tags) to the visual collected data.
As far as the in-situ data collection process in concerned, terrestrial laser scanning, close-range photographs and aerial imagery have been used. All these information are appropriately fused, using computer vision algorithms, machine learning methods and photogrammetry tools for a cost-effective, automated and precise time varying 3D modelling.
4D-CH-World supported various capturing procedures, including generation of dense 3D point clouds and terrestrial and aerial overlapping images. In the context of data analysis, novel computer vision tools have been adopted for fusing the heterogeneous collected data to improve the precision and accuracy of modelling towards an automated framework. Finally, synthetic data and computer graphics methods are exploited to fill out missing information from the data acquisition process.
To implement the time dimension of the modelling, data from different time instances have been collected. It is clear that for content referring to past periods, only RGB image data are available. These data are collected through developed search engine, retrieving images over distributed databases and the social media. More specifically, for the Calw case our data go back to 1850s and 1860s, when the first glass-plate photographs could be collected.
In order to understand and classify spatio-temporal regions in historic urban cities, deep learning strategies are employed. The term deep learning refer to a variety of techniques that can understand complicated patterns by automatically extracting the appropriate features for the task. In the context of 4D-CH-Word, we have focused on convolutional neural networks (CNNs) for two different scenarios; a) timber house identification and b) urban development monitoring. The deep learning approaches supported the 4D reconstruction; i.e. understanding the time period of the reconstructed areas by either, separating new from old buildings, or newly developed areas. In the end, deep learning and advanced filtering approaches allowed the creation of image clusters according to the time period.
The 4D-CH-World outcomes are presented via 4D visualization toolkits. The visualization methods have been developed in a Web-based interface. A user is able to freely navigate within the reconstructed environment and interact with it. Texturized detailed models depict accurate the current place, providing further information when requested by the user. Furthermore, for specific areas, user can experience the evolution through the years; see the current monument, how it was the previous decades and view the raw images that supported the 3D reconstruction. Finally, by selecting a year as a reference (e.g. 2000), we can see (if available) the difference in a specific area just by activating the time filter. That way, the 4D-CH-word team, successfully combined 3D models of a scene associated to different time instances, developing a system for 4D reconstruction and visualization of the results.
The final research achievement of 4D-CH-Word deal with recommendation system able to filter content according to both user preferences and visual attributes. The recommendation system empowers the user to interact with the platform and selects CH objects of interest displayed to him/her through suitable profiles. Users’ profiles are created by measuring the feedback of the users to the system through the activation of recursive machine learning algorithms that on-line update the system response to fit user’s preferences.
The produced software was designed for tablets and phones. It is also possible to run the platform in a standard desktop PC. However, touch screen is strongly suggested. The user is able to navigate freely to the 3D world either as pedestrian (road view) either from above. At specific areas, indicated on the map, we are able to see the actual image that led to the creation of the 3D object.
The tools developed are currently exploited for the digitalization of intangible cultural heritage such as dances through a collaboration with the TERPSICHORE and iTreasures EU funded projects. The innovative tools are also exploited under the framework of the international project being currently in progress regarding the restoration of the Holy Aedicule of the Holy Sepulcher Church in Jerusalem.