Periodic Reporting for period 2 - SILKNOW (SILKNOW. Silk heritage in the Knowledge Society: from punched cards to big data, deep learning and visual / tangible simulations)
Periodo di rendicontazione: 2019-04-01 al 2021-08-31
Silk was a major factor for progress in Europe, mostly along the Western Silk Road’s network of production and market centres. Silk trade also allowed for exchange of ideas and innovations. Punched cards were first used in Jacquard silk looms, long before modern computers were even imagined. Today, too, fashion and high-end textile industries have a huge impact in the EU, reaching €525 billion in annual turnover.
Silk, however, has become a seriously endangered heritage. Although many European specialized museums are devoted to its preservation, they usually lack size and resources to establish networks or connections with other collections. SILKNOW aims to produce an intelligent computational system that goes beyond current technologies in order to improve our understanding of European silk heritage. This legacy will be studied, showcased and preserved through the digital modelling of its weaving techniques (a “Virtual Loom”). Users will access the resulting information through visual and tangible simulations, and experience vastly enhanced search tools, providing better results through automatic visual recognition, advanced spatio-temporal visualization, multilingual and semantically enriched access to existing digital data.
Thus, SILKNOW will improve the understanding of EU heritage and its rich diversity, applying next-generation ICT research to the needs of various users (museums, education, tourism, creative industries, media…), and preserving an intangible heritage (ancient weaving techniques) for younger generations. Its research activities and outputs will have direct impact in computer science and big data management, focusing on searching digital content in heterogeneous, multilingual and multimodal databases. SILKNOW will be possible only with the close cooperation of a multidisciplinary team, including areas as ICT, text analytics, image processing, semantics, big data, 3D printing, art history, terminology, textile fabrication and conservation.
Silk, however, has become a seriously endangered heritage. Although many European specialized museums are devoted to its preservation, they usually lack size and resources to establish networks or connections with other collections. SILKNOW aims to produce an intelligent computational system that goes beyond current technologies in order to improve our understanding of European silk heritage. This legacy will be studied, showcased and preserved through the digital modelling of its weaving techniques (a “Virtual Loom”). Users will access the resulting information through visual and tangible simulations, and experience vastly enhanced search tools, providing better results through automatic visual recognition, advanced spatio-temporal visualization, multilingual and semantically enriched access to existing digital data.
Thus, SILKNOW will improve the understanding of EU heritage and its rich diversity, applying next-generation ICT research to the needs of various users (museums, education, tourism, creative industries, media…), and preserving an intangible heritage (ancient weaving techniques) for younger generations. Its research activities and outputs will have direct impact in computer science and big data management, focusing on searching digital content in heterogeneous, multilingual and multimodal databases. SILKNOW will be possible only with the close cooperation of a multidisciplinary team, including areas as ICT, text analytics, image processing, semantics, big data, 3D printing, art history, terminology, textile fabrication and conservation.
One of the main goals of the project is to make more understandable and accessible the vast digital information that is already available about European silk heritage, while keeping a special focus on small to medium size institutions. In order to meet this challenge, the following resources have been implemented so far (until April 2019):
- A first version of an image classification system, which predicts information like where and when was created, the technique used in its production, etc.
- An ontology for the project data, built on the Erlangen implementation of CIDOC-CRM.
- A thesaurus focused on specialized silk heritage terminology, in four languages (Spanish, Italian, English, French). Based on the general structure of the Getty AAT, it includes more than 500 terms, and will continue to be expanded.
- Tools for automatic semantic extraction and annotation of textual descriptions.
More than 80,000 images and textual information about European silk cultural heritage, gathered from nine online collections and also from project partners. Those data are now being used for training and testing the project tools.
The Virtual Loom is a software tool, now under development, that will be able to identify weaving techniques of some ancient textiles, based on photographic images. It will generate digital models of their weaving structure, thus allowing to generate 3D graphical representations, or tangible (3D-printed) copies. In this first period, methodologies to extract information from images have already allowed to implement in the VL one figured weaving (damask) and three basic (plain weave, twill and satin) techniques. The key variables and text information that must be provided to the tool have also been identified, so that visual data will be soon complemented with semantic data. During the Lyon meeting in September 2018, project members organized a session with textile creative industries discussing their possible uses and expectations for the Virtual Loom. Ongoing work is also focused on the adaptation of 3D printers for tangible reproductions of the weaves, as modeled by the VL, that could be used for education, research and the fashion industry.
A repository and search engine are being designed, with different user groups in mind: researchers, educators, students, visitors, designers, journalists... The following tasks have been carried out, in this regard:
- Technical design of the search engine website.
- Design and development of the user database with MongoDB.
- Review of map tile servers in order to embed maps in the spatio-temporal data viewer.
- Development of plugin prototypes of the Virtual Loom and data viewer.
- Review of the state of the art in ontology visualization.
Knowledge transference has been accomplished through almost 40 dissemination and communication activities, such as conferences, fairs, workshops, media outlets, etc.
- A first version of an image classification system, which predicts information like where and when was created, the technique used in its production, etc.
- An ontology for the project data, built on the Erlangen implementation of CIDOC-CRM.
- A thesaurus focused on specialized silk heritage terminology, in four languages (Spanish, Italian, English, French). Based on the general structure of the Getty AAT, it includes more than 500 terms, and will continue to be expanded.
- Tools for automatic semantic extraction and annotation of textual descriptions.
More than 80,000 images and textual information about European silk cultural heritage, gathered from nine online collections and also from project partners. Those data are now being used for training and testing the project tools.
The Virtual Loom is a software tool, now under development, that will be able to identify weaving techniques of some ancient textiles, based on photographic images. It will generate digital models of their weaving structure, thus allowing to generate 3D graphical representations, or tangible (3D-printed) copies. In this first period, methodologies to extract information from images have already allowed to implement in the VL one figured weaving (damask) and three basic (plain weave, twill and satin) techniques. The key variables and text information that must be provided to the tool have also been identified, so that visual data will be soon complemented with semantic data. During the Lyon meeting in September 2018, project members organized a session with textile creative industries discussing their possible uses and expectations for the Virtual Loom. Ongoing work is also focused on the adaptation of 3D printers for tangible reproductions of the weaves, as modeled by the VL, that could be used for education, research and the fashion industry.
A repository and search engine are being designed, with different user groups in mind: researchers, educators, students, visitors, designers, journalists... The following tasks have been carried out, in this regard:
- Technical design of the search engine website.
- Design and development of the user database with MongoDB.
- Review of map tile servers in order to embed maps in the spatio-temporal data viewer.
- Development of plugin prototypes of the Virtual Loom and data viewer.
- Review of the state of the art in ontology visualization.
Knowledge transference has been accomplished through almost 40 dissemination and communication activities, such as conferences, fairs, workshops, media outlets, etc.
So far, the first 12 months of the project already show promising results and progress beyond the state of the art.
- A first version of the multilingual thesaurus has been developed. It is a controlled vocabulary, specialized in silk textiles, with more than 500 terms that include historical weaving techniques, materials, equipment and local variants. It was converted into SKOS format and published according to the Linked Data principles. It can be browsed using the SKOSMOS tool. It was presented at the Technoheritage 2019 Conference in Seville.
- Development of the first prototype of the Virtual Loom. Different methodologies have been designed, aimed at extracting relevant information from images, adjusting to the appearance of the input data (vg., images of textiles, technical drawings, cartoons...). Techniques based on visual information have been implemented in order to automatically extract patterns.
- A number of meetings with end users have taken place, in order to design the user interface of the spatio-temporal data navigation and visualization component. In this design, visualization techniques (such as hypercubes) are mixed with other innovative interaction and visualization controls. This proposal will be presented in the International Conference on Computational Science (ICCS 2019) in Faro, Portugal.
- In the context of the image processing module, a new method for predicting variables related to silk fabrics provided a proof-of-concept that this task can be solved by deep learning, in principle. Exploiting the inherent relations between different variables of relevance, a new method for deep multitask learning could also be implemented and tested, including another new loss function for training, that allows multitask learning with incomplete samples. In the first sets of experiments, this method achieved quite promising results, with overall accuracies above 78% for all predicted variables.
- A new approach to extending semantic annotation has been developed, trained on Wikipedia concepts to include domain specific concepts. This will be used to incorporate the ontology in Wikifier, an existing multilingual tool for semantic annotation. It uses content and link structure of Wikipedia to probabilistically estimate the most suitable annotations of a given text. To incorporate domain specific terminology, it needs definition of the function estimating distance in the graph of interlinked concepts (the ontology) and relevant context of the terms (to be taken from term definitions in the thesaurus).
- A first version of the multilingual thesaurus has been developed. It is a controlled vocabulary, specialized in silk textiles, with more than 500 terms that include historical weaving techniques, materials, equipment and local variants. It was converted into SKOS format and published according to the Linked Data principles. It can be browsed using the SKOSMOS tool. It was presented at the Technoheritage 2019 Conference in Seville.
- Development of the first prototype of the Virtual Loom. Different methodologies have been designed, aimed at extracting relevant information from images, adjusting to the appearance of the input data (vg., images of textiles, technical drawings, cartoons...). Techniques based on visual information have been implemented in order to automatically extract patterns.
- A number of meetings with end users have taken place, in order to design the user interface of the spatio-temporal data navigation and visualization component. In this design, visualization techniques (such as hypercubes) are mixed with other innovative interaction and visualization controls. This proposal will be presented in the International Conference on Computational Science (ICCS 2019) in Faro, Portugal.
- In the context of the image processing module, a new method for predicting variables related to silk fabrics provided a proof-of-concept that this task can be solved by deep learning, in principle. Exploiting the inherent relations between different variables of relevance, a new method for deep multitask learning could also be implemented and tested, including another new loss function for training, that allows multitask learning with incomplete samples. In the first sets of experiments, this method achieved quite promising results, with overall accuracies above 78% for all predicted variables.
- A new approach to extending semantic annotation has been developed, trained on Wikipedia concepts to include domain specific concepts. This will be used to incorporate the ontology in Wikifier, an existing multilingual tool for semantic annotation. It uses content and link structure of Wikipedia to probabilistically estimate the most suitable annotations of a given text. To incorporate domain specific terminology, it needs definition of the function estimating distance in the graph of interlinked concepts (the ontology) and relevant context of the terms (to be taken from term definitions in the thesaurus).