Periodic Reporting for period 1 - SHIFT (MetamorphoSis of cultural Heritage Into augmented hypermedia assets For enhanced accessibiliTy and inclusion)
Période du rapport: 2022-10-01 au 2023-09-30
The SHIFT project will leverage on the advances of Artificial Intelligence (AI) and Machine Learning (ML) technologies to improve cultural heritage access to European citizens experiencing sensory impairment. The project aims enrich the experience of interacting with cultural assets through visual, auditory, and sensory modalities, such as using haptics. Additionally, the SHIFT project also will develop tools and methodologies to enable contemporary description of cultural assets through language evolution models.
The main Project Objectives:
o Enable cultural heritage institutions to successfully adopt digital strategies to promote public engagement
o Offer unique experiences to the visitors through a hybrid of visual-auditory-haptics sensors for experiencing cultural heritage
o Enable digital interactions with cultural assets using haptic technology
o Represent cultural heritage to contemporary times through the development of language models
o Adopt accessibility by design principles and tools to promote compliance with International standards
o Implement tools for creating ownership chain for protecting copyrights of digital media
The following technologies and platforms will be used:
• AI algorithms implemented for enabling visual accessibility of cultural heritage content for people experiencing partial visual impairment
• Language model development to map historical references to contemporary language usage.
• Design and implementation of haptic interfaces for enriching access to cultural heritage assets
• AI algorithms to translate text into audio format for achieving economies of scale in emotional audio transcription
The main Project Objectives:
o Enable cultural heritage institutions to successfully adopt digital strategies to promote public engagement
o Offer unique experiences to the visitors through a hybrid of visual-auditory-haptics sensors for experiencing cultural heritage
o Enable digital interactions with cultural assets using haptic technology
o Represent cultural heritage to contemporary times through the development of language models
o Adopt accessibility by design principles and tools to promote compliance with International standards
o Implement tools for creating ownership chain for protecting copyrights of digital media
The following technologies and platforms will be used:
• AI algorithms implemented for enabling visual accessibility of cultural heritage content for people experiencing partial visual impairment
• Language model development to map historical references to contemporary language usage.
• Design and implementation of haptic interfaces for enriching access to cultural heritage assets
• AI algorithms to translate text into audio format for achieving economies of scale in emotional audio transcription
The main activities performed in the first reporting period (M1 – M12) were the following:
WP1:
- User Requirements defined.
- Analysis of relevant case studies.
- Evidence of the need for systematic investments in IT infrastructure, standardization and compatibility.
- Survey of the main directions in which public libraries operate in Romania.
- Identification of the collections of representative CH assets.
- Delivered a comprehensive array of testing materials.
- SHIFT end-user ethics and legal framework was elaborated.
WP2:
- SRGAN (Super-Resolution Generative Adversarial Networks) and ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) models were utilized to enhance the quality of the image for subsequent stages.
- A systematic review of the human attention model was studied based on the RealEye dataset.
- Detailed review of the PINNs (Physics-informed neural networks) application areas.
- Extensive review of the literature in the context of generative AI.
- In order to identify one or more actions depicted in a video, advanced architectures (like GluonCV library) were utilized to generate actions from a predetermined list that we have established.
WP3:
- Develop and refine advanced Natural Language Processing (NLP) algorithms for creating inclusive and detailed textual representations of cultural heritage (CH) assets.
- Enriching narratives with contextual descriptions, employing and experimenting techniques such as prompt engineering and utilizing frameworks like LangChain for personalization and style transfer, particularly benefiting visually impaired individuals and children.
- Enhancing the versatility and potential of advanced models such as the LLaMA (Large Language Model Meta AI) and its successor, LLaMA2.
- Building the first version of the Text to Affective Speech production tool (TTS -Tool).
- In order to support an alternative approach for representing Cultural Heritage (CH) assets via haptic-based interaction an initial demo scene in a Virtual Reality (VR) environment has been developed as a testing ground.
- Exhaustive literature review to evaluate the state-of-the-art technologies in the domain of XR accessibility.
- Two demonstrative AR/VR applications were created that prioritize accessibility as a core principle.
WP4:
- Extensive research to explore the diverse domains extracted from various types of CH assets (factual information, contextual information and subjective information).
- Exploring the methods for interlinking assets and selecting the most suitable external databases.
- The study of current exiting cultural asset repositories, to identify existing means of integration.
- The elaboration of APIs used to expose functionality and resources existing in SHIFT Repository.
- Overview of the existing standards and semantic models for documenting the traceability of the digital media rights were conducted.
WP5:
- Define the main technical characteristics of the SHIFT Platform
- Establish the methodology to be used and definition of the Software Architecture.
- Definition of the communication protocols between modules, Starting the implementation of API REST services and a Common Gateway was created.
WP6:
- A communication and dissemination plan was compiled.
- In this period SHIFT social media accounts posted at least 87 times, reaching thousands of people across all media.
- Active participation in workshops, conferences, and exhibitions in order to explore opportunities to present the project and its solutions.
WP7:
- Administrative and financial management of the project was carried out.
- Two General Assembly face-to-face meetings were organized, in addition to numerous internal meetings.
- Data Management & Ethical Compliance plan was elaborated.
WP1:
- User Requirements defined.
- Analysis of relevant case studies.
- Evidence of the need for systematic investments in IT infrastructure, standardization and compatibility.
- Survey of the main directions in which public libraries operate in Romania.
- Identification of the collections of representative CH assets.
- Delivered a comprehensive array of testing materials.
- SHIFT end-user ethics and legal framework was elaborated.
WP2:
- SRGAN (Super-Resolution Generative Adversarial Networks) and ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) models were utilized to enhance the quality of the image for subsequent stages.
- A systematic review of the human attention model was studied based on the RealEye dataset.
- Detailed review of the PINNs (Physics-informed neural networks) application areas.
- Extensive review of the literature in the context of generative AI.
- In order to identify one or more actions depicted in a video, advanced architectures (like GluonCV library) were utilized to generate actions from a predetermined list that we have established.
WP3:
- Develop and refine advanced Natural Language Processing (NLP) algorithms for creating inclusive and detailed textual representations of cultural heritage (CH) assets.
- Enriching narratives with contextual descriptions, employing and experimenting techniques such as prompt engineering and utilizing frameworks like LangChain for personalization and style transfer, particularly benefiting visually impaired individuals and children.
- Enhancing the versatility and potential of advanced models such as the LLaMA (Large Language Model Meta AI) and its successor, LLaMA2.
- Building the first version of the Text to Affective Speech production tool (TTS -Tool).
- In order to support an alternative approach for representing Cultural Heritage (CH) assets via haptic-based interaction an initial demo scene in a Virtual Reality (VR) environment has been developed as a testing ground.
- Exhaustive literature review to evaluate the state-of-the-art technologies in the domain of XR accessibility.
- Two demonstrative AR/VR applications were created that prioritize accessibility as a core principle.
WP4:
- Extensive research to explore the diverse domains extracted from various types of CH assets (factual information, contextual information and subjective information).
- Exploring the methods for interlinking assets and selecting the most suitable external databases.
- The study of current exiting cultural asset repositories, to identify existing means of integration.
- The elaboration of APIs used to expose functionality and resources existing in SHIFT Repository.
- Overview of the existing standards and semantic models for documenting the traceability of the digital media rights were conducted.
WP5:
- Define the main technical characteristics of the SHIFT Platform
- Establish the methodology to be used and definition of the Software Architecture.
- Definition of the communication protocols between modules, Starting the implementation of API REST services and a Common Gateway was created.
WP6:
- A communication and dissemination plan was compiled.
- In this period SHIFT social media accounts posted at least 87 times, reaching thousands of people across all media.
- Active participation in workshops, conferences, and exhibitions in order to explore opportunities to present the project and its solutions.
WP7:
- Administrative and financial management of the project was carried out.
- Two General Assembly face-to-face meetings were organized, in addition to numerous internal meetings.
- Data Management & Ethical Compliance plan was elaborated.
The SHIFT project will bring benefits to society by supporting the mainstreaming of new digital tools for creative and commercial collaborations, as well as cutting-edge solutions. The expected impacts are summarized below:
- Developing, Integrating and strengthening the use of digital technologies to protect, conserve, restore cultural heritage and the arts in complementarity with other research methods.
- Facilitating and expanding access to cultural assets through digital technologies and tools and in parallel or as an alternative to physical access to cultural heritage.
- Supporting AI benchmarking, including time analysis and other digital means to improve innovation and knowledge sharing in the cultural and creative sectors.
- Increasing the competitiveness of the cultural and creative industries on the domestic and international market and providing opportunities for the creation of new and sustainable jobs.
- Exploring the role of digital tools such as 3D/4D simulations, virtual and augmented reality technologies in engaging with cultural heritage during and after the COVID-19 crisis.
- Using digital tools and other outputs to provide effective and comprehensive solutions to the real needs of access, protection and preservation of cultural heritage, including the "born digital".
- Developing, Integrating and strengthening the use of digital technologies to protect, conserve, restore cultural heritage and the arts in complementarity with other research methods.
- Facilitating and expanding access to cultural assets through digital technologies and tools and in parallel or as an alternative to physical access to cultural heritage.
- Supporting AI benchmarking, including time analysis and other digital means to improve innovation and knowledge sharing in the cultural and creative sectors.
- Increasing the competitiveness of the cultural and creative industries on the domestic and international market and providing opportunities for the creation of new and sustainable jobs.
- Exploring the role of digital tools such as 3D/4D simulations, virtual and augmented reality technologies in engaging with cultural heritage during and after the COVID-19 crisis.
- Using digital tools and other outputs to provide effective and comprehensive solutions to the real needs of access, protection and preservation of cultural heritage, including the "born digital".