Periodic Reporting for period 1 - SHIFT (MetamorphoSis of cultural Heritage Into augmented hypermedia assets For enhanced accessibiliTy and inclusion)
Okres sprawozdawczy: 2022-10-01 do 2023-09-30
The main Project Objectives:
o Enable cultural heritage institutions to successfully adopt digital strategies to promote public engagement
o Offer unique experiences to the visitors through a hybrid of visual-auditory-haptics sensors for experiencing cultural heritage
o Enable digital interactions with cultural assets using haptic technology
o Represent cultural heritage to contemporary times through the development of language models
o Adopt accessibility by design principles and tools to promote compliance with International standards
o Implement tools for creating ownership chain for protecting copyrights of digital media
The following technologies and platforms will be used:
• AI algorithms implemented for enabling visual accessibility of cultural heritage content for people experiencing partial visual impairment
• Language model development to map historical references to contemporary language usage.
• Design and implementation of haptic interfaces for enriching access to cultural heritage assets
• AI algorithms to translate text into audio format for achieving economies of scale in emotional audio transcription
WP1:
- User Requirements defined.
- Analysis of relevant case studies.
- Evidence of the need for systematic investments in IT infrastructure, standardization and compatibility.
- Survey of the main directions in which public libraries operate in Romania.
- Identification of the collections of representative CH assets.
- Delivered a comprehensive array of testing materials.
- SHIFT end-user ethics and legal framework was elaborated.
WP2:
- SRGAN (Super-Resolution Generative Adversarial Networks) and ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) models were utilized to enhance the quality of the image for subsequent stages.
- A systematic review of the human attention model was studied based on the RealEye dataset.
- Detailed review of the PINNs (Physics-informed neural networks) application areas.
- Extensive review of the literature in the context of generative AI.
- In order to identify one or more actions depicted in a video, advanced architectures (like GluonCV library) were utilized to generate actions from a predetermined list that we have established.
WP3:
- Develop and refine advanced Natural Language Processing (NLP) algorithms for creating inclusive and detailed textual representations of cultural heritage (CH) assets.
- Enriching narratives with contextual descriptions, employing and experimenting techniques such as prompt engineering and utilizing frameworks like LangChain for personalization and style transfer, particularly benefiting visually impaired individuals and children.
- Enhancing the versatility and potential of advanced models such as the LLaMA (Large Language Model Meta AI) and its successor, LLaMA2.
- Building the first version of the Text to Affective Speech production tool (TTS -Tool).
- In order to support an alternative approach for representing Cultural Heritage (CH) assets via haptic-based interaction an initial demo scene in a Virtual Reality (VR) environment has been developed as a testing ground.
- Exhaustive literature review to evaluate the state-of-the-art technologies in the domain of XR accessibility.
- Two demonstrative AR/VR applications were created that prioritize accessibility as a core principle.
WP4:
- Extensive research to explore the diverse domains extracted from various types of CH assets (factual information, contextual information and subjective information).
- Exploring the methods for interlinking assets and selecting the most suitable external databases.
- The study of current exiting cultural asset repositories, to identify existing means of integration.
- The elaboration of APIs used to expose functionality and resources existing in SHIFT Repository.
- Overview of the existing standards and semantic models for documenting the traceability of the digital media rights were conducted.
WP5:
- Define the main technical characteristics of the SHIFT Platform
- Establish the methodology to be used and definition of the Software Architecture.
- Definition of the communication protocols between modules, Starting the implementation of API REST services and a Common Gateway was created.
WP6:
- A communication and dissemination plan was compiled.
- In this period SHIFT social media accounts posted at least 87 times, reaching thousands of people across all media.
- Active participation in workshops, conferences, and exhibitions in order to explore opportunities to present the project and its solutions.
WP7:
- Administrative and financial management of the project was carried out.
- Two General Assembly face-to-face meetings were organized, in addition to numerous internal meetings.
- Data Management & Ethical Compliance plan was elaborated.
- Developing, Integrating and strengthening the use of digital technologies to protect, conserve, restore cultural heritage and the arts in complementarity with other research methods.
- Facilitating and expanding access to cultural assets through digital technologies and tools and in parallel or as an alternative to physical access to cultural heritage.
- Supporting AI benchmarking, including time analysis and other digital means to improve innovation and knowledge sharing in the cultural and creative sectors.
- Increasing the competitiveness of the cultural and creative industries on the domestic and international market and providing opportunities for the creation of new and sustainable jobs.
- Exploring the role of digital tools such as 3D/4D simulations, virtual and augmented reality technologies in engaging with cultural heritage during and after the COVID-19 crisis.
- Using digital tools and other outputs to provide effective and comprehensive solutions to the real needs of access, protection and preservation of cultural heritage, including the "born digital".