Periodic Reporting for period 2 - SURVANT (SURveillance Video Archives iNvestigation assisTant)
Période du rapport: 2018-01-01 au 2018-12-31
The SURVANT project will deliver a system that can assist investigators to search efficiently and effectively in video archives is expected to contribute towards fighting crime and illicit activities, improving the sense and essence of security for the citizens. Safety and security, reduction of loss/theft, vandalism prevention, harassment prevention and regulatory compliance are among the main driving applications of surveillance systems.
SURVANT will deliver an innovative system that will collect the relevant videos from heterogeneous repositories, extract inter/intra-camera video analytics, enrich the analytics using reasoning and inference technologies, and offer a unified search interface to the user. An intuitive interface with a relaxed learning curve will assist the user create accurate search queries and receive the results using advanced visualization tools. Ethics and legal issues are kept into account and data protection mechanisms are integrated in the system design.
SURVANT will deliver an innovative system that will collect the relevant videos from heterogeneous repositories, extract inter/intra-camera video analytics, enrich the analytics using reasoning and inference technologies, and offer a unified search interface to the user. An intuitive interface with a relaxed learning curve will assist the user create accurate search queries and receive the results using advanced visualization tools. Ethics and legal issues are kept into account and data protection mechanisms are integrated in the system design.
During the first year of the project, the major effort was spent in studying deeply all the aspects of video and image analysis and indexing, from the object detection and tracking, to people recognition, to the data protection and privacy assurance requirements that SURVANT has to satisfy.
Many use cases and scenarios have been discussed and analysed, in order to find the best suitable way for demonstrating the validity and the potential of SURVANT in its application domain. Legal and ethical issues were also explored in details, with a focus on the best practises for protecting privacy and personal data and the key differences between SURVANT and ADVISE.
In addition, from the definition of the technical requirements that SURVANT will cover, a first approach and design of analysis tools for information gathering and extraction was completed. Mockups were also realised in order to provide a complete idea of the interaction between users and system.
Many use cases and scenarios have been discussed and analysed, in order to find the best suitable way for demonstrating the validity and the potential of SURVANT in its application domain. Legal and ethical issues were also explored in details, with a focus on the best practises for protecting privacy and personal data and the key differences between SURVANT and ADVISE.
In addition, from the definition of the technical requirements that SURVANT will cover, a first approach and design of analysis tools for information gathering and extraction was completed. Mockups were also realised in order to provide a complete idea of the interaction between users and system.
Progress beyond the SotA and results
SURVANT aims to develop a product that will introduce solutions beyond the industry state-of-the-art to face the challenges identified in the markets targeted. A comprehensive list of those challenges, the solutions envisioned and the ambition involved, indicated by their technology readiness level (TRL), is provided:
1)Scalability: SURVANT will address system scalability issues that emerge from the explosion in the amount of available video content employing and extending the OpenZoo framework (TRL 7). OpenZoo is an open-source, MIT licensed, distributed, stream and batch processing framework and includes remote, automated deployment of processing services, transparent communication and allocation of resources, and cross-platform support. OpenZoo has been tested as a real time search and analytics framework, based on images shared through Twitter, during the CUbRIK project (GA 287704).
2) Video analysis: SURVANT will perform video analysis employing Deep Learning (DL) techniques (TRL 6). Specialized research will be performed to analyse the current state-of-the-art DL implementations and adjust them to the specific needs of surveillance video processing. Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) will be used to analyse static and motion content, respectively. Inter-camera tracking and re-identification will be at the core of attention. Optimal balance between speed and accuracy will be pursued.
3) Event Enrichment and Reasoning: SURVANT will deliver an inference framework able to combine together low-level information and semantic annotations to enable automated reasoning mechanisms to discover high-level events and/or investigative hypotheses. SURVANT will evolve the OWL tableau reasoning framework developed in ADVISE (TRL6), based on a SWRL (Semantic Web Rule Language) approach, applying the event calculus formalism in order to allow the event reconstruction in a narrative way taking into account spatial-temporal coordinates useful to track the crime and predict its evolution in the time and space.
4) Video and Image indexing: One of the main challenges when dealing with large scale image/video retrieval application is how to efficiently index the content extracted from large image/video repositories. While handling semantically-mapped content (event type and attributes) is relatively straightforward, using well-established data management systems, low and mid-level features are much more difficult to handle efficiently, due to their high-dimensional aspect.
5) Intelligent, human-in-the-loop, GIS-based user interface: In the scope of ADVISE, a GIS-based user interface (TRL 7) has been developed and demonstrated, allowing the user to execute targeted queries, in terms of location and time, and visualize the returned results in analytical or summarized forms. In LASIE, advanced relevant feedback tools (TRL 6) have been developed allowing the user targeted queries in terms of semantics. The user is assisted during the query formulation phase, while the query expansion is focussed on semantic analysis and takes into account user's interests by the analysis of previous queries.
6) Legal and ethical issues and LEP framework: SURVANT will deliver an intelligent and extensible framework for automated video analysis and analytics respectful of human rights according to European legal framework for privacy and data protection, avoiding disproportional use of personal data. ADVISE had adopted a privacy by design and privacy by default approach to develop a Content Agent Mediator (TRL7) focused on ontological structures in the specific field of surveillance to ensure privacy protection.
Based on the results already achieved, key functionalities have been identified that can provide high added value for potential customers compared to the competition have been identified and presented below:
1) Situational awareness framework.
2) Advanced content-based search.
3) Search expansion tools.
Potential Impacts
SURVANT will have a huge impact in terms of:
1) Improved productivity and effectiveness of LEA's crime analysts and operational teams;
2) Increased capability in crime prevention via a quicker and more effective criminal trail tracing and a quicker identification of crime behavioural patterns with the consequent reduction of the impact of criminals’ activities;
3) Improved capability of LEAs to analyse day by day the increasingly huge amount of surveillance systems videos and the reduction of the gaps between LEA’s capabilities and the growing variety and volume of 4) Develop a video analysis and event detection technology capable of discovering correlations among criminal facts and detecting criminal movements before and after the suspicions event, meeting the needs of European LEA and contributing to the fight against crime and terrorism;
5) Seamless interfacing with multiple heterogeneous repositories/sources to retrieve relevant videos and available metadata;
6) Exploit the partners’ innovation capacity to efficiently integrate the proposed innovative system into a pilot that will be set up at the ADM premises;
7) Strengthen the competitiveness and growth of the industrial partners, opening a complete new market.
SURVANT aims to develop a product that will introduce solutions beyond the industry state-of-the-art to face the challenges identified in the markets targeted. A comprehensive list of those challenges, the solutions envisioned and the ambition involved, indicated by their technology readiness level (TRL), is provided:
1)Scalability: SURVANT will address system scalability issues that emerge from the explosion in the amount of available video content employing and extending the OpenZoo framework (TRL 7). OpenZoo is an open-source, MIT licensed, distributed, stream and batch processing framework and includes remote, automated deployment of processing services, transparent communication and allocation of resources, and cross-platform support. OpenZoo has been tested as a real time search and analytics framework, based on images shared through Twitter, during the CUbRIK project (GA 287704).
2) Video analysis: SURVANT will perform video analysis employing Deep Learning (DL) techniques (TRL 6). Specialized research will be performed to analyse the current state-of-the-art DL implementations and adjust them to the specific needs of surveillance video processing. Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) will be used to analyse static and motion content, respectively. Inter-camera tracking and re-identification will be at the core of attention. Optimal balance between speed and accuracy will be pursued.
3) Event Enrichment and Reasoning: SURVANT will deliver an inference framework able to combine together low-level information and semantic annotations to enable automated reasoning mechanisms to discover high-level events and/or investigative hypotheses. SURVANT will evolve the OWL tableau reasoning framework developed in ADVISE (TRL6), based on a SWRL (Semantic Web Rule Language) approach, applying the event calculus formalism in order to allow the event reconstruction in a narrative way taking into account spatial-temporal coordinates useful to track the crime and predict its evolution in the time and space.
4) Video and Image indexing: One of the main challenges when dealing with large scale image/video retrieval application is how to efficiently index the content extracted from large image/video repositories. While handling semantically-mapped content (event type and attributes) is relatively straightforward, using well-established data management systems, low and mid-level features are much more difficult to handle efficiently, due to their high-dimensional aspect.
5) Intelligent, human-in-the-loop, GIS-based user interface: In the scope of ADVISE, a GIS-based user interface (TRL 7) has been developed and demonstrated, allowing the user to execute targeted queries, in terms of location and time, and visualize the returned results in analytical or summarized forms. In LASIE, advanced relevant feedback tools (TRL 6) have been developed allowing the user targeted queries in terms of semantics. The user is assisted during the query formulation phase, while the query expansion is focussed on semantic analysis and takes into account user's interests by the analysis of previous queries.
6) Legal and ethical issues and LEP framework: SURVANT will deliver an intelligent and extensible framework for automated video analysis and analytics respectful of human rights according to European legal framework for privacy and data protection, avoiding disproportional use of personal data. ADVISE had adopted a privacy by design and privacy by default approach to develop a Content Agent Mediator (TRL7) focused on ontological structures in the specific field of surveillance to ensure privacy protection.
Based on the results already achieved, key functionalities have been identified that can provide high added value for potential customers compared to the competition have been identified and presented below:
1) Situational awareness framework.
2) Advanced content-based search.
3) Search expansion tools.
Potential Impacts
SURVANT will have a huge impact in terms of:
1) Improved productivity and effectiveness of LEA's crime analysts and operational teams;
2) Increased capability in crime prevention via a quicker and more effective criminal trail tracing and a quicker identification of crime behavioural patterns with the consequent reduction of the impact of criminals’ activities;
3) Improved capability of LEAs to analyse day by day the increasingly huge amount of surveillance systems videos and the reduction of the gaps between LEA’s capabilities and the growing variety and volume of 4) Develop a video analysis and event detection technology capable of discovering correlations among criminal facts and detecting criminal movements before and after the suspicions event, meeting the needs of European LEA and contributing to the fight against crime and terrorism;
5) Seamless interfacing with multiple heterogeneous repositories/sources to retrieve relevant videos and available metadata;
6) Exploit the partners’ innovation capacity to efficiently integrate the proposed innovative system into a pilot that will be set up at the ADM premises;
7) Strengthen the competitiveness and growth of the industrial partners, opening a complete new market.