Periodic Reporting for period 2 - VICTORIA (Video analysis for Investigation of Criminal and TerrORIst Activities)
Berichtszeitraum: 2018-11-01 bis 2020-11-30
Video material collected and analysed by Law Enforcement Agencies (LEA) from varied sources is a critical component in investigations of major criminal acts and terrorist attacks.
After these acts, each hour is precious to find clues to help capture offenders and prevent further acts, to save lives and prevent further damage. There is a need for efficient tools to assist LEAs in their daily video investigation tasks, for processing huge volumes of video material with less effort and to find clues and evidence faster.
Core technology for such tools is video analytics. This technology from the computer vision domain, aims at finding and recognising objects, persons, scenes, etc. Many previous projects have developed them for other applications, also relevant for video investigation tools. However, their current TRL is only sufficient for applications in a controlled environment where camera position and parameters, lighting conditions, image quality, etc., are suitable.
These analytics TRLs are however too low for using them in video investigation tools, where the environment conditions of the videos vary a lot depending on their source. The analytics used for video investigation must be robust to different conditions such as camera characteristics, image quality, viewing angles, and moving cameras.
Available video investigation tools are generally closed systems, limited to video analysis technology mastered in-house by the vendor and not open to integrate other video analytics technologies provided by third parties. This limitation hinders the development of innovation activities leading to new products offering to end-users enhanced features and/or performance, and flexibility to better fit specific user requirements, and globally the need of video investigation systems to adapt to evolving user needs and the continuous progress of technology.
VICTORIA aimed at creating a real breakthrough regarding functionality and usability of video analysis tools used for legal investigations. It addressed the following objectives: develop TRL-6 video analysis technology boosting LEAs' video investigation capacity, increase the usability of video analysis tools for legal investigations, create a business model and ecosystem for the video investigation field to grow and prosper, train LEA investigators in the use of advanced video investigation tools, and ensure the VICTORIA tools are in line with EU Legal-Ethical-Privacy rules.
After these acts, each hour is precious to find clues to help capture offenders and prevent further acts, to save lives and prevent further damage. There is a need for efficient tools to assist LEAs in their daily video investigation tasks, for processing huge volumes of video material with less effort and to find clues and evidence faster.
Core technology for such tools is video analytics. This technology from the computer vision domain, aims at finding and recognising objects, persons, scenes, etc. Many previous projects have developed them for other applications, also relevant for video investigation tools. However, their current TRL is only sufficient for applications in a controlled environment where camera position and parameters, lighting conditions, image quality, etc., are suitable.
These analytics TRLs are however too low for using them in video investigation tools, where the environment conditions of the videos vary a lot depending on their source. The analytics used for video investigation must be robust to different conditions such as camera characteristics, image quality, viewing angles, and moving cameras.
Available video investigation tools are generally closed systems, limited to video analysis technology mastered in-house by the vendor and not open to integrate other video analytics technologies provided by third parties. This limitation hinders the development of innovation activities leading to new products offering to end-users enhanced features and/or performance, and flexibility to better fit specific user requirements, and globally the need of video investigation systems to adapt to evolving user needs and the continuous progress of technology.
VICTORIA aimed at creating a real breakthrough regarding functionality and usability of video analysis tools used for legal investigations. It addressed the following objectives: develop TRL-6 video analysis technology boosting LEAs' video investigation capacity, increase the usability of video analysis tools for legal investigations, create a business model and ecosystem for the video investigation field to grow and prosper, train LEA investigators in the use of advanced video investigation tools, and ensure the VICTORIA tools are in line with EU Legal-Ethical-Privacy rules.
Main results of first period included user requirements, real and synthetic video datasets, ethical and legal specifications, and Video Analytics Platform (VAP) data protection by design guidelines and impact assessment. Software and hardware architecture of VAP and video analytics plug-in framework were defined. First version of all algorithms integrated into VAP V1 and implementation of interfacing between VAP and all analytics components were accomplished as well as automatic scene reconstruction, manual correction, and motion reconstruction from camera networks regardless of calibration level and visualization of 4D scene reconstruction with metadata for straightforward user interaction.
Other accomplishments included backbone implementation for VAP V1, GUI implementation, training methodology and curriculum, dissemination strategy, and the VAFI community. Also, VICTORIA officially came up with a new work item proposal to upgrade and expand replacement of ISO 22311.
For the second period, the tools for synthetic video generation were improved based on their use by VICTORIA partners, and their feedback. Improved technical and organisational guidelines for the VAP and its plug-ins as well as ethical and a legal management toolkit were completed, including data protection.
A second version of the VAP were developed, integrating improved (robustness, accuracy, speed…) analytics modules from technical partners.
Technical progress included extension of Common Objects in Context (COCO) dataset, the large-scale object detection, segmentation, and captioning dataset, latest variant re-training and evaluation of YOLO – “You Only Look Once”, (YOLOv3 detector), and further development and evaluation of tracking approaches Kernelized Correlation Filter (KCF) tracker (Deep-KCF) and Deep Learning, long short-term memory (DeepSort-LSTM). Four patents were filed, three on methods for detection and tracking and one on deep learning models and training for textual description.
Audio Event Detection module for the VAP V2 was improved and new audio event detection tool was developed for footsteps detection, with different detectable audio events implemented in Python: Speech, Footstep, Car horn and Motors/engines detection. All these detection tools are based on the same principles: No need for training step and based on physical modelling of acoustic events.
Progress for audio similarity retrieval module audio synchronisation module between VAP1 and VAP2 was obtained as well as improvements on the semi-automatic scene reconstruction including, the 3D registration tool “CACTUS-3D” with gained online and local map server support with compass, with textured mesh with high texture resolution details for the static scene reconstruction, geo-registration improvements, and dynamic reconstruction of humans in videos.
Improvements were also achieved in the 4D crime scene investigation allowing Virtual Reality with full control over scene, collaborative exploration with VR, distance measuring, annotating scene with real distances, temporal multi-dimensional horizon charts and major performance improvements for larger, drone-captured scenes as well as dynamic point clouds.
Final implementation of all Graphical User Interface (GUI) components, with revision of workflows, implementation, and integration of the components from all technical partners into the VAP V1 and V2 and delivery of the VAP V2 with manuals to consortium end user sites was done.
Gathering of manuals of the VAP and analysis tools and their usage to produce training contents following defined curriculum, migration of the training content from PoliformaT to Moodle and organisation of two training workshops was carried out.
LEA’s carried out trial of VAP using data typical to that found in police investigations and in a couple of instances using the VAP on a live investigation, with LEAs demonstrating VAP to their colleagues and peers.
Post-trial workshop in Bochum allowed LEAs and technical partners to openly discuss their experience with the VAP and form a roadmap for future development.
The project produced a video trailer, which was broadcast on YouTube, participated in numerous conferences and produced a plan showing how partners envisage exploitation of their results after the end of the project.
Other accomplishments included backbone implementation for VAP V1, GUI implementation, training methodology and curriculum, dissemination strategy, and the VAFI community. Also, VICTORIA officially came up with a new work item proposal to upgrade and expand replacement of ISO 22311.
For the second period, the tools for synthetic video generation were improved based on their use by VICTORIA partners, and their feedback. Improved technical and organisational guidelines for the VAP and its plug-ins as well as ethical and a legal management toolkit were completed, including data protection.
A second version of the VAP were developed, integrating improved (robustness, accuracy, speed…) analytics modules from technical partners.
Technical progress included extension of Common Objects in Context (COCO) dataset, the large-scale object detection, segmentation, and captioning dataset, latest variant re-training and evaluation of YOLO – “You Only Look Once”, (YOLOv3 detector), and further development and evaluation of tracking approaches Kernelized Correlation Filter (KCF) tracker (Deep-KCF) and Deep Learning, long short-term memory (DeepSort-LSTM). Four patents were filed, three on methods for detection and tracking and one on deep learning models and training for textual description.
Audio Event Detection module for the VAP V2 was improved and new audio event detection tool was developed for footsteps detection, with different detectable audio events implemented in Python: Speech, Footstep, Car horn and Motors/engines detection. All these detection tools are based on the same principles: No need for training step and based on physical modelling of acoustic events.
Progress for audio similarity retrieval module audio synchronisation module between VAP1 and VAP2 was obtained as well as improvements on the semi-automatic scene reconstruction including, the 3D registration tool “CACTUS-3D” with gained online and local map server support with compass, with textured mesh with high texture resolution details for the static scene reconstruction, geo-registration improvements, and dynamic reconstruction of humans in videos.
Improvements were also achieved in the 4D crime scene investigation allowing Virtual Reality with full control over scene, collaborative exploration with VR, distance measuring, annotating scene with real distances, temporal multi-dimensional horizon charts and major performance improvements for larger, drone-captured scenes as well as dynamic point clouds.
Final implementation of all Graphical User Interface (GUI) components, with revision of workflows, implementation, and integration of the components from all technical partners into the VAP V1 and V2 and delivery of the VAP V2 with manuals to consortium end user sites was done.
Gathering of manuals of the VAP and analysis tools and their usage to produce training contents following defined curriculum, migration of the training content from PoliformaT to Moodle and organisation of two training workshops was carried out.
LEA’s carried out trial of VAP using data typical to that found in police investigations and in a couple of instances using the VAP on a live investigation, with LEAs demonstrating VAP to their colleagues and peers.
Post-trial workshop in Bochum allowed LEAs and technical partners to openly discuss their experience with the VAP and form a roadmap for future development.
The project produced a video trailer, which was broadcast on YouTube, participated in numerous conferences and produced a plan showing how partners envisage exploitation of their results after the end of the project.
VICTORIA better protects citizens against crime and terrorist acts through enhanced video investigation capabilities, enabling LEAs to set-up targeted operations to stop offenders, and provide evidence for lawsuits. The solving of more criminal cases, the acceleration of investigations will reinforce the confidence of European citizens in their police forces.
The market potential of video surveillance applications with products designed and produced in the EU is enabled, by joining capabilities of leading research institutes and industrial leaders, working with LEAs to develop the required technology for products meeting end-user needs.
The market potential of video surveillance applications with products designed and produced in the EU is enabled, by joining capabilities of leading research institutes and industrial leaders, working with LEAs to develop the required technology for products meeting end-user needs.