Final Report Summary - PROACTIVE (PRedictive reasOning and multi-source fusion empowering AntiCipation of attacks and Terrorist actions In Urban EnVironmEnts)
Executive Summary:
During the last decade the world has witnessed major terrorist attacks in urban environments, such as the collapse of New York’s Twin Towers on 11th September 2001, the bombing of packed commuter trains in Madrid on 11th March 2004, as well as the London bombings in July 2005. These incidents have manifested that modern cities are very susceptible to terrorist attacks. This weakness is directly related to the special characteristics of urban areas, which comprise a number of man-made structures (buildings, infrastructure and facilities), along with variable and complex dynamics of economic, political and cultural interactions. Broadly definable as “social variables” of the urban context, these interactions need to be addressed: an example is the concepts of “community” and “familiarity”: the strength of community is familiarity. The communities of urban-based migrant populations, left unassimilated within the larger scope of established metropolitan areas, often become inward focused and ethnocentric havens for undocumented travellers, criminal activity focused on immigration/documentation, and a base for material support to terrorists. Terrorists actively seek out the familiarity of communities sympathetic to their cause to solicit funding, support, documentation, jobs, and recruits.
Alternatively, the quality of familiarity in a dense metropolitan environment can be significantly diminished and also serve terrorist operational planning and goals. Terrorists embrace the relative anonymity of big cities in all phases of their operational activities. When selecting targets, they survey facilities or venues without being noticed. In this perspective it is crucial to develop original and integrated tools improving, among others: Identification and Tracking of Terrorist Suspects in Urban Environments; Suspicious Vehicular, Individual Behaviour/Pattern Recognition; Situational Awareness.
Combating terrorists in an urban environment is a very challenging task, given the complex social inter-relationships and interactions of the various groups engaged in a terrorist incident, the high degree of uncertainty associated with the location and origin of potential attacks, the need to protect several infrastructural facilities (such as telecommunications, energy, and transportation hubs), as well as the more general cultural, political and social conditions of an urban environment. In response to these challenges, security forces, counter-terrorism units, police authorities and intelligence agencies are increasingly looking into novel processes and tools that could help them to perceive indications of terrorist attacks, while also predicting potential incidents in order to adapt their operations accordingly.
A vast amount of information related to data exchanged through novel surveillance technologies should be analyzed in order to provide counter terror measures. This is difficult, since information scales exponentially in terms of location-space, time and means (multi-source information). In order to efficiently confront the creativity, innovation and adaptability of the terrorists, security forces can leverage information, knowledge and intelligence stemming from multiple levels including:
• The technical level, which deals with the discovery of technical knowledge about the technologies employed by terrorists.
• The imaginary level, which comprises knowledge collected from a variety of sensors including cameras, thermal cameras, etc.
• The signals level, which targets knowledge collected from the interception of communications.
• The measurement and signature levels, which concern knowledge, derived from analysis of collected data, for example though the use of sensors.
• The human intelligence level, which deals with knowledge that can be provided/collected from human sources.
• The public level, which targets publicly available knowledge from sources like the internet, real time and archived TV and radio news, blogs, podcasts, social networks, as well as internet news sources.
• The wealth of information, which can be derived from the above sources provide a sound basis of timely predicting and accordingly anticipating terroristic plans or activities. Nevertheless, this information is usually fragmented (and sometimes outdates) which limits its utility. Tools and techniques for timely collecting and promptly using the above information are therefore essential to its successful exploitation.
Recent advances in ICT technologies can boost the efficient acquisition, fusion and integration of information from the above sources. In particular, the proliferation of multi-purpose sensors provides ample room for sensing the physical world. For example, a host of visual processing algorithms can provide credible information about the context of a given actor (e.g. location, emotion, behaviour). At the same time Wireless Sensor Networks (WSN) provide the means for autonomic continuous information collection, in the scope of large scale heterogeneous environments. However, even with these technologies at hand, there is a need for the fusion of information captured by multiple sensors and modalities to the end of identifying situations, especially in the scope of highly distributed heterogeneous and volatile environments where people and entities (i.e. people, sensors, vehicle etc.) may dynamically join and leave. Advanced reasoning mechanisms are required in order to predict situations and incidents, while at the same time adapting security forces operations to the results of the prediction. Furthermore, tools and techniques for complex event processing can enable the proactive identification of asymmetric threats in the urban environment.
Project Context and Objectives:
The main goal of PROACTIVE is to research a holistic citizen-friendly multi sensor fusion and intelligent reasoning framework enabling the prediction, detection, understanding and efficient response to terrorist interests, goals and courses of actions in an urban environment. To this end, PROACTIVE will rely on the fusion of both static knowledge (i.e. intelligence information) and dynamic information (i.e. data observed from sensors deployed in the urban environment). The framework will be user-driven, given that the project is supported by a rich set of end-users, which are either members of the consortium or members of a special end-user advisory board.
From a technological perspective, PROACTIVE will integrate a host of novel technologies enabling the fusion of multi-sensor data with contextual information (notably 3D digital terrain data), while also resolving the ambiguities of the fusion process. Moreover, the PROACTIVE framework will incorporate advanced reasoning techniques (such as adversarial reasoning) in order to intelligently process and derive high level terroristic semantics from a multitude of source streams. The later techniques will be adapted to the terrorist domain, in order to facilitate prediction and anticipation of actions and goals of the terroristic entities.
Overall, PROACTIVE will leverage cutting-edge technologies such as the Net-centric Enable Capability (NEC) approach and the emerging “Internet-of-Things” concept, which are key enablers of new capabilities associated with real-time awareness of the physical environment, as well as with tracking and analyzing human behaviour. PROACTIVE will address the technological challenges that inhibit the wider deployment of NEC/ IoT in anti-terrorist applications.
Following the deployment and evaluation of the framework, PROACTIVE will produce a set of best practices and blueprints, which will contribute to a common EU approach to terrorist prevention in an urban environments.
PROACTIVE will (1) generate, either on-demand or in response to urban situation evolution, predictions of terroristic intention and plan the security force response; (2) observe, detect, classify and track potential threats, as well to provide the feedback elements to improve the successive predictions. In particular, PROACTIVE will perform the following functions:
• estimation of the likelihood of each of the terrorist courses of action taking place as a result of the information received during the intelligence collection phase (static prediction);
• game-playing schemes, according to which security forces and terrorists are two enemy groups attempting to maximize their respective objectives;
• planning strategies to response the to the terrorist attacks;
• risk management techniques, attempting to determine which actions of the security forces could benefit the terrorist forces;
• pattern recognition aiming at analyzing anomalies and movements of the terrorists in spatial and temporal locations with a view to identifying their most likely actions;
• object categorisation, recognition, event detection and scene understanding;
• robust localization of objects and people in the scene and their tracking (i.e. generation of trajectories) with the aim to facilitate situation awareness;
• analysis of various scenarios of urban environments to derive a set of relevant symbols and their relationships, used as concepts for describing activities;
• detection of anomalous motion patterns of objects in crowded scenes.
The prediction, simulation and planning functions are performed by a Terrorist Reasoning Kernel (TRK) at central level (Command & Control Center), while the real-time observation of the urban environment evolution for threat detection, classification and tracking is performed by a Context Awareness (CA) distributed at central and peripheral level (field units).
As information regarding situation awareness becomes available or changes, either in the game-playing mode or during the execution of the operation, the TRK generates a new or modified set of predictions, including most dangerous and most likely prediction, each characterized by its likelihood. In formulating the predictions, the TRK takes into account such factors as high-level objectives, intents and preferences of the terrorist commanders (these are either entered by the PROACTIVE user or estimated and assumed by PROACTIVE), physical capabilities and needs of the assets available, mutual influence of actions of police forces and terrorists, terrain, weather, ethical, cultural and doctrinal aspects, psychological factors affecting units and commanders, prior evolution of the operation, etc. With this input information, the TRK generates a detailed prediction (at a rate specified by the user) from the current moment, including sequences of actions, situated in time and space, performed by the terrorists. The set of predictions should be broad enough to provide the police commander with a sense of possible alternative futures, and yet small enough that it can be rapidly reviewed in real-time of urban field. In particular, the TRK should provide a suitable abstraction of each alternative prediction so that it could become a basis for displaying graphically as a rapidly simplified sketch.
The proposed framework for the CA will be based on a system of device agents that can cooperate in re-configurable groups whilst integrating into a hierarchical structure. Each device-agent (sensor or decision maker, physical, virtual or human) will have the ability to share the acquired information within his/her domain of responsibility. PROACTIVE will develop a context awareness distributed in a grid architecture as an integrating middleware infrastructure for intelligence sources, urban sensors – fixed (i.e. cameras installed in the city) - and mobile (UAVs) and control nodes. This infrastructure will incorporate a multi source fusion engine and will be designed as a novel pervasive grid environment enabling seamless, secure, on-demand access to, and aggregation of, all distributed, heterogeneous resources (i.e. sensors, computing nodes).
Figure 1. PROACTIVE concept.
Considering e.g. the instrumented policeman (equipped with individual localization device, track camera and/or IED sensor, RFID, wearable computer and head up display), he will take benefits from PROACTIVE acting as both a consumer and producer of CA within the larger operating network. For a team to perform effectively, each individual needs to have CA of all areas of the police force unit that are relevant to the individual and the team’s overall security and mission objectives. With network connectivity extending to the front-line instrumented policeman, an opportunity exists for the individual policeman to benefit from both accessing and assessing information available from the fusion grid and, at the same time, also acting as a sensor, providing information back to the grid and benefiting team-mates and the unit of Command & Control.
This “fusion grid” as proposed is a peer-to-peer architecture, propagation is needs based and it is extensible, accommodating the addition of peer nodes merely by re-configuring nearby nodes to reflect the addition of the new nodes, survivable - there is no ‘single point of failure’. Each node will be equipped with a Node Manager, which will manage the interchange of data with peers. It will contain an intelligent agent that represents the information needs and capabilities of that node, as depicted in the following figure. As the sensors or other sources on the grid generate local information, each agent will evaluate that information against the needs of the grid it represents. The coordination between the Instrumented Policeman group and UAVs, and consequently the effectiveness of the response, will be improved implementing cognitive functions and interaction schemes for the peripheral mobile grid nodes.
Figure 2. Main node types of PROACTIVE framework.
Note that the PROACTIVE “fusion grid” will provide capabilities for low-level short-term (and real-time) fusion, as well as for higher level fusion dealing with higher level semantics and operating at coarser time scales. To this end, it will provide support for fusion levels according to the popular JDL taxonomy.
The TR and CA functionalities of PROACTIVE system, will take into account the social, political and economical dynamics of the urban environment, which are likely to impact the results of the above-mentioned reasoning schemes. Special emphasis will be given to the creation, evolution and tracking of terrorist profiles, based on a combination of terrorist profiling characteristics (known in the literature), and information derived dynamically from the physical world (such as vision based behavioural analysis).
Overall, the PROACTIVE grid will be an advanced instantiation of the emerging Internet-of-Things (IoT) paradigm, which will provide enhanced CA in the urban environment along with sensor-driven analytics required by the prevention of a terrorist attack.
Project Results:
In this section we provide more details on the PROACTICE technologies and components and their relation to the demonstration scenario. Five main PROACTIVE components were developed and demonstrated in near to operational environment to show case the functionality and the efficiency of the PROACTIVE platform.
SCENE ANALYSIS WITH VISUAL INFORMATION.
The goals of the PROACTIVE scene analysis can be summarised below:
- Research algorithms for multispectral sensor networks
- Utilize multispectral sensor fusion algorithms for high level scene analysis
- Develop real-time framework for multiple source processing and event detection
PROACTIVE employed arrays of cameras scattered across potentially large areas. Cameras captured visual data that have been combined locally as visual information and relayed globally if and when required, depending on the task at hand. Videos of captured data have been employed either real-time to identify unusual activities or offline to offer a knowledge base of video information that can be searched and analysed.
In order to implement semantic scene annotation: (a) PROACTIVE researched models based object classification and extraction methods. The learning process needs only a limited number of positive samples, (b) PROACTIVE contains a multi-object and scene categorization engine for scene annotation learned with scenario and context specific visual contents, (c) PROACTIVE support high dimension, real time visual information retrieval. A disk-based state-of-the-art search engine has been implemented. The task of this engine is the spatio-temporal correlation measurement and diarization, (d) The PROACTIVE engine applies different metrics and distance computation algorithms depending on the queried feature type. Automatic and configurable weighting functions are implemented combining several feature search results into the final list.
AD-HOC NETWORKING SOLUTIONS
Starting from Commercial-of-the-Shelf (COTS) technologies, a main objective of the work-package is to introduce a reliable, flexible an extendable network.
The Solutions aim at:
- Presenting the Network Monitoring System developed by AGH and the Automated Bootstrap for the Network Self-configuration.
- Presenting the Android-base Ad-hoc Network and demonstrating the media transmission over Multi-hops Communication.
- Drawing attention to the integration efforts carried out with other partners in PROACTIVE.
The monitoring system supports the network self-configuration process in order to provide a topology which consists of relay-nodes and a gateway to which other nodes may use to access external network (in case of this demonstration it was a Vitrociset's LAN network where C&C (Command-and-Control centre) was located).
The first part of the demo was devoted to present the automation process of wireless ad-hoc network topology setup. By means of scanning, constant quality monitoring, detection of nodes and selection of the best available frequency channel, AGH’s software can form a network topology which is capable to transport different type of content, including live video, voice, data from sensors. One of the challenges accepted by the AGH was to implement and integrate self-configuration mechanisms allowing to create a fully operating, multi-hop wireless network topology and provide backhaul networking service to all of mobile PROACTIVE's end devices: UAV, Instrumented Police Officer (IPO) or sensors (i.e. camera, smoke or explosion detector). The PROACTIVE multi-radio enabled nodes to provide a backhaul network topology which overcomes the performance limitation of a single channel ad-hoc WiFi operating mode. The key advantages of a WiFi technology in this case include rapid development, low-cost equipment and non-licensed radio spectrum utilization.
TERRORIST REASONING KERNEL
CMR, according to the other partners, has developed the PROACTIVE Terrorist Reasoning Kernel (TRK). The TRK was focused on a sub-scenario composed by two monitored environments where taken in consideration. We have three principal TRK subcomponents:
- Short Term Reasoning (STR)
- Medium Term Reasoning (MTR)
- Long Term Reasoning (LTR)
Short-term reasoning activities is performed by micro-environments and is focused on threats detection. A micro-environment is a software component devoted to real-time processing of event streams coming from a specific physical environment (e.g. a building). Each micro-environment exploits a Hidden Markov Model (HMM) micro-model to analyze the symbolic low-level events recognized by virtual sensors and to generate threat notifications. This activity is performed near real-time and it’s a function of the incoming event stream. Each micro-environment relies only on incoming events, processes them in real-time and does not rely on detailed event histories, which are subsumed by the current state of the HMM. The sensitivity of a micro-environment to incoming events is influenced by its Used Alert Level (UAL), which is updated by the medium-term reasoning activities of TRK.
The Medium-term reasoning activities performed by TRK are focused on alert levels selection and sensitivity levels of the micro-environments definition. MTR analyses information with a wider spatial and temporal scope (i.e. histories of threats generated by the micro-environments over a limited time horizon) and tunes the UAL of specific micro-environments on the basis of the current situation, of long-term predictions (Predicted Alert Levels, PAL) generated by TRK long-term reasoning activities and possible alert levels set by domain expert users. A pre-defined events history has been presented in order to highlight the MTR ability to propose the UAL updating according to:
- Histories of threats generated by the micro-environments in the previous 10 minutes
- Spatial constrains (a set of rules specific for this simple DEMO scenario)
- Manually user UAL selection
Finally, the Long-term reasoning activities LTR is focused on the generation of Predicted Criticality Levels (PCL) for specific types of micro-environments, corresponding to types of physical environments (e.g. public buildings, metro stations, and so on). LTR exploits histories of threats and the associated events on a long time horizon. Long-term prediction, in turn, relies on “global” information over a long time horizon, provided by intelligence and activities schedule (e.g. future events like “the visit of the minister”). In this way LTR captures models of similar attacks grouping instances from events (ex. GTD database) or threats (event histories). For the PROACTIVE Project The Long-term reasoning activities have analyzed a pre-stored history of events occurred in a time window of a month.
UNMANNED AERIAL VECHICLE
The functionality of the PROACTIVE Core System and then focused on the concept of the UAV, its capabilities and performance, followed by integration efforts with other partners. The Consortium has produced a video for a total length of 9 minutes and 23 seconds. To allow visitors experience the UAV in flight this video has been made, since national civil aviation authority did not allow actual flight at the location of the demo due to the close proximity to high trafficked roads.
Here the main advantages in deploying UAV are described i.e. providing unobscured view from Buildings or natural Objects, expansion of covered surveillance area, dynamic relocation for tracking. However using such Aircraft also creates a number of challenges such as a robust wireless data connection, direct control by the network operator and automated extraction of scene content.The UAV capabilities have been tested frequently using a test bed consisting of a virtual simulator and a demonstration aircraft for actual test flights. To enable the operator to utilize the flying platform without facing extensive additional workload, ways had to be found beyond conventional automation schemes for UAV guidance.Task-based Guidance shall relieve the operator from the additional burden of flight guidance. The idea of this concept is to only issue high-level commands. An on-board intelligent agent then determines the steps necessary to accomplish the given task and splits it into subtasks process-able by subsequent systems (i.e. a Flight Management System, or the Sensor Payload). Therefore, the PROACTIVE operator formulates information requests which war forwards as tasks to the UAV. These requests contain the objective (e.g. person detection, object tracking, streaming), the geographical specified region of operation (e.g. Area, Line, Point), the start- and stop-conditions and the stop behaviour, which allows the UAV to repeat the same task, wait for further orders or return to base. The Sensor & Perception Management is an additional step to simplify the handling of the airborne sensor node for the operator. The general approach is to automatically select and parametrize the best combination of sensor and data processing algorithms for a given surveillance task.
Within the PROACTIVE context the most prominent demand for video processing on aerial footage was laid on object, more precisely person tracking. To track objects a co-variance tracker has been selected. Taking advantage of the fact that this tracker requires a local search window, and the availability of positioning data, ego-motion compensation is performed by transforming the local search window. Resulting tracker outputs and raw images are provided to the proactive command and control centre using a wireless Ethernet network.
Due to the ad-hoc nature of the Proactive system combined with the specifics of urban operations, multi-hop technologies were of special interest to some partners. Here, the mesh topology of the ad-hoc network should allow any node in the network to relay traffic between any other nodes in the same network even if they are out of each other’s communication range i.e. to relay traffic between the UAV and the destination node. This mesh includes a wired network connected to the wireless domain via a gateway node. The range of this wireless network is extended by relay nodes. Mobile nodes for instance an android smartphone or an aircraft provide sensor data, but can also serve as relay nodes. Thus, all wireless nodes allow multi-hopping creating a wireless network mesh. Other network technologies such as self-configuration or quality of service are also implemented to enhance the robustness of the network
COMMON OPERATIONAL PICTURE, DATA FUSION ENGINE AND INSTRUMENTED MOBILE NODE
The Presentation Layer of the PROACTIVE Project is a Graphical Unit Interface completely developed by Vitrociset in the frame of the PROACTIVE Project and it is called C2HMI-Command and Control Human Machine Interface. The C2HMI is providing to the Decision Makers an integrated Common Operation Picture with an augmented situational awareness which is supporting and facilitating the operational management of the events.The adoption of a web based architecture makes the front –end platform usability more flexible to both internal and external usage from the Monitoring Centre and makes the solution portable on mobile devices. The solution is based on last generation open source technologies and uses the last HTML5 specifications, it implements the communication paradigm defined in the web socket recommendation and is compatible with the geographical services defined by the Open Geospatial Consortium (OGC). During the final demonstrations of the PROACTIVE Project the following features of the front-end platform have been demonstrated:
• Open Platform - the proposed platform has been integrated with the TRK and with the guidance UAV module. Moreover the platform can be integrated also with other monitoring systems.
• Modularity and Flexibility - it integrates the multiple display management from different technologies
• Common Operational Picture - One of the main component of the front-end is the COP, through which the aggregation of integrated information is displayed and presented to the decision makers. On the COP the User have the opportunity to localize mobile and fixed nodes, to visualize alarms and specific events, to visualize the video streaming form deployed sensors, etc.
• Web Browser Based - portability and easy access from everywhere.
• Information Management Features - It facilitates the real time info sharing among actors involved in the same event
• Authentication and Profiling - Different Users can access the platform with different rights.
• Command and Control Features - Integrated UAV control and command interface
• 3D Presentation - 3D Map Module implemented
DATA FUSION ENGINE
The participation in many R&D European projects as coordinator and WP leader in the security topic, both in the civilian and military field, has led Vitrociset to industrialize its know-how in the data fusion field. Starting from JDL and after a deep analysis of CEP-enabled COTS products, Vitrociset has begun to develop its own CEP system named MuSES2, Multi Sensor Expert Stochastic System.
This system makes massive usage of open standards for the definition of the data models and the processing classes. Sensor Web Enablement (SWE) Common Data Model is used for the definition of the data model structure at every level of JDL, and Observations & Measurements (O&M) for sensor observations definition. This allows the system not only to manage information in an efficient way, but also allows interoperability amongst all the systems compliant to the standard.
MuSES2 comprises an advanced Situation Data Model, whose aim is to describe and to represent the objects of the real world and the relations in between. SWE is employed here as the basis structure for the data modeling.
The processing model used in MuSES2 is based on SensorML, another OGC standard, that offers a complete representation and classification of several types of data processors. Particular attention is given to the systematic composition of processors that allows to create new complex data processors from a set of more simpler ones.
The employment of a standard model both for representing data and the processing cycle enables the user to focus only on algorithms, having all the non-functional aspects of the fusion efficiently and automatically managed by the platform.
MuSES2 is PROACTIVE is integrated with the following data processing frameworks:
• a real-time expert system capable to evaluate spatial-temporal rules expressed in a human readable language;
• a machine learning library which can generate inductive models for classification;
• a library of complex stochastic algorithms.
Moreover, the system offers classic CEP design patterns, enabling a simple implementation of ad-hoc algorithms.
MuSES2 architecture resides on GSN as sensor middleware. GSN manages many non-functional requirements, such as the reliability, persistence, load distribution, etc.
PostGIS/PostgreSQL as persistence engine gives to the platform high performance and reliability, in addition to natively geometric and geographic capabilities.
Moreover, the system provides standard interfaces for exchanging the data collected and inferred. Specifically, the integration of GeoServer enables the usage of WMS, WFS, WCS, and SOS for data provisioning and the RabbitMQ messaging service integration adds an asynchronous Publish&Subscribe interface.
An additional characteristic of MuSES2 is the user-friendly interface for the configuration of all the components. In fact, a powerful Web consol enables the user to define:
• Parsing of input data structures;
• Transformation of the input raw data into SWE Common Data Model;
• Input and output structures of data processors;
• Processing modules of data processors;
• The spatial-temporal rules of the expert system;
• Complex functions and ad-hoc algorithms.
INSTRUMENTED MOBILE NODE
The Mobile Smart Station (MoSS) is the software interface of the IP (Instrumented Policeman). The main objective of the MoSS is to create a contact and communication channel with the other instrumented policemen and the Central Monitoring System (CMS). The MoSS has been installed on 3 Android devices with at least Android OS 4.0 (Ice Cream Sandwich) for the demo purposes.
The MoSS nodes were fully integrated in the networking layer and the MoSS were capable of providing detailed information to the C2HMI from the operational field.
The app mobile allows the device to display 2D map with annotation related to the position of all first responders included itself. This is possible because the app mobile send telemetries to all devices and to Proactive system. Also the position of all first responders can be displayed into the COP C2HMI 2D and 3D presentation maps.
The app mobile is entirely configurable, it can provide a set of interactions between first responders and COP system operators.
The interactions included :
• Messages
- Text message
- Text message + Photo
- Text Message + Movie
• Configurable Event Set
- Events configured
- Events configured + Photo
- Events configured + Movie
• Real Time Streaming
- Streaming with COP
- Streaming with First Responders
- Streaming with All
• Telemetry Sensors
- Localization
- Accelerometer
- Gyroscope
- Magnetic Field
- Temperature
- Humidity
- Light
Potential Impact:
Technological Impact:
It is rather evident how urban environment characterized by unpredictable terroristic threats requires a situational awareness and decision making system that has to be flexible and pervasive enough to be automatically reconfigured and quickly redeployed.
This project is aimed toward the exploitation of technological concepts and approaches already been proposed for the military operational fields toward the use on (civil) security. Those concepts, and in particular the Network Enabled Capabilities (NEC), have practically never been applied in civilian applications, and even in the military sectors are new. At a global level however there are already focus groups questioning on how to apply the NEC philosophy to the security applications (especially in the USA), thus is imperative to start the development of an European blueprint on this topic. One of the technological impacts of PROACTIVE is the generic Internet-of-Things approach towards creating a civil security C2 situation assessment and decision aiding framework.
Sensor networks oriented to civil security behaviour analysis have traditionally been aimed towards creating a connected closed circuit survey system, surveyed in a central C2 command post. Algorithmical tools for (semi-)automatic behaviour analysis and situation assessment aiding are in the focus of future research, since such augmented systems would greatly help terrorism prevention situation analysis among the large amounts of available data collected by sensor networks. Specifically, the Context Awareness Module (CAM) and the Terrorist reasoning Kernel (TRK) module of the proposed framework will help in aiding C2 decision making by providing functionalities dealing with prediction, detection, classification and tracking of possible threat entities and scenarios by entity/activity evolution analysis and contextual prediction. One of the technological impacts of the proposed system is the concentration on human/vehicle behaviour analysis in an open urban environment, specifically targeting civil applications.
It must be stressed that the proposed project's results and its outcomes are not tailored toward any urban scenario in particular, as we believe that the proposed solution can represent an scalable, affordable and easy methodology to deploy system to respond to terroristic threats that should be applicable to a broad range of urban scenarios with minimal or practically no effort. Moreover, in order to cope with the increased complexity, it has been necessary to include in the project rigours approaches to scenario sketching and field trials for validating and assessing the operational functionality of the prediction, simulation, detection and tracking. This is a crucial point to ensure that such systems is not a weak point in security per-se, and is innovative and has never been recommended so far. From this point of view, PROACTIVE could lead in the short-medium term to European recommendations as initial step of a standardization process.
Socio-economic benefits
Terror attacks at cities are an expensive occurrences, since they can have serious impact on the health, safety, security or economic well-being of citizens, as well as on the effective functioning of governments in the Member States. Such attacks can result in loss of life and material, the destruction of infrastructure, and the waste of thousands of man-hours. Thus, societies literally cannot afford to ignore the possibilities of terror attack at cities, and must do their utmost to limit their economic impact. PROACTIVE will do much to ameliorate the high costs of terror against cities, since it can be said almost without fail that an effective detection and alert also translates to substantial monetary savings. Both direct and indirect costs can be reduced by implementing effective strategies.
Direct costs may be reduced through the limitation of terror impact by improving the effective readiness and Command & Control. Open platform between all the PROACTIVE elements will assure interoperability between the different components, and therefore will enable the security authorities/ agencies to select different vendors for the different PROACTIVE components, thus increasing its cost-effectiveness. PROACTIVE will allow security agents to execute more targeted actions to counter threats. Such actions will often be less disruptive to urban operations and therefore reduce disruptions of business activities at cities, avoid loss of revenue and increase the public perceptions of cities as safe environments. A reduction in medical costs due to effective eviction of the citizens from danger zones in or near the airport will provide economic savings. By avoiding disaster the cost of rebuilding destroyed parts of the urban assets will not have to be met. Sharing data between security authorities/agencies in real-time will be cost- effective as well as it will enable the potential aversion of multiple disasters across Europe, and all the associated costs. Often, however, the true costs are indirect and they occur (and can be measured) only some time after the event. Terrorist attacks may result in a loss of tourist confidence leading to a decline in economic activity and to a decline in investor confidence in the municipality/nation’s ability to protect investments. By creating a Euro-centric knowledge base focused on city security, PROACTIVE promises to move European industry to the forefront of this ever growing market. Moreover, PROACTIVE abates indirect costs stemming from the proliferation of one-shot short-term strategies implemented in an non-coordinated fashion by Local Authorities to answer to an increased societal perception of non-security. Finally, in the current global context, leading industries able to offer effective integrated city security solutions stand to gain a significant market advantage. By bringing together European based companies, PROACTIVE will help ensure that many of the revenues from this market will flow into the European economy.
Dissemination and/or exploitation of project results
In the context of the exploitation and dissemination of the results, the project were based on the following actions:
- Publications to international journals and conferences: The PROACTIVE partners, academic and industrial, pursued dissemination activities in international refereed, scientific and technical, journals (e.g. IEEE Security, IEEE Intelligent Systems, IEEE Distributed Systems, IEEE Transactions on Software Engineering, IEEE Communications Magazine, IEEE Networks). Likewise, the partners pursued dissemination activities in international, refereed conferences (e.g. IEEE Symposium on Security and Privacy, IEEE Workshops on Dependability and Security, IEEE Wireless Communications and Networking Conference, IEEE International Conference on Communications, IEEE Globecom).
- Project website: A website dedicated to the project has been designed and developed. This work amongst others required the initial content collection from all the partners in the consortium, the creation of additional content related to the project, the regular content update, based on the communication, interaction and feedback provided by the other partners.
- Project documentation: Upon the completion of the project a number of documents, papers, deliverables, technical reports, and presentations are available. The project provides an extended fact sheet about the project, as well as a 2-pages brochure and an update that presents it. The majority of the project documents have a common look, while a common template for the presentations, deliverables, reports, meeting minutes, and in general any other document related to the project is available. Finally, a remarkable general-purpose presentation of the entire project has been created.
- Projects demos: A number of demonstrations relating to PROACTIVE middleware platforms and related applications/trials has been provided and documented with video also.These demonstrations has been used to present the project in prominent events relating to interactive multimedia content, multimedia systems and solutions. Also, events relating to project application domains (e.g. security/surveillance of large scale unpredictable environments, on-line collaboration) will be pursued.
- Project dissemination beyond Europe: The innovative character of the project makes it subject to interest beyond the border of Europe. Therefore, the project, and its accompanying technologies, has been presented to various events outside Europe. Through scientific paper contributions the partners in the consortium made their achievements accessible and globally known.
List of Websites:
Mr. Marco Cosentino
Vitrociset s.p.a
Tel: 0039 0688202567
Fax: +39 06 8820 4316
E-mail: ricerca@vitrociset.it
During the last decade the world has witnessed major terrorist attacks in urban environments, such as the collapse of New York’s Twin Towers on 11th September 2001, the bombing of packed commuter trains in Madrid on 11th March 2004, as well as the London bombings in July 2005. These incidents have manifested that modern cities are very susceptible to terrorist attacks. This weakness is directly related to the special characteristics of urban areas, which comprise a number of man-made structures (buildings, infrastructure and facilities), along with variable and complex dynamics of economic, political and cultural interactions. Broadly definable as “social variables” of the urban context, these interactions need to be addressed: an example is the concepts of “community” and “familiarity”: the strength of community is familiarity. The communities of urban-based migrant populations, left unassimilated within the larger scope of established metropolitan areas, often become inward focused and ethnocentric havens for undocumented travellers, criminal activity focused on immigration/documentation, and a base for material support to terrorists. Terrorists actively seek out the familiarity of communities sympathetic to their cause to solicit funding, support, documentation, jobs, and recruits.
Alternatively, the quality of familiarity in a dense metropolitan environment can be significantly diminished and also serve terrorist operational planning and goals. Terrorists embrace the relative anonymity of big cities in all phases of their operational activities. When selecting targets, they survey facilities or venues without being noticed. In this perspective it is crucial to develop original and integrated tools improving, among others: Identification and Tracking of Terrorist Suspects in Urban Environments; Suspicious Vehicular, Individual Behaviour/Pattern Recognition; Situational Awareness.
Combating terrorists in an urban environment is a very challenging task, given the complex social inter-relationships and interactions of the various groups engaged in a terrorist incident, the high degree of uncertainty associated with the location and origin of potential attacks, the need to protect several infrastructural facilities (such as telecommunications, energy, and transportation hubs), as well as the more general cultural, political and social conditions of an urban environment. In response to these challenges, security forces, counter-terrorism units, police authorities and intelligence agencies are increasingly looking into novel processes and tools that could help them to perceive indications of terrorist attacks, while also predicting potential incidents in order to adapt their operations accordingly.
A vast amount of information related to data exchanged through novel surveillance technologies should be analyzed in order to provide counter terror measures. This is difficult, since information scales exponentially in terms of location-space, time and means (multi-source information). In order to efficiently confront the creativity, innovation and adaptability of the terrorists, security forces can leverage information, knowledge and intelligence stemming from multiple levels including:
• The technical level, which deals with the discovery of technical knowledge about the technologies employed by terrorists.
• The imaginary level, which comprises knowledge collected from a variety of sensors including cameras, thermal cameras, etc.
• The signals level, which targets knowledge collected from the interception of communications.
• The measurement and signature levels, which concern knowledge, derived from analysis of collected data, for example though the use of sensors.
• The human intelligence level, which deals with knowledge that can be provided/collected from human sources.
• The public level, which targets publicly available knowledge from sources like the internet, real time and archived TV and radio news, blogs, podcasts, social networks, as well as internet news sources.
• The wealth of information, which can be derived from the above sources provide a sound basis of timely predicting and accordingly anticipating terroristic plans or activities. Nevertheless, this information is usually fragmented (and sometimes outdates) which limits its utility. Tools and techniques for timely collecting and promptly using the above information are therefore essential to its successful exploitation.
Recent advances in ICT technologies can boost the efficient acquisition, fusion and integration of information from the above sources. In particular, the proliferation of multi-purpose sensors provides ample room for sensing the physical world. For example, a host of visual processing algorithms can provide credible information about the context of a given actor (e.g. location, emotion, behaviour). At the same time Wireless Sensor Networks (WSN) provide the means for autonomic continuous information collection, in the scope of large scale heterogeneous environments. However, even with these technologies at hand, there is a need for the fusion of information captured by multiple sensors and modalities to the end of identifying situations, especially in the scope of highly distributed heterogeneous and volatile environments where people and entities (i.e. people, sensors, vehicle etc.) may dynamically join and leave. Advanced reasoning mechanisms are required in order to predict situations and incidents, while at the same time adapting security forces operations to the results of the prediction. Furthermore, tools and techniques for complex event processing can enable the proactive identification of asymmetric threats in the urban environment.
Project Context and Objectives:
The main goal of PROACTIVE is to research a holistic citizen-friendly multi sensor fusion and intelligent reasoning framework enabling the prediction, detection, understanding and efficient response to terrorist interests, goals and courses of actions in an urban environment. To this end, PROACTIVE will rely on the fusion of both static knowledge (i.e. intelligence information) and dynamic information (i.e. data observed from sensors deployed in the urban environment). The framework will be user-driven, given that the project is supported by a rich set of end-users, which are either members of the consortium or members of a special end-user advisory board.
From a technological perspective, PROACTIVE will integrate a host of novel technologies enabling the fusion of multi-sensor data with contextual information (notably 3D digital terrain data), while also resolving the ambiguities of the fusion process. Moreover, the PROACTIVE framework will incorporate advanced reasoning techniques (such as adversarial reasoning) in order to intelligently process and derive high level terroristic semantics from a multitude of source streams. The later techniques will be adapted to the terrorist domain, in order to facilitate prediction and anticipation of actions and goals of the terroristic entities.
Overall, PROACTIVE will leverage cutting-edge technologies such as the Net-centric Enable Capability (NEC) approach and the emerging “Internet-of-Things” concept, which are key enablers of new capabilities associated with real-time awareness of the physical environment, as well as with tracking and analyzing human behaviour. PROACTIVE will address the technological challenges that inhibit the wider deployment of NEC/ IoT in anti-terrorist applications.
Following the deployment and evaluation of the framework, PROACTIVE will produce a set of best practices and blueprints, which will contribute to a common EU approach to terrorist prevention in an urban environments.
PROACTIVE will (1) generate, either on-demand or in response to urban situation evolution, predictions of terroristic intention and plan the security force response; (2) observe, detect, classify and track potential threats, as well to provide the feedback elements to improve the successive predictions. In particular, PROACTIVE will perform the following functions:
• estimation of the likelihood of each of the terrorist courses of action taking place as a result of the information received during the intelligence collection phase (static prediction);
• game-playing schemes, according to which security forces and terrorists are two enemy groups attempting to maximize their respective objectives;
• planning strategies to response the to the terrorist attacks;
• risk management techniques, attempting to determine which actions of the security forces could benefit the terrorist forces;
• pattern recognition aiming at analyzing anomalies and movements of the terrorists in spatial and temporal locations with a view to identifying their most likely actions;
• object categorisation, recognition, event detection and scene understanding;
• robust localization of objects and people in the scene and their tracking (i.e. generation of trajectories) with the aim to facilitate situation awareness;
• analysis of various scenarios of urban environments to derive a set of relevant symbols and their relationships, used as concepts for describing activities;
• detection of anomalous motion patterns of objects in crowded scenes.
The prediction, simulation and planning functions are performed by a Terrorist Reasoning Kernel (TRK) at central level (Command & Control Center), while the real-time observation of the urban environment evolution for threat detection, classification and tracking is performed by a Context Awareness (CA) distributed at central and peripheral level (field units).
As information regarding situation awareness becomes available or changes, either in the game-playing mode or during the execution of the operation, the TRK generates a new or modified set of predictions, including most dangerous and most likely prediction, each characterized by its likelihood. In formulating the predictions, the TRK takes into account such factors as high-level objectives, intents and preferences of the terrorist commanders (these are either entered by the PROACTIVE user or estimated and assumed by PROACTIVE), physical capabilities and needs of the assets available, mutual influence of actions of police forces and terrorists, terrain, weather, ethical, cultural and doctrinal aspects, psychological factors affecting units and commanders, prior evolution of the operation, etc. With this input information, the TRK generates a detailed prediction (at a rate specified by the user) from the current moment, including sequences of actions, situated in time and space, performed by the terrorists. The set of predictions should be broad enough to provide the police commander with a sense of possible alternative futures, and yet small enough that it can be rapidly reviewed in real-time of urban field. In particular, the TRK should provide a suitable abstraction of each alternative prediction so that it could become a basis for displaying graphically as a rapidly simplified sketch.
The proposed framework for the CA will be based on a system of device agents that can cooperate in re-configurable groups whilst integrating into a hierarchical structure. Each device-agent (sensor or decision maker, physical, virtual or human) will have the ability to share the acquired information within his/her domain of responsibility. PROACTIVE will develop a context awareness distributed in a grid architecture as an integrating middleware infrastructure for intelligence sources, urban sensors – fixed (i.e. cameras installed in the city) - and mobile (UAVs) and control nodes. This infrastructure will incorporate a multi source fusion engine and will be designed as a novel pervasive grid environment enabling seamless, secure, on-demand access to, and aggregation of, all distributed, heterogeneous resources (i.e. sensors, computing nodes).
Figure 1. PROACTIVE concept.
Considering e.g. the instrumented policeman (equipped with individual localization device, track camera and/or IED sensor, RFID, wearable computer and head up display), he will take benefits from PROACTIVE acting as both a consumer and producer of CA within the larger operating network. For a team to perform effectively, each individual needs to have CA of all areas of the police force unit that are relevant to the individual and the team’s overall security and mission objectives. With network connectivity extending to the front-line instrumented policeman, an opportunity exists for the individual policeman to benefit from both accessing and assessing information available from the fusion grid and, at the same time, also acting as a sensor, providing information back to the grid and benefiting team-mates and the unit of Command & Control.
This “fusion grid” as proposed is a peer-to-peer architecture, propagation is needs based and it is extensible, accommodating the addition of peer nodes merely by re-configuring nearby nodes to reflect the addition of the new nodes, survivable - there is no ‘single point of failure’. Each node will be equipped with a Node Manager, which will manage the interchange of data with peers. It will contain an intelligent agent that represents the information needs and capabilities of that node, as depicted in the following figure. As the sensors or other sources on the grid generate local information, each agent will evaluate that information against the needs of the grid it represents. The coordination between the Instrumented Policeman group and UAVs, and consequently the effectiveness of the response, will be improved implementing cognitive functions and interaction schemes for the peripheral mobile grid nodes.
Figure 2. Main node types of PROACTIVE framework.
Note that the PROACTIVE “fusion grid” will provide capabilities for low-level short-term (and real-time) fusion, as well as for higher level fusion dealing with higher level semantics and operating at coarser time scales. To this end, it will provide support for fusion levels according to the popular JDL taxonomy.
The TR and CA functionalities of PROACTIVE system, will take into account the social, political and economical dynamics of the urban environment, which are likely to impact the results of the above-mentioned reasoning schemes. Special emphasis will be given to the creation, evolution and tracking of terrorist profiles, based on a combination of terrorist profiling characteristics (known in the literature), and information derived dynamically from the physical world (such as vision based behavioural analysis).
Overall, the PROACTIVE grid will be an advanced instantiation of the emerging Internet-of-Things (IoT) paradigm, which will provide enhanced CA in the urban environment along with sensor-driven analytics required by the prevention of a terrorist attack.
Project Results:
In this section we provide more details on the PROACTICE technologies and components and their relation to the demonstration scenario. Five main PROACTIVE components were developed and demonstrated in near to operational environment to show case the functionality and the efficiency of the PROACTIVE platform.
SCENE ANALYSIS WITH VISUAL INFORMATION.
The goals of the PROACTIVE scene analysis can be summarised below:
- Research algorithms for multispectral sensor networks
- Utilize multispectral sensor fusion algorithms for high level scene analysis
- Develop real-time framework for multiple source processing and event detection
PROACTIVE employed arrays of cameras scattered across potentially large areas. Cameras captured visual data that have been combined locally as visual information and relayed globally if and when required, depending on the task at hand. Videos of captured data have been employed either real-time to identify unusual activities or offline to offer a knowledge base of video information that can be searched and analysed.
In order to implement semantic scene annotation: (a) PROACTIVE researched models based object classification and extraction methods. The learning process needs only a limited number of positive samples, (b) PROACTIVE contains a multi-object and scene categorization engine for scene annotation learned with scenario and context specific visual contents, (c) PROACTIVE support high dimension, real time visual information retrieval. A disk-based state-of-the-art search engine has been implemented. The task of this engine is the spatio-temporal correlation measurement and diarization, (d) The PROACTIVE engine applies different metrics and distance computation algorithms depending on the queried feature type. Automatic and configurable weighting functions are implemented combining several feature search results into the final list.
AD-HOC NETWORKING SOLUTIONS
Starting from Commercial-of-the-Shelf (COTS) technologies, a main objective of the work-package is to introduce a reliable, flexible an extendable network.
The Solutions aim at:
- Presenting the Network Monitoring System developed by AGH and the Automated Bootstrap for the Network Self-configuration.
- Presenting the Android-base Ad-hoc Network and demonstrating the media transmission over Multi-hops Communication.
- Drawing attention to the integration efforts carried out with other partners in PROACTIVE.
The monitoring system supports the network self-configuration process in order to provide a topology which consists of relay-nodes and a gateway to which other nodes may use to access external network (in case of this demonstration it was a Vitrociset's LAN network where C&C (Command-and-Control centre) was located).
The first part of the demo was devoted to present the automation process of wireless ad-hoc network topology setup. By means of scanning, constant quality monitoring, detection of nodes and selection of the best available frequency channel, AGH’s software can form a network topology which is capable to transport different type of content, including live video, voice, data from sensors. One of the challenges accepted by the AGH was to implement and integrate self-configuration mechanisms allowing to create a fully operating, multi-hop wireless network topology and provide backhaul networking service to all of mobile PROACTIVE's end devices: UAV, Instrumented Police Officer (IPO) or sensors (i.e. camera, smoke or explosion detector). The PROACTIVE multi-radio enabled nodes to provide a backhaul network topology which overcomes the performance limitation of a single channel ad-hoc WiFi operating mode. The key advantages of a WiFi technology in this case include rapid development, low-cost equipment and non-licensed radio spectrum utilization.
TERRORIST REASONING KERNEL
CMR, according to the other partners, has developed the PROACTIVE Terrorist Reasoning Kernel (TRK). The TRK was focused on a sub-scenario composed by two monitored environments where taken in consideration. We have three principal TRK subcomponents:
- Short Term Reasoning (STR)
- Medium Term Reasoning (MTR)
- Long Term Reasoning (LTR)
Short-term reasoning activities is performed by micro-environments and is focused on threats detection. A micro-environment is a software component devoted to real-time processing of event streams coming from a specific physical environment (e.g. a building). Each micro-environment exploits a Hidden Markov Model (HMM) micro-model to analyze the symbolic low-level events recognized by virtual sensors and to generate threat notifications. This activity is performed near real-time and it’s a function of the incoming event stream. Each micro-environment relies only on incoming events, processes them in real-time and does not rely on detailed event histories, which are subsumed by the current state of the HMM. The sensitivity of a micro-environment to incoming events is influenced by its Used Alert Level (UAL), which is updated by the medium-term reasoning activities of TRK.
The Medium-term reasoning activities performed by TRK are focused on alert levels selection and sensitivity levels of the micro-environments definition. MTR analyses information with a wider spatial and temporal scope (i.e. histories of threats generated by the micro-environments over a limited time horizon) and tunes the UAL of specific micro-environments on the basis of the current situation, of long-term predictions (Predicted Alert Levels, PAL) generated by TRK long-term reasoning activities and possible alert levels set by domain expert users. A pre-defined events history has been presented in order to highlight the MTR ability to propose the UAL updating according to:
- Histories of threats generated by the micro-environments in the previous 10 minutes
- Spatial constrains (a set of rules specific for this simple DEMO scenario)
- Manually user UAL selection
Finally, the Long-term reasoning activities LTR is focused on the generation of Predicted Criticality Levels (PCL) for specific types of micro-environments, corresponding to types of physical environments (e.g. public buildings, metro stations, and so on). LTR exploits histories of threats and the associated events on a long time horizon. Long-term prediction, in turn, relies on “global” information over a long time horizon, provided by intelligence and activities schedule (e.g. future events like “the visit of the minister”). In this way LTR captures models of similar attacks grouping instances from events (ex. GTD database) or threats (event histories). For the PROACTIVE Project The Long-term reasoning activities have analyzed a pre-stored history of events occurred in a time window of a month.
UNMANNED AERIAL VECHICLE
The functionality of the PROACTIVE Core System and then focused on the concept of the UAV, its capabilities and performance, followed by integration efforts with other partners. The Consortium has produced a video for a total length of 9 minutes and 23 seconds. To allow visitors experience the UAV in flight this video has been made, since national civil aviation authority did not allow actual flight at the location of the demo due to the close proximity to high trafficked roads.
Here the main advantages in deploying UAV are described i.e. providing unobscured view from Buildings or natural Objects, expansion of covered surveillance area, dynamic relocation for tracking. However using such Aircraft also creates a number of challenges such as a robust wireless data connection, direct control by the network operator and automated extraction of scene content.The UAV capabilities have been tested frequently using a test bed consisting of a virtual simulator and a demonstration aircraft for actual test flights. To enable the operator to utilize the flying platform without facing extensive additional workload, ways had to be found beyond conventional automation schemes for UAV guidance.Task-based Guidance shall relieve the operator from the additional burden of flight guidance. The idea of this concept is to only issue high-level commands. An on-board intelligent agent then determines the steps necessary to accomplish the given task and splits it into subtasks process-able by subsequent systems (i.e. a Flight Management System, or the Sensor Payload). Therefore, the PROACTIVE operator formulates information requests which war forwards as tasks to the UAV. These requests contain the objective (e.g. person detection, object tracking, streaming), the geographical specified region of operation (e.g. Area, Line, Point), the start- and stop-conditions and the stop behaviour, which allows the UAV to repeat the same task, wait for further orders or return to base. The Sensor & Perception Management is an additional step to simplify the handling of the airborne sensor node for the operator. The general approach is to automatically select and parametrize the best combination of sensor and data processing algorithms for a given surveillance task.
Within the PROACTIVE context the most prominent demand for video processing on aerial footage was laid on object, more precisely person tracking. To track objects a co-variance tracker has been selected. Taking advantage of the fact that this tracker requires a local search window, and the availability of positioning data, ego-motion compensation is performed by transforming the local search window. Resulting tracker outputs and raw images are provided to the proactive command and control centre using a wireless Ethernet network.
Due to the ad-hoc nature of the Proactive system combined with the specifics of urban operations, multi-hop technologies were of special interest to some partners. Here, the mesh topology of the ad-hoc network should allow any node in the network to relay traffic between any other nodes in the same network even if they are out of each other’s communication range i.e. to relay traffic between the UAV and the destination node. This mesh includes a wired network connected to the wireless domain via a gateway node. The range of this wireless network is extended by relay nodes. Mobile nodes for instance an android smartphone or an aircraft provide sensor data, but can also serve as relay nodes. Thus, all wireless nodes allow multi-hopping creating a wireless network mesh. Other network technologies such as self-configuration or quality of service are also implemented to enhance the robustness of the network
COMMON OPERATIONAL PICTURE, DATA FUSION ENGINE AND INSTRUMENTED MOBILE NODE
The Presentation Layer of the PROACTIVE Project is a Graphical Unit Interface completely developed by Vitrociset in the frame of the PROACTIVE Project and it is called C2HMI-Command and Control Human Machine Interface. The C2HMI is providing to the Decision Makers an integrated Common Operation Picture with an augmented situational awareness which is supporting and facilitating the operational management of the events.The adoption of a web based architecture makes the front –end platform usability more flexible to both internal and external usage from the Monitoring Centre and makes the solution portable on mobile devices. The solution is based on last generation open source technologies and uses the last HTML5 specifications, it implements the communication paradigm defined in the web socket recommendation and is compatible with the geographical services defined by the Open Geospatial Consortium (OGC). During the final demonstrations of the PROACTIVE Project the following features of the front-end platform have been demonstrated:
• Open Platform - the proposed platform has been integrated with the TRK and with the guidance UAV module. Moreover the platform can be integrated also with other monitoring systems.
• Modularity and Flexibility - it integrates the multiple display management from different technologies
• Common Operational Picture - One of the main component of the front-end is the COP, through which the aggregation of integrated information is displayed and presented to the decision makers. On the COP the User have the opportunity to localize mobile and fixed nodes, to visualize alarms and specific events, to visualize the video streaming form deployed sensors, etc.
• Web Browser Based - portability and easy access from everywhere.
• Information Management Features - It facilitates the real time info sharing among actors involved in the same event
• Authentication and Profiling - Different Users can access the platform with different rights.
• Command and Control Features - Integrated UAV control and command interface
• 3D Presentation - 3D Map Module implemented
DATA FUSION ENGINE
The participation in many R&D European projects as coordinator and WP leader in the security topic, both in the civilian and military field, has led Vitrociset to industrialize its know-how in the data fusion field. Starting from JDL and after a deep analysis of CEP-enabled COTS products, Vitrociset has begun to develop its own CEP system named MuSES2, Multi Sensor Expert Stochastic System.
This system makes massive usage of open standards for the definition of the data models and the processing classes. Sensor Web Enablement (SWE) Common Data Model is used for the definition of the data model structure at every level of JDL, and Observations & Measurements (O&M) for sensor observations definition. This allows the system not only to manage information in an efficient way, but also allows interoperability amongst all the systems compliant to the standard.
MuSES2 comprises an advanced Situation Data Model, whose aim is to describe and to represent the objects of the real world and the relations in between. SWE is employed here as the basis structure for the data modeling.
The processing model used in MuSES2 is based on SensorML, another OGC standard, that offers a complete representation and classification of several types of data processors. Particular attention is given to the systematic composition of processors that allows to create new complex data processors from a set of more simpler ones.
The employment of a standard model both for representing data and the processing cycle enables the user to focus only on algorithms, having all the non-functional aspects of the fusion efficiently and automatically managed by the platform.
MuSES2 is PROACTIVE is integrated with the following data processing frameworks:
• a real-time expert system capable to evaluate spatial-temporal rules expressed in a human readable language;
• a machine learning library which can generate inductive models for classification;
• a library of complex stochastic algorithms.
Moreover, the system offers classic CEP design patterns, enabling a simple implementation of ad-hoc algorithms.
MuSES2 architecture resides on GSN as sensor middleware. GSN manages many non-functional requirements, such as the reliability, persistence, load distribution, etc.
PostGIS/PostgreSQL as persistence engine gives to the platform high performance and reliability, in addition to natively geometric and geographic capabilities.
Moreover, the system provides standard interfaces for exchanging the data collected and inferred. Specifically, the integration of GeoServer enables the usage of WMS, WFS, WCS, and SOS for data provisioning and the RabbitMQ messaging service integration adds an asynchronous Publish&Subscribe interface.
An additional characteristic of MuSES2 is the user-friendly interface for the configuration of all the components. In fact, a powerful Web consol enables the user to define:
• Parsing of input data structures;
• Transformation of the input raw data into SWE Common Data Model;
• Input and output structures of data processors;
• Processing modules of data processors;
• The spatial-temporal rules of the expert system;
• Complex functions and ad-hoc algorithms.
INSTRUMENTED MOBILE NODE
The Mobile Smart Station (MoSS) is the software interface of the IP (Instrumented Policeman). The main objective of the MoSS is to create a contact and communication channel with the other instrumented policemen and the Central Monitoring System (CMS). The MoSS has been installed on 3 Android devices with at least Android OS 4.0 (Ice Cream Sandwich) for the demo purposes.
The MoSS nodes were fully integrated in the networking layer and the MoSS were capable of providing detailed information to the C2HMI from the operational field.
The app mobile allows the device to display 2D map with annotation related to the position of all first responders included itself. This is possible because the app mobile send telemetries to all devices and to Proactive system. Also the position of all first responders can be displayed into the COP C2HMI 2D and 3D presentation maps.
The app mobile is entirely configurable, it can provide a set of interactions between first responders and COP system operators.
The interactions included :
• Messages
- Text message
- Text message + Photo
- Text Message + Movie
• Configurable Event Set
- Events configured
- Events configured + Photo
- Events configured + Movie
• Real Time Streaming
- Streaming with COP
- Streaming with First Responders
- Streaming with All
• Telemetry Sensors
- Localization
- Accelerometer
- Gyroscope
- Magnetic Field
- Temperature
- Humidity
- Light
Potential Impact:
Technological Impact:
It is rather evident how urban environment characterized by unpredictable terroristic threats requires a situational awareness and decision making system that has to be flexible and pervasive enough to be automatically reconfigured and quickly redeployed.
This project is aimed toward the exploitation of technological concepts and approaches already been proposed for the military operational fields toward the use on (civil) security. Those concepts, and in particular the Network Enabled Capabilities (NEC), have practically never been applied in civilian applications, and even in the military sectors are new. At a global level however there are already focus groups questioning on how to apply the NEC philosophy to the security applications (especially in the USA), thus is imperative to start the development of an European blueprint on this topic. One of the technological impacts of PROACTIVE is the generic Internet-of-Things approach towards creating a civil security C2 situation assessment and decision aiding framework.
Sensor networks oriented to civil security behaviour analysis have traditionally been aimed towards creating a connected closed circuit survey system, surveyed in a central C2 command post. Algorithmical tools for (semi-)automatic behaviour analysis and situation assessment aiding are in the focus of future research, since such augmented systems would greatly help terrorism prevention situation analysis among the large amounts of available data collected by sensor networks. Specifically, the Context Awareness Module (CAM) and the Terrorist reasoning Kernel (TRK) module of the proposed framework will help in aiding C2 decision making by providing functionalities dealing with prediction, detection, classification and tracking of possible threat entities and scenarios by entity/activity evolution analysis and contextual prediction. One of the technological impacts of the proposed system is the concentration on human/vehicle behaviour analysis in an open urban environment, specifically targeting civil applications.
It must be stressed that the proposed project's results and its outcomes are not tailored toward any urban scenario in particular, as we believe that the proposed solution can represent an scalable, affordable and easy methodology to deploy system to respond to terroristic threats that should be applicable to a broad range of urban scenarios with minimal or practically no effort. Moreover, in order to cope with the increased complexity, it has been necessary to include in the project rigours approaches to scenario sketching and field trials for validating and assessing the operational functionality of the prediction, simulation, detection and tracking. This is a crucial point to ensure that such systems is not a weak point in security per-se, and is innovative and has never been recommended so far. From this point of view, PROACTIVE could lead in the short-medium term to European recommendations as initial step of a standardization process.
Socio-economic benefits
Terror attacks at cities are an expensive occurrences, since they can have serious impact on the health, safety, security or economic well-being of citizens, as well as on the effective functioning of governments in the Member States. Such attacks can result in loss of life and material, the destruction of infrastructure, and the waste of thousands of man-hours. Thus, societies literally cannot afford to ignore the possibilities of terror attack at cities, and must do their utmost to limit their economic impact. PROACTIVE will do much to ameliorate the high costs of terror against cities, since it can be said almost without fail that an effective detection and alert also translates to substantial monetary savings. Both direct and indirect costs can be reduced by implementing effective strategies.
Direct costs may be reduced through the limitation of terror impact by improving the effective readiness and Command & Control. Open platform between all the PROACTIVE elements will assure interoperability between the different components, and therefore will enable the security authorities/ agencies to select different vendors for the different PROACTIVE components, thus increasing its cost-effectiveness. PROACTIVE will allow security agents to execute more targeted actions to counter threats. Such actions will often be less disruptive to urban operations and therefore reduce disruptions of business activities at cities, avoid loss of revenue and increase the public perceptions of cities as safe environments. A reduction in medical costs due to effective eviction of the citizens from danger zones in or near the airport will provide economic savings. By avoiding disaster the cost of rebuilding destroyed parts of the urban assets will not have to be met. Sharing data between security authorities/agencies in real-time will be cost- effective as well as it will enable the potential aversion of multiple disasters across Europe, and all the associated costs. Often, however, the true costs are indirect and they occur (and can be measured) only some time after the event. Terrorist attacks may result in a loss of tourist confidence leading to a decline in economic activity and to a decline in investor confidence in the municipality/nation’s ability to protect investments. By creating a Euro-centric knowledge base focused on city security, PROACTIVE promises to move European industry to the forefront of this ever growing market. Moreover, PROACTIVE abates indirect costs stemming from the proliferation of one-shot short-term strategies implemented in an non-coordinated fashion by Local Authorities to answer to an increased societal perception of non-security. Finally, in the current global context, leading industries able to offer effective integrated city security solutions stand to gain a significant market advantage. By bringing together European based companies, PROACTIVE will help ensure that many of the revenues from this market will flow into the European economy.
Dissemination and/or exploitation of project results
In the context of the exploitation and dissemination of the results, the project were based on the following actions:
- Publications to international journals and conferences: The PROACTIVE partners, academic and industrial, pursued dissemination activities in international refereed, scientific and technical, journals (e.g. IEEE Security, IEEE Intelligent Systems, IEEE Distributed Systems, IEEE Transactions on Software Engineering, IEEE Communications Magazine, IEEE Networks). Likewise, the partners pursued dissemination activities in international, refereed conferences (e.g. IEEE Symposium on Security and Privacy, IEEE Workshops on Dependability and Security, IEEE Wireless Communications and Networking Conference, IEEE International Conference on Communications, IEEE Globecom).
- Project website: A website dedicated to the project has been designed and developed. This work amongst others required the initial content collection from all the partners in the consortium, the creation of additional content related to the project, the regular content update, based on the communication, interaction and feedback provided by the other partners.
- Project documentation: Upon the completion of the project a number of documents, papers, deliverables, technical reports, and presentations are available. The project provides an extended fact sheet about the project, as well as a 2-pages brochure and an update that presents it. The majority of the project documents have a common look, while a common template for the presentations, deliverables, reports, meeting minutes, and in general any other document related to the project is available. Finally, a remarkable general-purpose presentation of the entire project has been created.
- Projects demos: A number of demonstrations relating to PROACTIVE middleware platforms and related applications/trials has been provided and documented with video also.These demonstrations has been used to present the project in prominent events relating to interactive multimedia content, multimedia systems and solutions. Also, events relating to project application domains (e.g. security/surveillance of large scale unpredictable environments, on-line collaboration) will be pursued.
- Project dissemination beyond Europe: The innovative character of the project makes it subject to interest beyond the border of Europe. Therefore, the project, and its accompanying technologies, has been presented to various events outside Europe. Through scientific paper contributions the partners in the consortium made their achievements accessible and globally known.
List of Websites:
Mr. Marco Cosentino
Vitrociset s.p.a
Tel: 0039 0688202567
Fax: +39 06 8820 4316
E-mail: ricerca@vitrociset.it