Final Report Summary - IPATCH (Intelligent Piracy Avoidance using Threat detection and Countermeasure Heuristics)
Executive Summary:
The first phase of the IPATCH project focussed on a detailed study of the piracy phenomenon and the ways in which the commercial shipping industry was addressing the challenges. The project created a large database of piracy incident reports by collecting information from publicly available sources (IMB, IMO, etc.) and fusing it with other data sets, such as weather conditions and maritime traffic. The data has been rationalised and structured into a format suitable for statistical and inferential analysis. Such analysis was performed to derive the characteristics of piracy attacks, with regard to place, time, method, etc., and the effectiveness of different types of countermeasures in different situations. The resulting piracy knowledgebase provides a basis for the rest of the project but it is also a useful resource for the industry in itself.
An additional study has been carried out on the countermeasures themselves. In particular, the project has performed an analysis of the legal, ethical and societal issues surrounding the use of different countermeasures. This information, along with details of the technical and operational characteristics of the countermeasures, has been compiled into a ‘manual’ intended to be another resource for the industry. The manual is available in the form of a document as well as an interactive electronic tool or ‘e-Manual’. This allows for more efficient searching of the information as well as providing a database of countermeasures information that can be used by other systems.
In the second year of the project, the focus shifted to the technical development work on the on-board situational awareness and decision support platform. Algorithms for detection and tracking, situational awareness, and threat recognition were developed and implemented. In addition, a decision support tool was created, along with a human-machine interface (HMI) to display the output of the system to the crew. All components were designed according to a common architecture and processing data model which facilitates integration of the complete system. Physical integration is achieved through an integration platform which acts as a middleware between the different modules. To support the technical development, a maritime dataset was collected consisting of data from radar, AIS and GPS sensors, and thermal and visual cameras. This has also been published for use by the academic and industry research and development communities.
The final year of the project focused around preparing for and conducting the two IPATCH field trials. The first one took place in September 2016 on board the VN Partisan, a training vessel frequently used by the French Navy. The second and final trials took place in May 2017 on board the Kamari in Corinth (Greece), a Suezmax crude oil carrier. In both trials, a number of scenarios were acted out with speedboats acting as pirate vessels, including ‘normal’ behaviour as well as simulated attacks. In the final trials, the prototype IPATCH system proved capable of detecting boats at a distance of up to 4km, of tracking them, of reasoning over the given situation and of alerting the crew of an impending attack. During the trials, a rich maritime dataset was collected that will be of value for further research and development work in the computer vision and maritime security domain.
Finally, alongside the technical work in the project, exploitation and dissemination activities have been running to feed into development, as well as ensure awareness of the project’s outputs and facilitate potential uptake. As well as a number of peer-reviewed papers being published, IPATCH has been presented at conferences, workshops and industry events worldwide.
Project Context and Objectives:
Funded by the European 7th Framework Programme, the IPATCH project addresses Security Topic SEC-2013.2.4-2: Non-military protection measures for merchant shipping against piracy.
The beginning of the 21st Century saw an unprecedented resurgence of piracy. In particular, the breakdown of the Somali government and the ensuing lack of “law and order” in the region turned the Gulf of Aden into one of the world’s most dangerous places for commercial and private vessels. Piracy is also on the rise in other regions of the world, including West Africa, South East Asia and South America.
The international community reacted to these threats with an increased military presence around the Horn of Africa. However, the immense costs of these operations as well as the large size of the area that needs to be patrolled demand that further non-military options need to be explored. Many different countermeasures are currently in use by commercial shipping, but an inappropriate use of protection measures can result in unnecessary extra cost for stakeholders and can actually place the ship and its crew at further risk. A proper lookout remains the single most effective method of ship protection where early warning of a suspicious approach or attack is assured, and where defences can be readily deployed.
In this context, the IPATCH project had three high-level objectives:
(1) To perform an in-depth analysis of piracy incidents and the legal, ethical, economic and societal implications of existing counter piracy measures.
(2) To produce well-founded recommendations to the industry in the form of a manual, extending and complementing the BMP4 document and to support the use and further development of countermeasures.
(3) To develop an on-board automated surveillance and decision support system providing early detection and classification of piracy threats and supporting the captain and crew in selecting the most appropriate countermeasures against a given piracy threat.
The analysis performed under (1) will lead to recommendations for the use of countermeasures in a range of scenarios, structured as a manual (2), and development and implementation of an on-board surveillance and threat detection system (3). The developed on-board system has been demonstrated in real-life maritime environments with simulated piracy attacks.
The IPATCH on-board system consists of three elements:
1. A sensor suite, incorporating existing surveillance capabilities of the vessel (e.g. radar, AIS) and extending and complementing them with the use of advanced visual, thermal and infra-red cameras.
2. A threat recognition system which fuses data from the various sensors and employs new detection, tracking and situational awareness algorithms to give early warning of piracy threats.
3. A decision support tool which provides real-time information to the captain and crew and helps them select the most appropriate countermeasures and best course of action to take to protect the ship from potential piracy threats as they develop.
Project Results:
Piracy knowledgebase - IPATCH has collected detailed data on piracy incidents from available maritime data sources. The data includes details on the incident, the vessel, the attacker, the location and the ensuing costs. One of the key outputs of the project is a purpose-built database which fuses the results and makes them available for analysis and consumption by other systems and users. IPATCH has also analysed the collected historical data to identify the most common threat scenarios and to quantify, rank and compare countermeasures in terms of their economic, legal, societal and ethical implications. The insight generated will be made available in reports and have been further used in the project as input to the countermeasures manual and on-board surveillance and decision support system.
Piracy manual for the industry - The results of the above analysis have been compiled in a manual to provide practical recommendations and guidance for maritime stakeholders, particularly seafarers. This manual is intended to extend and complement the IMO’s “Best Management Practices” and will support all phases of shipping operations, including crew training, voyage planning and vessel preparedness, dynamic response support for emerging threats, and post-incident reporting.
Maritime situational awareness dataset - IPATCH has collected a comprehensive set of sensor data for use in developing algorithms and evaluating system performance in detection, tracking, classification and threat recognition. This dataset will be fully annotated and documented, and published for the benefit of both the maritime and research communities. Methodologies for validating and certifying the detection system efficiency will contribute towards establishing a European reference and set of tools for the evaluation of detection and monitoring systems.
On-board situational awareness and decision support system - IPATCH has developed an on-board system for the early detection, classification and mitigation of piracy threats. This includes the use of a range of sensors (visible light cameras, thermal cameras, radar, AIS), data fusion algorithms, and the implementation of intelligent algorithms for situation analysis and threat detection. A decision support tool and human-machine interface have been developed to provide maximal situation awareness to captains and crews and to guide them in implementing mitigating countermeasures in the safest and most effective way. The on board system has been tested on to different vessels under near-operational conditions, using a range of simulated piracy attack scenarios.
Potential Impact:
The knowledge resources – databases, manual, maritime sensor dataset and on-board system architecture – will be transferred to an appropriate organization willing and able to take over the role of provider, or to make them publicly available. Suitable licenses have been identified as two of the Creative Commons licenses which ensure that the work continues to be shared and extended, and is credited properly. One license allows commercial use (for the piracy knowledgebase and countermeasures manual, to allow uptake by maritime security businesses, as well as industry organisations), and one license permits only non-commercial use (for the maritime dataset and the architecture / data model specifications).
The strategy adopted is to approach industry organisations who could hold the data after the project has finished, with a view to sharing it with their members or maritime security companies in the future. The rationale for this is that, if commercial organisations are looking for data sources in the future, they are more likely to approach industry organisations, rather than the partners in IPATCH. The project is approaching BIMCO, Intertanko and the Maritime Security Alliance but at the time of writing, no agreement has yet been reached. The main obstacle remains the issue of access rights for the data that was collected during the project.
The maritime dataset consists of sensor data corresponding to realistic and representative piracy scenarios at sea, acquired using a range of different sensors, including visible and thermal cameras, radar, AIS, navigation and environmental data. For the dataset, two exploitation routes are being followed by the IPATCH partners: 1) Publication through the International IEEE Workshop on Performance Evaluation of Tracking in Surveillance (PETS), and 2) Publication through an open data service (Mendeley Data).
To promote uptake of the on-board system architecture and data model, IPATCH makes the specifications available through an open hosting platform (such as Github) so that they can be used and adopted by as wide a range of people as possible. If the specifications are useful, they will become more and more popular and a process of de facto standardisation will occur naturally.
The commercial exploitation of the entire on-board system as an integrated solution is considered to be very challenging for a number of reasons. As a result, IPATCH partners will not pursue commercialisation of the on-board system as an integrated product targeted at shipping companies. However, the on-board system is composed of many different modules and algorithms that have been developed by different partners and each component will be exploited in its own right. In particular, it is thought that there will be great demand for the elements of the IPATCH system in the context of VTS systems, port perimeter security, and maritime domain awareness. IPATCH results can easily be integrated in new products to address capability gaps in sensing and maritime surveillance.
In the course of the project, a significant number of scientific papers have been published and representatives of the consortium have disseminated results at a wide range of events. A video was produced at the end of the project, based around the final sea trials in Greece, and is available on the project website: www.ipatchproject.eu
List of Websites:
www.ipatchproject.eu
Coordinator: Tom Cane
BMT Group Ltd
Email: tom.cane@bmtglobal.com
Phone: +44 (0) 20 8943 5544
The first phase of the IPATCH project focussed on a detailed study of the piracy phenomenon and the ways in which the commercial shipping industry was addressing the challenges. The project created a large database of piracy incident reports by collecting information from publicly available sources (IMB, IMO, etc.) and fusing it with other data sets, such as weather conditions and maritime traffic. The data has been rationalised and structured into a format suitable for statistical and inferential analysis. Such analysis was performed to derive the characteristics of piracy attacks, with regard to place, time, method, etc., and the effectiveness of different types of countermeasures in different situations. The resulting piracy knowledgebase provides a basis for the rest of the project but it is also a useful resource for the industry in itself.
An additional study has been carried out on the countermeasures themselves. In particular, the project has performed an analysis of the legal, ethical and societal issues surrounding the use of different countermeasures. This information, along with details of the technical and operational characteristics of the countermeasures, has been compiled into a ‘manual’ intended to be another resource for the industry. The manual is available in the form of a document as well as an interactive electronic tool or ‘e-Manual’. This allows for more efficient searching of the information as well as providing a database of countermeasures information that can be used by other systems.
In the second year of the project, the focus shifted to the technical development work on the on-board situational awareness and decision support platform. Algorithms for detection and tracking, situational awareness, and threat recognition were developed and implemented. In addition, a decision support tool was created, along with a human-machine interface (HMI) to display the output of the system to the crew. All components were designed according to a common architecture and processing data model which facilitates integration of the complete system. Physical integration is achieved through an integration platform which acts as a middleware between the different modules. To support the technical development, a maritime dataset was collected consisting of data from radar, AIS and GPS sensors, and thermal and visual cameras. This has also been published for use by the academic and industry research and development communities.
The final year of the project focused around preparing for and conducting the two IPATCH field trials. The first one took place in September 2016 on board the VN Partisan, a training vessel frequently used by the French Navy. The second and final trials took place in May 2017 on board the Kamari in Corinth (Greece), a Suezmax crude oil carrier. In both trials, a number of scenarios were acted out with speedboats acting as pirate vessels, including ‘normal’ behaviour as well as simulated attacks. In the final trials, the prototype IPATCH system proved capable of detecting boats at a distance of up to 4km, of tracking them, of reasoning over the given situation and of alerting the crew of an impending attack. During the trials, a rich maritime dataset was collected that will be of value for further research and development work in the computer vision and maritime security domain.
Finally, alongside the technical work in the project, exploitation and dissemination activities have been running to feed into development, as well as ensure awareness of the project’s outputs and facilitate potential uptake. As well as a number of peer-reviewed papers being published, IPATCH has been presented at conferences, workshops and industry events worldwide.
Project Context and Objectives:
Funded by the European 7th Framework Programme, the IPATCH project addresses Security Topic SEC-2013.2.4-2: Non-military protection measures for merchant shipping against piracy.
The beginning of the 21st Century saw an unprecedented resurgence of piracy. In particular, the breakdown of the Somali government and the ensuing lack of “law and order” in the region turned the Gulf of Aden into one of the world’s most dangerous places for commercial and private vessels. Piracy is also on the rise in other regions of the world, including West Africa, South East Asia and South America.
The international community reacted to these threats with an increased military presence around the Horn of Africa. However, the immense costs of these operations as well as the large size of the area that needs to be patrolled demand that further non-military options need to be explored. Many different countermeasures are currently in use by commercial shipping, but an inappropriate use of protection measures can result in unnecessary extra cost for stakeholders and can actually place the ship and its crew at further risk. A proper lookout remains the single most effective method of ship protection where early warning of a suspicious approach or attack is assured, and where defences can be readily deployed.
In this context, the IPATCH project had three high-level objectives:
(1) To perform an in-depth analysis of piracy incidents and the legal, ethical, economic and societal implications of existing counter piracy measures.
(2) To produce well-founded recommendations to the industry in the form of a manual, extending and complementing the BMP4 document and to support the use and further development of countermeasures.
(3) To develop an on-board automated surveillance and decision support system providing early detection and classification of piracy threats and supporting the captain and crew in selecting the most appropriate countermeasures against a given piracy threat.
The analysis performed under (1) will lead to recommendations for the use of countermeasures in a range of scenarios, structured as a manual (2), and development and implementation of an on-board surveillance and threat detection system (3). The developed on-board system has been demonstrated in real-life maritime environments with simulated piracy attacks.
The IPATCH on-board system consists of three elements:
1. A sensor suite, incorporating existing surveillance capabilities of the vessel (e.g. radar, AIS) and extending and complementing them with the use of advanced visual, thermal and infra-red cameras.
2. A threat recognition system which fuses data from the various sensors and employs new detection, tracking and situational awareness algorithms to give early warning of piracy threats.
3. A decision support tool which provides real-time information to the captain and crew and helps them select the most appropriate countermeasures and best course of action to take to protect the ship from potential piracy threats as they develop.
Project Results:
Piracy knowledgebase - IPATCH has collected detailed data on piracy incidents from available maritime data sources. The data includes details on the incident, the vessel, the attacker, the location and the ensuing costs. One of the key outputs of the project is a purpose-built database which fuses the results and makes them available for analysis and consumption by other systems and users. IPATCH has also analysed the collected historical data to identify the most common threat scenarios and to quantify, rank and compare countermeasures in terms of their economic, legal, societal and ethical implications. The insight generated will be made available in reports and have been further used in the project as input to the countermeasures manual and on-board surveillance and decision support system.
Piracy manual for the industry - The results of the above analysis have been compiled in a manual to provide practical recommendations and guidance for maritime stakeholders, particularly seafarers. This manual is intended to extend and complement the IMO’s “Best Management Practices” and will support all phases of shipping operations, including crew training, voyage planning and vessel preparedness, dynamic response support for emerging threats, and post-incident reporting.
Maritime situational awareness dataset - IPATCH has collected a comprehensive set of sensor data for use in developing algorithms and evaluating system performance in detection, tracking, classification and threat recognition. This dataset will be fully annotated and documented, and published for the benefit of both the maritime and research communities. Methodologies for validating and certifying the detection system efficiency will contribute towards establishing a European reference and set of tools for the evaluation of detection and monitoring systems.
On-board situational awareness and decision support system - IPATCH has developed an on-board system for the early detection, classification and mitigation of piracy threats. This includes the use of a range of sensors (visible light cameras, thermal cameras, radar, AIS), data fusion algorithms, and the implementation of intelligent algorithms for situation analysis and threat detection. A decision support tool and human-machine interface have been developed to provide maximal situation awareness to captains and crews and to guide them in implementing mitigating countermeasures in the safest and most effective way. The on board system has been tested on to different vessels under near-operational conditions, using a range of simulated piracy attack scenarios.
Potential Impact:
The knowledge resources – databases, manual, maritime sensor dataset and on-board system architecture – will be transferred to an appropriate organization willing and able to take over the role of provider, or to make them publicly available. Suitable licenses have been identified as two of the Creative Commons licenses which ensure that the work continues to be shared and extended, and is credited properly. One license allows commercial use (for the piracy knowledgebase and countermeasures manual, to allow uptake by maritime security businesses, as well as industry organisations), and one license permits only non-commercial use (for the maritime dataset and the architecture / data model specifications).
The strategy adopted is to approach industry organisations who could hold the data after the project has finished, with a view to sharing it with their members or maritime security companies in the future. The rationale for this is that, if commercial organisations are looking for data sources in the future, they are more likely to approach industry organisations, rather than the partners in IPATCH. The project is approaching BIMCO, Intertanko and the Maritime Security Alliance but at the time of writing, no agreement has yet been reached. The main obstacle remains the issue of access rights for the data that was collected during the project.
The maritime dataset consists of sensor data corresponding to realistic and representative piracy scenarios at sea, acquired using a range of different sensors, including visible and thermal cameras, radar, AIS, navigation and environmental data. For the dataset, two exploitation routes are being followed by the IPATCH partners: 1) Publication through the International IEEE Workshop on Performance Evaluation of Tracking in Surveillance (PETS), and 2) Publication through an open data service (Mendeley Data).
To promote uptake of the on-board system architecture and data model, IPATCH makes the specifications available through an open hosting platform (such as Github) so that they can be used and adopted by as wide a range of people as possible. If the specifications are useful, they will become more and more popular and a process of de facto standardisation will occur naturally.
The commercial exploitation of the entire on-board system as an integrated solution is considered to be very challenging for a number of reasons. As a result, IPATCH partners will not pursue commercialisation of the on-board system as an integrated product targeted at shipping companies. However, the on-board system is composed of many different modules and algorithms that have been developed by different partners and each component will be exploited in its own right. In particular, it is thought that there will be great demand for the elements of the IPATCH system in the context of VTS systems, port perimeter security, and maritime domain awareness. IPATCH results can easily be integrated in new products to address capability gaps in sensing and maritime surveillance.
In the course of the project, a significant number of scientific papers have been published and representatives of the consortium have disseminated results at a wide range of events. A video was produced at the end of the project, based around the final sea trials in Greece, and is available on the project website: www.ipatchproject.eu
List of Websites:
www.ipatchproject.eu
Coordinator: Tom Cane
BMT Group Ltd
Email: tom.cane@bmtglobal.com
Phone: +44 (0) 20 8943 5544