Periodic Reporting for period 2 - AUGGMED (Automated Serious Game Scenario Generator for Mixed Reality Training)
Reporting period: 2016-12-01 to 2018-05-31
Terrorism remains a real and major threat to human life and critical infrastructure in Europe. Terrorists and perpetrators of organised crime are continually inventing new and unpredictable methods to carry out attacks, which are increasingly unfamiliar to security forces and therefore difficult to predict, prepare and defend against. Currently, security forces are trained using traditional methods such as desktop exercises or live role-play. The latter type of training can be very costly and time-consuming; in addition, it can only offer a limited number of scenarios. Moreover, it is difficult to maintain uniform levels of skill, judgement and preparedness across teams.
The aim of AUGGMED is to develop a serious gaming platform to train law enforcement agencies, first responders and operators of critical infrastructure in their response to terrorist and organised crime attacks.
The AUGGMED platform enables both single ‘player’ and team-based training. It fosters the acquisition of emotional management, analytical thinking, problem solving and decision making skills. The game scenarios include reality-based simulations of operational environments, crowd behaviour and threats (including fire and explosions); it is delivered through a number of different methods including a conventional desktop computer game, immersive VR and Augmented Reality (AR) with multimodal interfaces.
The final AUGGMED platform delivers highly realistic training scenarios and advanced interactivity while encouraging infrastructure security staff and first responders to engage and actively participate in the training process.
The aim of AUGGMED is to develop a serious gaming platform to train law enforcement agencies, first responders and operators of critical infrastructure in their response to terrorist and organised crime attacks.
The AUGGMED platform enables both single ‘player’ and team-based training. It fosters the acquisition of emotional management, analytical thinking, problem solving and decision making skills. The game scenarios include reality-based simulations of operational environments, crowd behaviour and threats (including fire and explosions); it is delivered through a number of different methods including a conventional desktop computer game, immersive VR and Augmented Reality (AR) with multimodal interfaces.
The final AUGGMED platform delivers highly realistic training scenarios and advanced interactivity while encouraging infrastructure security staff and first responders to engage and actively participate in the training process.
AUGGMED has developped a serious gaming platform to train law enforcement agencies, first responders and operators of critical infrastructure, supporting both single ‘player’ and team-based training. The development work was based on the end user requirements collected at the beginning of the project; immersiveness and ease of use were the guiding design principles.
AUGGMED has developed and tested three versions its prototype system:
- A conventional desktop game. This was tested in the first Pilot demonstration by firearms police officers in a number of weapons attacks in a (fictitious) UK airport in 2016.
- An immersive virtual reality version. This was tested in the second Pilot (2017) using in a 3D model of Muntaner train station (Barcelona) and featuring suspicious object and post-explosion triage scenarios, aimed at railway staff and paramedics, respectively;
- An augmented reality version. This version was set in a cruise ship terminal in the Port of Piraeus, and featuring a range of post-explosion and weapons attack scenarios. The Hellenic coast guard participated as end users in the third Pilot (March 2018).
In addition, AUGGMED has developed a haptic/thermal feedback vest which is fully integrated into the VR system and is intended to intensify the sense of realism and immersion for players.
The AUGGMED system was demonstrated at a large number of conferences and workshops, including the Security and Counter Terror Expo (London, 3-4 May 2017). In addition, the project was covered on the BBC's science programme 'Click' in 2017 (https://www.youtube.com/watch?v=DlDZWFXtYmU). A number of scientific papers have been written or are in preparation, including detailed technical work on the haptic feedback vest as well as the use of virtual reality environments in first responder training.
Exploitation efforts are ongoing and will continue beyond the end of the project.
AUGGMED has developed and tested three versions its prototype system:
- A conventional desktop game. This was tested in the first Pilot demonstration by firearms police officers in a number of weapons attacks in a (fictitious) UK airport in 2016.
- An immersive virtual reality version. This was tested in the second Pilot (2017) using in a 3D model of Muntaner train station (Barcelona) and featuring suspicious object and post-explosion triage scenarios, aimed at railway staff and paramedics, respectively;
- An augmented reality version. This version was set in a cruise ship terminal in the Port of Piraeus, and featuring a range of post-explosion and weapons attack scenarios. The Hellenic coast guard participated as end users in the third Pilot (March 2018).
In addition, AUGGMED has developed a haptic/thermal feedback vest which is fully integrated into the VR system and is intended to intensify the sense of realism and immersion for players.
The AUGGMED system was demonstrated at a large number of conferences and workshops, including the Security and Counter Terror Expo (London, 3-4 May 2017). In addition, the project was covered on the BBC's science programme 'Click' in 2017 (https://www.youtube.com/watch?v=DlDZWFXtYmU). A number of scientific papers have been written or are in preparation, including detailed technical work on the haptic feedback vest as well as the use of virtual reality environments in first responder training.
Exploitation efforts are ongoing and will continue beyond the end of the project.
There are several 3D simulation tools on the market that can be used for the training of soldiers, police officers or paramedical staff. While they are capable of rendering a convincing illusion of the real world, they all share one feature: complex processes like crowd behaviour or the spreading of fire and smoke are not (or only to a very limited degree) based on realistic, physical models and instead modelled ‘by hand’ to create a mere impression of realism. For a tool that is meant to prepare first responders for real-life disasters, this is a serious short-coming as their training is reliant not upon hard evidence but ultimately on an artist’s view of reality. AUGGMED addresses this issue by linking a 3D games engine to a state-of-the-art evacuation modelling tool (EXODUS), which in turn is using powerful fire simulation software (SMARTFIRE) to realistically model the evolution of a disaster scenario. Likewise, the effects of explosions (in particular the number of casualties, their spatial distribution and the severity of their injuries) is based on the best available evidence, thus providing users with the confidence that the rendered scenarios are as close to reality as is currently possible. To the best of the consortium's knowledge, no other currently available training system provides this functionality.
Moreover, the AUGGMED serious game allows players to directly interact with crowd agents via hand gestures or spoken commands, thus allowing them to take control of a situation in a way that closely resembles a real-life scenario. In particular, players can interact with other players which enhances team work and/or collaboration with members of other organisations. The immersion of players into this type of realistically simulated environment will allow them to train their decision making skills and to experience (to a certain degree) emotional reactions that are inextricably linked to crisis scenarios.
In addition, AUGGMED makes use of a range of external devices (like VR headsets and thermal/haptic feedback vests) to enhance the experience of immersion for the players.
The AUGGMED serious game was developed in three stages: the first mode provides basic VR (in the form of a conventional desktop computer game), whereas the second and third mode add head-mounted devices and haptic vests to provide a fully immersive VR and/or AR environment.
Each of the three prototypes was tested in a Pilot demonstrator which included a wide range of potential end users: police firearms officers, railway staff, paramedics, coast guard and port police. The feedback received from the end users was very positive; participants in the pilots appreciated the realism of the simulation and considered the AUGGMED system to be a useful potential addition to their training regime.
Moreover, the AUGGMED consortium has undertaken a study comparing the effects of consolidation training in VR or in live role-plays. This study included 80 trainee police officers who acted out a number of suspicious object scenarios (either in VR or in a live role-play or a combination of the two). The analysis of the results is ongoing and will continue past the end of the project; the aim is to quantify the effects of consolidation training in VR and to investigate to which degree this approach can be a viable alternative to live role-play in first responder training.
Moreover, the AUGGMED serious game allows players to directly interact with crowd agents via hand gestures or spoken commands, thus allowing them to take control of a situation in a way that closely resembles a real-life scenario. In particular, players can interact with other players which enhances team work and/or collaboration with members of other organisations. The immersion of players into this type of realistically simulated environment will allow them to train their decision making skills and to experience (to a certain degree) emotional reactions that are inextricably linked to crisis scenarios.
In addition, AUGGMED makes use of a range of external devices (like VR headsets and thermal/haptic feedback vests) to enhance the experience of immersion for the players.
The AUGGMED serious game was developed in three stages: the first mode provides basic VR (in the form of a conventional desktop computer game), whereas the second and third mode add head-mounted devices and haptic vests to provide a fully immersive VR and/or AR environment.
Each of the three prototypes was tested in a Pilot demonstrator which included a wide range of potential end users: police firearms officers, railway staff, paramedics, coast guard and port police. The feedback received from the end users was very positive; participants in the pilots appreciated the realism of the simulation and considered the AUGGMED system to be a useful potential addition to their training regime.
Moreover, the AUGGMED consortium has undertaken a study comparing the effects of consolidation training in VR or in live role-plays. This study included 80 trainee police officers who acted out a number of suspicious object scenarios (either in VR or in a live role-play or a combination of the two). The analysis of the results is ongoing and will continue past the end of the project; the aim is to quantify the effects of consolidation training in VR and to investigate to which degree this approach can be a viable alternative to live role-play in first responder training.