Final Report Summary - VALCRI (Visual Analytics for Sense-making in CRiminal Intelligence analysis)
Executive Summary:
INTRODUCTION
The purpose of VALCRI was to develop the next generation criminal intelligence analysis system for European LEAs. Working closely with three European police forces, the project researched and developed at TRL-5, an integrated system of over 75 software components of advanced data processing, analytic and sense-making tools. It includes multiple applications spanning strategic intelligence analysis to tactical intelligence and individual case management.
The VALCRI system was routinely evaluated with project end-users. In the final nine months, it has been evaluated with 214 LEA officers in 50 agencies in 16 countries and 2 international LEAs (Europol and NATO Intelligence Fusion Centre). It is undergoing trials with actual data at the London Metropolitan Police, and the Pasco County Sheriff’s Department in Florida. Negotiations are also underway to purchase or licence various VALCRI technologies and non-software outcomes.
VALCRI used a cognitive engineering approach to create a human-technology team that combined advanced concepts of human reasoning and analytic discourse with machine learning and database technologies. The result has been a semi-automated human-mediated semantic knowledge extraction capability that can facilitate and improve investigative sense making and problem solving in crime analysis and criminal investigation in a high ambiguity and constantly evolving environment.
KEY DISTINGUISHING FEATURES
1. SUPPORT HOW ANALYSTS THINK, RATHER THAN WHAT ANALYSTS DO
If VALCRI were designed to mainly support what analysts do, then the system would primarily automate current tasks and workflows. Instead, by designing for how analysts think, the VALCRI system is better able to respond to the variety of sense making, reasoning and inference making and problem solving strategies presented by human analysts.
2. FACILITATE EXPERT INTUITION TO SCIENTIFIC METHOD
In many investigations, analysts are often only presented with fragments of data from which to create an understanding of the situation and to anticipate what might happen. Expert intuition is very useful in generating “hunches”, or early, plausible and tentative hypotheses. However, hunches can be error prone and subject to cognitive biases. VALCRI has designed quick ways for analysts to use the scientific method to test their hunches so that they may easily discard it if proven wrong.
VALCRI
FP7-IP-608142
6
3. HUMANS DECIDE, MACHINES DO THE HEAVY LIFTING
VALCRI has been designed so that humans and machines do what each is good at: Humans make decisions under ambiguity; machines are fast at tedious and repetitive task. So, when an analyst instructs VALCRI to “find me more reports like this ...”, the machine learning-based automation will trawl through large volumes of structured and un-structured data (e.g. free text) to retrieve, triage, collate, thematically analyse the data, and then combines and presents the reports in context of the crime problem being investigated, e.g. Comparative Case Analysis.
4. ETHICS, LEGAL AND PRIVACY BY DESIGN
In many LEA data analytics systems, once a person’s data is enmeshed in the system-data networks, that person will continue to be linked to those criminal profiles. Such profiles will be used by the system to predict membership characteristics and to set up alerts for “persons of interest”. This can lead to further stops and searches of the person, even though he may be innocent. This interferes with his private life. VALCRI advocates the need for ‘computational transparency’ as a mitigating approach: make visible the inner workings of ‘black box’ automated algorithms. A lower TRL prototype has been implemented in VALCRI to investigate how fine grain data access controls may be combined with computational transparency so that analysts and investigators are aware of the provenance of algorithm’s computed results and protect the rights of individuals.
5. UP-SKILLING OF ANALYTIC ABILITIES
VALCRI has also identified and addressed varying deficiencies in the abilities of the intelligence analysis community. Some of this have been formalised in a new Master degree level analytics training course at Aston University in Birmingham, in partnership with the West Midlands Police; and some have been formalised into commercial intelligence analysis training packages focusing on analytic reasoning.
6. RESEARCH DATA
Partner AES worked with West Midlands Police to make anonymous three years of actual police data: over 1 million crime reports including structured and un-structured data, and over 6 million ANPR records. This data includes spelling errors, duplicates, similar but different data, and so forth. This dataset has been a crucial enabler.
Project Context and Objectives:
OBJECTIVE 1: Human Issues Framework DELIVERED
(a) Ethics, Privacy, Law. Comparative analysis of law in Germany, Belgium and UK, led to specification of legal requirements in VALCRI; Evaluated impact of removing ethically sensitive data from data analysis; Developed understanding of Ethics by Design in VALCRI; Set up Ethics Working Group in West Midlands Police to assess ethical issues in criminal intelligence analysis; (b) Cognitive bias and sense making. Operationalized insight, imagination, fluidity and rigour, transparency for experimental evaluation; Evaluated visualisation designs for insight, sense-making, cognitive bias, structuring of arguments.
OBJECTIVE 2: Analyst User Interface DELIVERED, INTEGRATED, TRL-5
A suite of AUI tools based around the reasoning workspace developed to orchestrate ML and database capabilities with interaction and visualisation functions to facilitate analytic reasoning and investigative sense
VALCRI
FP7-IP-608142
7
making. The AUI tools include maps and timelines, network evolution, dispersion diagrams, and statistical process charts, with touch-enabled, multiple-coordinated views. It is designed to encourage analysts to ask questions – an important part of sense making and coping with ambiguity.
OBJECTIVE 3: Semantic search and retrieval DELIVERED, INTEGRATED, TRL-5
Semantic search capabilities include an interactive dimension-reduction tool for data exploration and sense making with the Knowledge Generation Model. ML algorithms applied to read and select appropriate texts from crime reports, show feature set and create a first draft Comparative Case Analysis table. Associative Search identifies new associations or links between criminal entities by exploiting information, criminal behavior, modus operandi, geographical and temporal proximity, and associations between unsolved crimes and offenders to generate suspects lists.
OBJECTIVE 4: Crime situation re-construction DELIVERED, INTEGRATED, TRL-5
Developed a method for visual storytelling using argumentation theory to assist with the re-construction of crime situations. Explanations comprising fragments of data can then be formulated into defensible assessments. It enables analysts to record their evolving reasoning during investigations based on inferences from data, visualisations, and can be linked to conclusions through inferential networks.
OBJECTIVE 5: Secure, scalable and distributed architecture DELIVERED, INTEGRATED, TRL-5
The security architecture is implemented through OpenPMF with a Domain Specific Language DLS to configure Attribute and Proximity Based Access Controls (ABAC, PBAC) that translates human readable security policies into machine enforceable code; PET (Privacy Enhancing Technologies) to rapidly anonymise or pseudonymise data so it can be used without compromising privacy; HALA security test-bed set up for High Assurance Logging and Audit method based on a ‘Vault’ to provide hardware separation.
OBJECTIVE 6: Anonymised dataset DELIVERED, TRL-5, NOT RELEASEABLE
Partner WMP supplied three years of actual fine-grain police data comprising over 6 million crime reports and others, and over 58 million ANPR records. Led by AES, the data was anonymised at a deep level. This dataset was used in the development of the VALCRI system. However, internal tests showed that it was possible to de-anonymise the data. For confidentiality reasons, the data will not be released to the research community.
Project Results:
OTHER RESULTS
Harvester (SPACE). A stand-alone application where police users can search and mark up interesting text in PDF documents, harvest and store in a knowledge base.
Analysts Training Courses. The VALCRI Analytic Reasoning Training Curriculum (TN 13.4) has been developed into commercial courses: i-Intel’s 3-day CPD courses in intelligence analysis have been evaluated with 123 LEA officers in 40 agencies in 13 countries; A Master-level Advanced Analyst qualification had been developed by AES, WMP, and Aston University, Birmingham.
Provenance. Recording, playback and state saving features integrated at TRL-5, with advanced analytic provenance being researched (TRL 2-3).
Dissemination and Engagement Activities. Published 119 peer-reviewed publications; VALCRI evaluated with 214 LEA officers from 50 agencies in 16 countries; demonstrated to an estimated audience of 1500 persons at 5 international scientific conferences, 1 EU project event and 5 intelligence events in 6 countries.
User trials. Installed at WMP, LPA, BFP police partner sites for evaluation with anonymised data. Installed at Pasco Sheriff’s Office, FL, and the Metropolitan Police, London, for evaluation with actual data.
Potential Impact:
1. The main outcome is an integrated multi-application criminal intelligence analysis system at TRL-5. Using a cognitive engineering approach, we implemented the concept of a joint cognitive system, demonstrating how mixed-initiative systems can be developed to enable proactive and reactive system behaviours to create a human-machine team. This creates a test-bed for further research: (i) study the impact on operational use of criminal intelligence analysis systems of how the laws and privacy regulations are implemented, (ii) advancement of the semantic search algorithms, (iii) inclusion of formal concept analysis techniques to associative search, (iv) application of hybrid AI techniques to semantic knowledge extraction, (v) investigate alternative methods for storytelling and argumentation to support work with uncertainty, ambiguity and deception, (vi) It will also create opportunities to re-factor the integration platform code to enable plug and play capability, (vii) provide an environment for police to experiment with new methods based on the new VALCRI capabilities, (viii) use behavioural markers for automatic classification of analytic reasoning activities from user interactions with the system.
2. The VALCRI system is not one single application, but a complex multi-application industrial scale system using the following technology stack: Java, Javascript, GWT and ERRAI, Docker containers, RESTful interfaces, Jena/Fuseki RDF triple store, MongoDB, SQL Postgres DB with Elasticsearch, OpenPMF and a Central Authority Service, Graylog, NLP pipeline for concept extraction, ML-based semantic search functions.
3. Training courses have been developed around the analytic reasoning research in VALCRI. These courses are in high demand. New insights about analytic reasoning and new VALRI technologies have created opportunities for new techniques to be developed. By embedding the knowledge into CPD and Master-level courses, opportunities are being created for propagating the knowledge beyond the police intelligence communities.
4. Research into legal, ethical and privacy requirements in Europe has identified key issues and translated them into system design specifications and implementation trade-offs e.g. how to show data or node in a network visualisation graph that may be confidential for security, privacy or ethical reasons?
5. Cognitive engineering research has helped us understand how analysts think. This has enabled us to design how software might facilitate the reasoning in uncertain, ambiguous and deceptive environments through designs that encourage the asking of questions.
6. Partners have implemented different methods for semantic knowledge extraction and associative search. This opens opportunities for new research e.g. computational transparency – how we make the results of black box automated analyses understandable and verifiable by users; computational steering of algorithms such as the use of sub-space clustering methods to discover low frequency but operationally significant events; use VALCRI as a test-bed for investigating hybrid intelligent technologies in a joint cognitive system approach; navigating uncertainty when using the products of such methods given ambiguous and deceptive situations.
VALCRI
FP7-IP-608142
9
7. WMP provided real data that was large and complex enough for developing real systems. The data was anonymised and used to develop the VALCRI prototype system. However, internal evaluations determined that the data could be de-anonymised due to the richness of the data contained in the un-structured text. Therefore the anonymised data cannot be released to the research community as originally planned.
8. Exploitation. A variety of IP has been produced with plans for commercial exploitation and further research. Instead of tying partners down to the usual single exploitation plan, an exploitation agreement was reached for VALCRI that freed partners to exploit the IP they owned as they wish. The 9-point agreement is based on three ideas (a) freedom to commercially exploit IP that is individually owned, (b) freedom to join another partner to create products or services that create commercial value, and (c) profits to be shared only by those who generated the profit.
9. Impact. Most significant is the independent decisions by the Metropolitan Police Service London and the Pasco County Sheriff’s Department in Florida to adopt the VALCRI system for trials with actual data. The VALCRI system was installed at both sites. They are in the process of ingesting actual data to solve actual cases. They are not members of the project consortium and are not obliged to adopt nor trial the VALCRI system.
List of Websites:
www.valcri.org
INTRODUCTION
The purpose of VALCRI was to develop the next generation criminal intelligence analysis system for European LEAs. Working closely with three European police forces, the project researched and developed at TRL-5, an integrated system of over 75 software components of advanced data processing, analytic and sense-making tools. It includes multiple applications spanning strategic intelligence analysis to tactical intelligence and individual case management.
The VALCRI system was routinely evaluated with project end-users. In the final nine months, it has been evaluated with 214 LEA officers in 50 agencies in 16 countries and 2 international LEAs (Europol and NATO Intelligence Fusion Centre). It is undergoing trials with actual data at the London Metropolitan Police, and the Pasco County Sheriff’s Department in Florida. Negotiations are also underway to purchase or licence various VALCRI technologies and non-software outcomes.
VALCRI used a cognitive engineering approach to create a human-technology team that combined advanced concepts of human reasoning and analytic discourse with machine learning and database technologies. The result has been a semi-automated human-mediated semantic knowledge extraction capability that can facilitate and improve investigative sense making and problem solving in crime analysis and criminal investigation in a high ambiguity and constantly evolving environment.
KEY DISTINGUISHING FEATURES
1. SUPPORT HOW ANALYSTS THINK, RATHER THAN WHAT ANALYSTS DO
If VALCRI were designed to mainly support what analysts do, then the system would primarily automate current tasks and workflows. Instead, by designing for how analysts think, the VALCRI system is better able to respond to the variety of sense making, reasoning and inference making and problem solving strategies presented by human analysts.
2. FACILITATE EXPERT INTUITION TO SCIENTIFIC METHOD
In many investigations, analysts are often only presented with fragments of data from which to create an understanding of the situation and to anticipate what might happen. Expert intuition is very useful in generating “hunches”, or early, plausible and tentative hypotheses. However, hunches can be error prone and subject to cognitive biases. VALCRI has designed quick ways for analysts to use the scientific method to test their hunches so that they may easily discard it if proven wrong.
VALCRI
FP7-IP-608142
6
3. HUMANS DECIDE, MACHINES DO THE HEAVY LIFTING
VALCRI has been designed so that humans and machines do what each is good at: Humans make decisions under ambiguity; machines are fast at tedious and repetitive task. So, when an analyst instructs VALCRI to “find me more reports like this ...”, the machine learning-based automation will trawl through large volumes of structured and un-structured data (e.g. free text) to retrieve, triage, collate, thematically analyse the data, and then combines and presents the reports in context of the crime problem being investigated, e.g. Comparative Case Analysis.
4. ETHICS, LEGAL AND PRIVACY BY DESIGN
In many LEA data analytics systems, once a person’s data is enmeshed in the system-data networks, that person will continue to be linked to those criminal profiles. Such profiles will be used by the system to predict membership characteristics and to set up alerts for “persons of interest”. This can lead to further stops and searches of the person, even though he may be innocent. This interferes with his private life. VALCRI advocates the need for ‘computational transparency’ as a mitigating approach: make visible the inner workings of ‘black box’ automated algorithms. A lower TRL prototype has been implemented in VALCRI to investigate how fine grain data access controls may be combined with computational transparency so that analysts and investigators are aware of the provenance of algorithm’s computed results and protect the rights of individuals.
5. UP-SKILLING OF ANALYTIC ABILITIES
VALCRI has also identified and addressed varying deficiencies in the abilities of the intelligence analysis community. Some of this have been formalised in a new Master degree level analytics training course at Aston University in Birmingham, in partnership with the West Midlands Police; and some have been formalised into commercial intelligence analysis training packages focusing on analytic reasoning.
6. RESEARCH DATA
Partner AES worked with West Midlands Police to make anonymous three years of actual police data: over 1 million crime reports including structured and un-structured data, and over 6 million ANPR records. This data includes spelling errors, duplicates, similar but different data, and so forth. This dataset has been a crucial enabler.
Project Context and Objectives:
OBJECTIVE 1: Human Issues Framework DELIVERED
(a) Ethics, Privacy, Law. Comparative analysis of law in Germany, Belgium and UK, led to specification of legal requirements in VALCRI; Evaluated impact of removing ethically sensitive data from data analysis; Developed understanding of Ethics by Design in VALCRI; Set up Ethics Working Group in West Midlands Police to assess ethical issues in criminal intelligence analysis; (b) Cognitive bias and sense making. Operationalized insight, imagination, fluidity and rigour, transparency for experimental evaluation; Evaluated visualisation designs for insight, sense-making, cognitive bias, structuring of arguments.
OBJECTIVE 2: Analyst User Interface DELIVERED, INTEGRATED, TRL-5
A suite of AUI tools based around the reasoning workspace developed to orchestrate ML and database capabilities with interaction and visualisation functions to facilitate analytic reasoning and investigative sense
VALCRI
FP7-IP-608142
7
making. The AUI tools include maps and timelines, network evolution, dispersion diagrams, and statistical process charts, with touch-enabled, multiple-coordinated views. It is designed to encourage analysts to ask questions – an important part of sense making and coping with ambiguity.
OBJECTIVE 3: Semantic search and retrieval DELIVERED, INTEGRATED, TRL-5
Semantic search capabilities include an interactive dimension-reduction tool for data exploration and sense making with the Knowledge Generation Model. ML algorithms applied to read and select appropriate texts from crime reports, show feature set and create a first draft Comparative Case Analysis table. Associative Search identifies new associations or links between criminal entities by exploiting information, criminal behavior, modus operandi, geographical and temporal proximity, and associations between unsolved crimes and offenders to generate suspects lists.
OBJECTIVE 4: Crime situation re-construction DELIVERED, INTEGRATED, TRL-5
Developed a method for visual storytelling using argumentation theory to assist with the re-construction of crime situations. Explanations comprising fragments of data can then be formulated into defensible assessments. It enables analysts to record their evolving reasoning during investigations based on inferences from data, visualisations, and can be linked to conclusions through inferential networks.
OBJECTIVE 5: Secure, scalable and distributed architecture DELIVERED, INTEGRATED, TRL-5
The security architecture is implemented through OpenPMF with a Domain Specific Language DLS to configure Attribute and Proximity Based Access Controls (ABAC, PBAC) that translates human readable security policies into machine enforceable code; PET (Privacy Enhancing Technologies) to rapidly anonymise or pseudonymise data so it can be used without compromising privacy; HALA security test-bed set up for High Assurance Logging and Audit method based on a ‘Vault’ to provide hardware separation.
OBJECTIVE 6: Anonymised dataset DELIVERED, TRL-5, NOT RELEASEABLE
Partner WMP supplied three years of actual fine-grain police data comprising over 6 million crime reports and others, and over 58 million ANPR records. Led by AES, the data was anonymised at a deep level. This dataset was used in the development of the VALCRI system. However, internal tests showed that it was possible to de-anonymise the data. For confidentiality reasons, the data will not be released to the research community.
Project Results:
OTHER RESULTS
Harvester (SPACE). A stand-alone application where police users can search and mark up interesting text in PDF documents, harvest and store in a knowledge base.
Analysts Training Courses. The VALCRI Analytic Reasoning Training Curriculum (TN 13.4) has been developed into commercial courses: i-Intel’s 3-day CPD courses in intelligence analysis have been evaluated with 123 LEA officers in 40 agencies in 13 countries; A Master-level Advanced Analyst qualification had been developed by AES, WMP, and Aston University, Birmingham.
Provenance. Recording, playback and state saving features integrated at TRL-5, with advanced analytic provenance being researched (TRL 2-3).
Dissemination and Engagement Activities. Published 119 peer-reviewed publications; VALCRI evaluated with 214 LEA officers from 50 agencies in 16 countries; demonstrated to an estimated audience of 1500 persons at 5 international scientific conferences, 1 EU project event and 5 intelligence events in 6 countries.
User trials. Installed at WMP, LPA, BFP police partner sites for evaluation with anonymised data. Installed at Pasco Sheriff’s Office, FL, and the Metropolitan Police, London, for evaluation with actual data.
Potential Impact:
1. The main outcome is an integrated multi-application criminal intelligence analysis system at TRL-5. Using a cognitive engineering approach, we implemented the concept of a joint cognitive system, demonstrating how mixed-initiative systems can be developed to enable proactive and reactive system behaviours to create a human-machine team. This creates a test-bed for further research: (i) study the impact on operational use of criminal intelligence analysis systems of how the laws and privacy regulations are implemented, (ii) advancement of the semantic search algorithms, (iii) inclusion of formal concept analysis techniques to associative search, (iv) application of hybrid AI techniques to semantic knowledge extraction, (v) investigate alternative methods for storytelling and argumentation to support work with uncertainty, ambiguity and deception, (vi) It will also create opportunities to re-factor the integration platform code to enable plug and play capability, (vii) provide an environment for police to experiment with new methods based on the new VALCRI capabilities, (viii) use behavioural markers for automatic classification of analytic reasoning activities from user interactions with the system.
2. The VALCRI system is not one single application, but a complex multi-application industrial scale system using the following technology stack: Java, Javascript, GWT and ERRAI, Docker containers, RESTful interfaces, Jena/Fuseki RDF triple store, MongoDB, SQL Postgres DB with Elasticsearch, OpenPMF and a Central Authority Service, Graylog, NLP pipeline for concept extraction, ML-based semantic search functions.
3. Training courses have been developed around the analytic reasoning research in VALCRI. These courses are in high demand. New insights about analytic reasoning and new VALRI technologies have created opportunities for new techniques to be developed. By embedding the knowledge into CPD and Master-level courses, opportunities are being created for propagating the knowledge beyond the police intelligence communities.
4. Research into legal, ethical and privacy requirements in Europe has identified key issues and translated them into system design specifications and implementation trade-offs e.g. how to show data or node in a network visualisation graph that may be confidential for security, privacy or ethical reasons?
5. Cognitive engineering research has helped us understand how analysts think. This has enabled us to design how software might facilitate the reasoning in uncertain, ambiguous and deceptive environments through designs that encourage the asking of questions.
6. Partners have implemented different methods for semantic knowledge extraction and associative search. This opens opportunities for new research e.g. computational transparency – how we make the results of black box automated analyses understandable and verifiable by users; computational steering of algorithms such as the use of sub-space clustering methods to discover low frequency but operationally significant events; use VALCRI as a test-bed for investigating hybrid intelligent technologies in a joint cognitive system approach; navigating uncertainty when using the products of such methods given ambiguous and deceptive situations.
VALCRI
FP7-IP-608142
9
7. WMP provided real data that was large and complex enough for developing real systems. The data was anonymised and used to develop the VALCRI prototype system. However, internal evaluations determined that the data could be de-anonymised due to the richness of the data contained in the un-structured text. Therefore the anonymised data cannot be released to the research community as originally planned.
8. Exploitation. A variety of IP has been produced with plans for commercial exploitation and further research. Instead of tying partners down to the usual single exploitation plan, an exploitation agreement was reached for VALCRI that freed partners to exploit the IP they owned as they wish. The 9-point agreement is based on three ideas (a) freedom to commercially exploit IP that is individually owned, (b) freedom to join another partner to create products or services that create commercial value, and (c) profits to be shared only by those who generated the profit.
9. Impact. Most significant is the independent decisions by the Metropolitan Police Service London and the Pasco County Sheriff’s Department in Florida to adopt the VALCRI system for trials with actual data. The VALCRI system was installed at both sites. They are in the process of ingesting actual data to solve actual cases. They are not members of the project consortium and are not obliged to adopt nor trial the VALCRI system.
List of Websites:
www.valcri.org
