Periodic Reporting for period 1 - HACID (Hybrid Human Artificial Collective Intelligence in Open-Ended Decision Making)
Reporting period: 2022-09-01 to 2024-02-29
HACID aims at harnessing the hybrid collective intelligence of human experts and AI systems to address open-ended problems—i.e. problems in which the solutions are not constrained to a (predefined, limited) set of alternatives. We aim to develop a general methodology and apply it to different application domains, namely medical diagnostics and climate services.
In medical diagnostics, the identification of a disease from a set of symptoms and results of tests may be particularly complex, as it deals with a large variety of possible diseases. It is therefore necessary to relate case descriptions to the large existing body of supporting evidence, and determine what is the most likely condition that affects the patient.
Climate services represent a relatively new area of decision-making but already supported by large formal and informal bodies of knowledge, demanding the integration of multiple knowledge domains into local contexts. The main challenges here arise from matching peculiar service requests by organisations seeking support for climate change adaptation management with the growing knowledge about climate science and the socio-economical factors that impact on the evolution of climate change.
A promising way to improve decision making in complex open-ended problems is exploiting collective intelligence (CI). HACID aims at developing a hybrid collective intelligence decision support system capable of providing support to evidence-based decision-making, and aggregating and expanding the solutions provided by multiple experts, ultimately obtaining higher efficacy and efficiency, as well as higher user satisfaction, explainability and trust. The proposed system leverages complementarities between domain expertise from humans and the AI ability of reasoning on and analyzing vast amounts of data. Using a participatory approach, HACID aims at deploying an AI system capable to deal with complex, high stakes application domains and decision-making contexts.
In medical diagnostics, the identification of a disease from a set of symptoms and results of tests may be particularly complex, as it deals with a large variety of possible diseases. It is therefore necessary to relate case descriptions to the large existing body of supporting evidence, and determine what is the most likely condition that affects the patient.
Climate services represent a relatively new area of decision-making but already supported by large formal and informal bodies of knowledge, demanding the integration of multiple knowledge domains into local contexts. The main challenges here arise from matching peculiar service requests by organisations seeking support for climate change adaptation management with the growing knowledge about climate science and the socio-economical factors that impact on the evolution of climate change.
A promising way to improve decision making in complex open-ended problems is exploiting collective intelligence (CI). HACID aims at developing a hybrid collective intelligence decision support system capable of providing support to evidence-based decision-making, and aggregating and expanding the solutions provided by multiple experts, ultimately obtaining higher efficacy and efficiency, as well as higher user satisfaction, explainability and trust. The proposed system leverages complementarities between domain expertise from humans and the AI ability of reasoning on and analyzing vast amounts of data. Using a participatory approach, HACID aims at deploying an AI system capable to deal with complex, high stakes application domains and decision-making contexts.
HACID bases the hybrid collective intelligence approach for decision support on formal knowledge about an application domain structured in a Domain Knowledge Graph (DKG). The DKG describes domain knowledge in terms of concepts and relations (e.g. the Legionnaires' disease has the Legionella as causative agent). Thanks to the DKG, it is possible to identify concepts that are relevant for a specific case, hence obtaining the so-called Case Knowledge Graph (CKG), which can be exploited to support the hybrid collective intelligence approach.
HACID has developed a DKG for medical diagnostics by repurposing SNOMED-CT, a comprehensive collection of medical terms that is used as a reference terminology worldwide for the medical domain. The work performed on SNOMED-CT enables a richer representation of the information by making explicit the semantic information included in SNOMED-CT as concept attributes. Additionally, additional data is linked from available databases (e.g. abbreviations) and possibly also open-source repositories (e.g. Wikipedia). Finally, methodologies have been developed for linking supporting evidence from scientific and grey literature, so that user can navigate the DKG and find relevant information and support for decision making.
HACID is also developing a DKG for climate change adaptation management, which represents a relevant resource for a continuously growing application domain. Differently from medical diagnostics, for climate services there is no readily available resource that encompasses all relevant concepts for climate science and service provision, hence it has been necessary to design a new model ontology in strict collaboration with climate service experts. The available datasets from intercomparison projects like CMIP6 or from repositories such as Copernicus offer invaluable sources of information for the DKG, which however need to be harmonised into the devised semantic model. Also, evidence from international reports (e.g. IPCC) needs to be linked to the DKG through automatic information extraction methods. Once these challenges have been successfully tackled, HAVID will rely on a large knowledge base on which to provide decision support for climate services.
Besides building the DKGs, HACID also developed general-purpose methods for automatic identification of relevant information about specific cases in order to obtain the CKG, as well as interfaces for knowledge exploration and semantic annotation that can help experts enrich the available information about a case directly through the KG structure. Finally, methods have been developed for the aggregation of solutions crowdsources from domain experts, exploiting the CKG as the reference resource on which to match and aggregate suggested concepts.
The research and innovation activities have been performed with a user-centric perspective, taking into account stakeholders' needs from the very beginning. User research has allowed us to surface the needs of experts in both medical diagnostics and climate services, suggesting how a decision support system can contribute to daily activities. Then, opportunities for hybrid collective intelligence have been scoped out, indicating the most promising aspects to be investigated within HACID. These have been just the first steps moved by our Participatory AI approach that promises inclusion of stakeholders needs and opinions in every part of the technology development.
HACID has developed a DKG for medical diagnostics by repurposing SNOMED-CT, a comprehensive collection of medical terms that is used as a reference terminology worldwide for the medical domain. The work performed on SNOMED-CT enables a richer representation of the information by making explicit the semantic information included in SNOMED-CT as concept attributes. Additionally, additional data is linked from available databases (e.g. abbreviations) and possibly also open-source repositories (e.g. Wikipedia). Finally, methodologies have been developed for linking supporting evidence from scientific and grey literature, so that user can navigate the DKG and find relevant information and support for decision making.
HACID is also developing a DKG for climate change adaptation management, which represents a relevant resource for a continuously growing application domain. Differently from medical diagnostics, for climate services there is no readily available resource that encompasses all relevant concepts for climate science and service provision, hence it has been necessary to design a new model ontology in strict collaboration with climate service experts. The available datasets from intercomparison projects like CMIP6 or from repositories such as Copernicus offer invaluable sources of information for the DKG, which however need to be harmonised into the devised semantic model. Also, evidence from international reports (e.g. IPCC) needs to be linked to the DKG through automatic information extraction methods. Once these challenges have been successfully tackled, HAVID will rely on a large knowledge base on which to provide decision support for climate services.
Besides building the DKGs, HACID also developed general-purpose methods for automatic identification of relevant information about specific cases in order to obtain the CKG, as well as interfaces for knowledge exploration and semantic annotation that can help experts enrich the available information about a case directly through the KG structure. Finally, methods have been developed for the aggregation of solutions crowdsources from domain experts, exploiting the CKG as the reference resource on which to match and aggregate suggested concepts.
The research and innovation activities have been performed with a user-centric perspective, taking into account stakeholders' needs from the very beginning. User research has allowed us to surface the needs of experts in both medical diagnostics and climate services, suggesting how a decision support system can contribute to daily activities. Then, opportunities for hybrid collective intelligence have been scoped out, indicating the most promising aspects to be investigated within HACID. These have been just the first steps moved by our Participatory AI approach that promises inclusion of stakeholders needs and opinions in every part of the technology development.
We have developed a baseline automated aggregation of differential diagnoses provided by human experts about a case, which demonstrates how collective intelligence can be applied to open-ended domains resulting in higher diagnostic accuracy. Our approach exploits the DKG to align solutions suggested by different medical professionals that independently attempted to solve a case. Once such an alignment is in place, it is possible to determine the most supported diagnoses, producing a collective differential diagnosis that outperforms individual users. We are currently integrating large language models (LLMs) in our pipeline, and preliminary results suggests that hybrid teams of humans and AI can lead to even better diagnoses than what human-only teams can achieve. Further improvements are expected in future wor, by integrating meta-cognitive reasoning and peer predictions, and by exploiting at large the structured information encoded in the CKG.