Periodic Reporting for period 2 - HiDALGO (HPC and Big Data Technologies for Global Systems)
Berichtszeitraum: 2020-06-01 bis 2022-02-28
• What is the problem/issue being addressed?
HiDALGO aims at progressing in the implementation of HPC tools that can be used for addressing Global Challenges. We consider as Global Challenges two kinds of problems: problems that affect a certain area, but happen in many places in the world (i.e. urban air pollution); and those that affect large areas, including several countries, cultures and societies (i.e. migration) or even the entire ‘virtual’ world (social networks).
HiDALGO researches several useful tools for running simulations in those environments. It usually requires using agent-based simulations, coupling mechanisms (since it is necessary to take into account multiple aspects) and HPDA (as it is necessary to simulate multiple times with different configurations and a later analysis of the results). Because of the complexity of the problems addressed, HiDALGO provides tools that will scale in such a way that simulations can be executed with enough detail and resolution.
• Why is it important for society?
HiDALGO addresses problems that affect every country in the world. Thanks to the solutions implemented in HiDALGO, it will be possible to know which are the hottest points in cities (with respect to air pollution) and identify the best actions to mitigate the problems. In the case of migration, countries can be prepared for the expected movements, also moving support (i.e. doctors, supplies) wherever they will be necessary. In the case of social networks, it will be possible to understand how fake news are spread, identifying actions to mitigate their effects, etc.
Additionally, due to the COVID-19 outbreak, we worked in simulations about the virus spread. Thanks to them it is possible to predict how the disease could spread in a certain area, and to identify which actions could mitigate the problem (i.e. curfews, places to wear face masks, close certain economic activities, etc…).
In general, HiDALGO takes care of multiple tools that aim at solving daily problems affecting the general society, by providing the adequate implementations and guaranteeing that it will be possible to address the problem from multiple areas and with enough ‘resolution’.
• What are the overall objectives?
HiDALGO aims at:
-Providing tools for performing simulations and large data analytics and for exploiting Artificial Intelligence to the Global Challenges domain.
-Improving the scalability of the tools to be provided as CoE, both adapting existing tools and implementing new ones.
-Demonstrating the CoE utility by implementing a few representative scenarios (Urban Air Pollution, Migration, Social Networks and COVID-19).
-Closing the gap between research communities (Global Challenges and HPC), carrying out an important set of dissemination activities and enabling training.
HiDALGO aims at progressing in the implementation of HPC tools that can be used for addressing Global Challenges. We consider as Global Challenges two kinds of problems: problems that affect a certain area, but happen in many places in the world (i.e. urban air pollution); and those that affect large areas, including several countries, cultures and societies (i.e. migration) or even the entire ‘virtual’ world (social networks).
HiDALGO researches several useful tools for running simulations in those environments. It usually requires using agent-based simulations, coupling mechanisms (since it is necessary to take into account multiple aspects) and HPDA (as it is necessary to simulate multiple times with different configurations and a later analysis of the results). Because of the complexity of the problems addressed, HiDALGO provides tools that will scale in such a way that simulations can be executed with enough detail and resolution.
• Why is it important for society?
HiDALGO addresses problems that affect every country in the world. Thanks to the solutions implemented in HiDALGO, it will be possible to know which are the hottest points in cities (with respect to air pollution) and identify the best actions to mitigate the problems. In the case of migration, countries can be prepared for the expected movements, also moving support (i.e. doctors, supplies) wherever they will be necessary. In the case of social networks, it will be possible to understand how fake news are spread, identifying actions to mitigate their effects, etc.
Additionally, due to the COVID-19 outbreak, we worked in simulations about the virus spread. Thanks to them it is possible to predict how the disease could spread in a certain area, and to identify which actions could mitigate the problem (i.e. curfews, places to wear face masks, close certain economic activities, etc…).
In general, HiDALGO takes care of multiple tools that aim at solving daily problems affecting the general society, by providing the adequate implementations and guaranteeing that it will be possible to address the problem from multiple areas and with enough ‘resolution’.
• What are the overall objectives?
HiDALGO aims at:
-Providing tools for performing simulations and large data analytics and for exploiting Artificial Intelligence to the Global Challenges domain.
-Improving the scalability of the tools to be provided as CoE, both adapting existing tools and implementing new ones.
-Demonstrating the CoE utility by implementing a few representative scenarios (Urban Air Pollution, Migration, Social Networks and COVID-19).
-Closing the gap between research communities (Global Challenges and HPC), carrying out an important set of dissemination activities and enabling training.
The work has been focused on the following points:
-Requirements collection and management.
-Define and implement the features of the pilots and the simulation tools.
-Benchmark and optimization of different tools applicable to Global Challenges, for simulation, HPDA and AI.
-Develop and improve solutions for visualization.
-Define and implement features based on AI for the pilots.
-Develop a portal for exposing HiDALGO services.
-Enable different ways of coupling in several pilots (using data files, specific interfaces and message passing).
In the case of the pilots, migration and UAP implemented more complex models and coupling (i.e. with ECMWF weather), while Social Networks implemented a new tool for networks reconstruction. Additionally, a new pilot for COVID-19 simulations was implemented, using Flee as the base, that has been parallelized as well.
The main results of the period are:
-Implementation of the three pilots (migration, urban air pollution and social networks) applied to several locations/areas + a new pilot (COVID-19).
-Analysis of scalability of applications and implementation of multiple optimizations (Flee, FACS, OpenFOAM, FluidSolver, EigHist, HPDA tools).
-Implementation of coupling mechanisms.
-Operational portal and tools available (two tools for visualization, Jupyter notebooks, matchmaking, marketplace, Q&A forum, ticketing, advanced orchestrator, wiki, data catalogue and training).
-Requirements collection and management.
-Define and implement the features of the pilots and the simulation tools.
-Benchmark and optimization of different tools applicable to Global Challenges, for simulation, HPDA and AI.
-Develop and improve solutions for visualization.
-Define and implement features based on AI for the pilots.
-Develop a portal for exposing HiDALGO services.
-Enable different ways of coupling in several pilots (using data files, specific interfaces and message passing).
In the case of the pilots, migration and UAP implemented more complex models and coupling (i.e. with ECMWF weather), while Social Networks implemented a new tool for networks reconstruction. Additionally, a new pilot for COVID-19 simulations was implemented, using Flee as the base, that has been parallelized as well.
The main results of the period are:
-Implementation of the three pilots (migration, urban air pollution and social networks) applied to several locations/areas + a new pilot (COVID-19).
-Analysis of scalability of applications and implementation of multiple optimizations (Flee, FACS, OpenFOAM, FluidSolver, EigHist, HPDA tools).
-Implementation of coupling mechanisms.
-Operational portal and tools available (two tools for visualization, Jupyter notebooks, matchmaking, marketplace, Q&A forum, ticketing, advanced orchestrator, wiki, data catalogue and training).
HiDALGO has progresses beyond the state of the art in several areas:
-Improve scalability of the tools used (i.e. Flee, EigHist, FACS, FluidSolver), also benchmarking and testing in new architectures.
-Improve the complexity of simulations for the domains addressed by the pilots, adding features and aspects to the models.
-Implementation of a new highly scalable tools for two pilots (SNs reconstruction and FACS) and one tool for CFD and model order reduction using GPUs (FluidSolver).
-Ease the usage and access of HPDA, by providing tools enabling data management with the data catalogue and adding HPDA support to the orchestrator.
-Easy access to the CoE services and the execution of simulations, hiding the complexity of HPC infrastructures and data management through a web portal.
By the end of the project, we managed to increase the scalability of the pilots by:
-Increasing the complexity of the models, adding new modules and coupling more inputs (i.e. real-time data from sensors, weather data).
-Increase the ‘resolution’ and amount of simulations (i.e. fine grain cells, many more agents, more ensemble runs for parametrization, etc…).
-Addition of AI and HPDA-based features (i.e. process information from multiple ensemble runs to extract knowledge).
Taking into account the kind of problems addressed (migration, air pollution, COVID-19 and fake news in SNs), HiDALGO has a high potential to impact positively in the society, generating more knowledge around these areas and providing solutions that could support decision makers, since they may learn which policies to apply, according to the simulations and HPDA results. It will have a positive impact in the society in terms of health, pollution, people support and even economic impact of certain decisions and policies. In fact, HiDALGO had some close collaborations with external stakeholders (Save the children, NHS, Bosch, Hospital 12 de Octubre and ENCCS, among others) that resulted in success stories for the project, and we expect further similar experiences in the mid and long term.
-Improve scalability of the tools used (i.e. Flee, EigHist, FACS, FluidSolver), also benchmarking and testing in new architectures.
-Improve the complexity of simulations for the domains addressed by the pilots, adding features and aspects to the models.
-Implementation of a new highly scalable tools for two pilots (SNs reconstruction and FACS) and one tool for CFD and model order reduction using GPUs (FluidSolver).
-Ease the usage and access of HPDA, by providing tools enabling data management with the data catalogue and adding HPDA support to the orchestrator.
-Easy access to the CoE services and the execution of simulations, hiding the complexity of HPC infrastructures and data management through a web portal.
By the end of the project, we managed to increase the scalability of the pilots by:
-Increasing the complexity of the models, adding new modules and coupling more inputs (i.e. real-time data from sensors, weather data).
-Increase the ‘resolution’ and amount of simulations (i.e. fine grain cells, many more agents, more ensemble runs for parametrization, etc…).
-Addition of AI and HPDA-based features (i.e. process information from multiple ensemble runs to extract knowledge).
Taking into account the kind of problems addressed (migration, air pollution, COVID-19 and fake news in SNs), HiDALGO has a high potential to impact positively in the society, generating more knowledge around these areas and providing solutions that could support decision makers, since they may learn which policies to apply, according to the simulations and HPDA results. It will have a positive impact in the society in terms of health, pollution, people support and even economic impact of certain decisions and policies. In fact, HiDALGO had some close collaborations with external stakeholders (Save the children, NHS, Bosch, Hospital 12 de Octubre and ENCCS, among others) that resulted in success stories for the project, and we expect further similar experiences in the mid and long term.