Periodic Reporting for period 3 - DeepCube (EXPLAINABLE AI PIPELINES FOR BIG COPERNICUS DATA)
Reporting period: 2023-01-01 to 2023-12-31
DeepCube uses AI to unlock the potential of big Copernicus data. We address ambitious problems of high socio-environmental impact, aiming to enhance our understanding of Earth’s processes correlated with climate change and to generate business value by enlarging the EO-centered ecosystem. We extended mature technologies, such as the Earth System Data Cube, the semantic system Plato, the Hopsworks platform, the EO data visualization tool Sextant, and a state-of-the-art visualization tool for social media data, integrating them into a cloud-agnostic platform. We developed novel pipelines that extract value from EO data, and AI architectures that extend to non-conventional problems, opening up the DL black box through Explainable AI and Causality. We showcased these in six Use Cases which achieved the following: 1) predict localized impacts of meteorological drought and heatwaves in sub-Saharan Africa, 2) uncover causal relations between climate and displacement in Africa by mining climate anomalies and migration flows, characterizing also socio-economic and environmental indicators of migration patterns, 3) better predict the outbreak and spread of wildfires in the Mediterranean, 4) exploit the abundance of InSAR data to model ground deformation patterns of volcanic unrest, 5) create a commercial tool based on InSAR time series products to monitor infrastructure in Italy, and 6) incorporate the environmental dimension in pricing touristic packages.
Our UCs achieved the following:
-UC1 created the Earthnet2023 dataset for vegetation impact forecasting in Africa, developed a new model for vegetation drought forecasting in high resolution over Africa, and a tool to generate on-the-fly minicubes. UC1 work was published in NeurIPS 2022.
-UC2 created a displacement dataset for Somalia from socioeconomic, conflict and climate migration dimensions, with EO and non-EO data and a causal graph in Baidoa, which shows that decreases in vegetation and cattle prices cause drought displacement. We developed models that allow discovery of drivers, timelags and strength of causal links. UC2 published - among others - a paper in Nature Communications.
-UC3 created 2 datacubes for the wildfire research community from meteorological, satellite and geospatial data sources, developed models for next-day fire danger forecasting in Greece and set up a semantic cube for risk assessment. We developed a service that daily provides next-day forecasts to HFS, using explainability to uncover the drivers behind predictions. We published papers in NeurIPS 2021 & 2023, and AGU Geophysical Research Letters.
-UC4a developed the global volcanic unrest early warning service Pluto, synchronized with the COMET-LiCS InSAR portal. Pluto was published at the EGU General Assembly 2023. We created the Hephaestus manually annotated dataset of 19,919 Sentinel-1 interferograms acquired over 44 volcanoes and published it at EarthVision Workshop in CVPR 2022. UC4a work was also published in IEEE Geoscience and Remote Sensing Letters and IEEE Transactions on Geoscience and Remote Sensing.
-UC4b developed reliability map, trend variation and motion classification products for infrastructure stability monitoring, using DL methodologies applied to large PSI point cloud databases. Classification of risk is delivered based on user-defined thresholds.
-UC5 created a tourism impact evaluation engine based on a model trained with environmental and tourism data from Orange FluxVision. The load capacity of a destination can be inferred from this model. It resulted in a product currently proposed in the Murmuration catalogue.
To achieve our technological objectives, we performed the following:
-We developed the DeepCube platform, which is delivered as Infrastructure as Code, it is container and GPU ready and can utilize scalable clusters. We improved automation for scalability and we integrated technological components.
-We extended the Earth System Data Cube for on-the-fly regridding and we improved it for advanced cube analytics. A new interface was created for rechunking large data cubes in Julia and a batch shuffler was added for efficient sampling from datacubes.
-We developed the semantic cube system Plato and applied it in 3 Use Cases. We further developed PostgreSQL Foreign Data Wrappers for accessing external cubes, we created new versions of the ontologies for UC2, UC3 and UC5 and we set up demos in Sextant adding queries defined by the UC leaders.
-We collected social media data extracting location information and visual concepts, and we developed a social media API and a Web App to display the data. We collected data for the UCs, performed sentiment analysis and added Twitter analytics algorithms.
-We deployed Hopsworks and trained the DeepCube platform on its use. We upgraded it to version 2.5 which includes a new UI, and we worked towards a more Python centric Feature Store. In 2022 Hopsworks leveraged its work on semi-supervised and unsupervised ML to partner with NVIDIA.
DeepCube has 24 conference and journal publications, including 17 peer-reviewed papers. We published 10 preprints in arXiv, 8 new datasets and 4 ontologies. We maintain 2 public code repositories and we created open demos, including DeepCube platform, 4 UC demos and 5 visualization tools.
-UC1 created the Earthnet2023 dataset for vegetation impact forecasting in Africa, developed a new model for vegetation drought forecasting in high resolution over Africa, and a tool to generate on-the-fly minicubes. UC1 work was published in NeurIPS 2022.
-UC2 created a displacement dataset for Somalia from socioeconomic, conflict and climate migration dimensions, with EO and non-EO data and a causal graph in Baidoa, which shows that decreases in vegetation and cattle prices cause drought displacement. We developed models that allow discovery of drivers, timelags and strength of causal links. UC2 published - among others - a paper in Nature Communications.
-UC3 created 2 datacubes for the wildfire research community from meteorological, satellite and geospatial data sources, developed models for next-day fire danger forecasting in Greece and set up a semantic cube for risk assessment. We developed a service that daily provides next-day forecasts to HFS, using explainability to uncover the drivers behind predictions. We published papers in NeurIPS 2021 & 2023, and AGU Geophysical Research Letters.
-UC4a developed the global volcanic unrest early warning service Pluto, synchronized with the COMET-LiCS InSAR portal. Pluto was published at the EGU General Assembly 2023. We created the Hephaestus manually annotated dataset of 19,919 Sentinel-1 interferograms acquired over 44 volcanoes and published it at EarthVision Workshop in CVPR 2022. UC4a work was also published in IEEE Geoscience and Remote Sensing Letters and IEEE Transactions on Geoscience and Remote Sensing.
-UC4b developed reliability map, trend variation and motion classification products for infrastructure stability monitoring, using DL methodologies applied to large PSI point cloud databases. Classification of risk is delivered based on user-defined thresholds.
-UC5 created a tourism impact evaluation engine based on a model trained with environmental and tourism data from Orange FluxVision. The load capacity of a destination can be inferred from this model. It resulted in a product currently proposed in the Murmuration catalogue.
To achieve our technological objectives, we performed the following:
-We developed the DeepCube platform, which is delivered as Infrastructure as Code, it is container and GPU ready and can utilize scalable clusters. We improved automation for scalability and we integrated technological components.
-We extended the Earth System Data Cube for on-the-fly regridding and we improved it for advanced cube analytics. A new interface was created for rechunking large data cubes in Julia and a batch shuffler was added for efficient sampling from datacubes.
-We developed the semantic cube system Plato and applied it in 3 Use Cases. We further developed PostgreSQL Foreign Data Wrappers for accessing external cubes, we created new versions of the ontologies for UC2, UC3 and UC5 and we set up demos in Sextant adding queries defined by the UC leaders.
-We collected social media data extracting location information and visual concepts, and we developed a social media API and a Web App to display the data. We collected data for the UCs, performed sentiment analysis and added Twitter analytics algorithms.
-We deployed Hopsworks and trained the DeepCube platform on its use. We upgraded it to version 2.5 which includes a new UI, and we worked towards a more Python centric Feature Store. In 2022 Hopsworks leveraged its work on semi-supervised and unsupervised ML to partner with NVIDIA.
DeepCube has 24 conference and journal publications, including 17 peer-reviewed papers. We published 10 preprints in arXiv, 8 new datasets and 4 ontologies. We maintain 2 public code repositories and we created open demos, including DeepCube platform, 4 UC demos and 5 visualization tools.
DeepCube has achieved high impact through its UCs:
-UC1 worked with domain experts from GRC and WFP, and in October 2023 co-organized the AI for Climate Risk Mitigation Workshop at the WCRP Open Science Conference in Rwanda. It has inspired 3 follow-up works from groups at renowned scientific institutions (ETH Zürich, Uni Bern, DLR Munich).
-UC2 worked closely with iDMC, JRC KCDM, WFP Vulnerability Analysis Mapping, IDP Working Group, Danish Refugee Council, Adelphi, Potsdam Institute for Climate Research, Hugo Observatory, Institute for Peace and Security Studies, IFRC, IOM, UNHCR, and NGOs. UVEG's state of the art drought modeling report is among the top five downloads at iDMC’s website. In collaboration with iDMC, UC2 organized a workshop with the IDP Working group in Somalia to publish its findings in the GRID report.
-UC3 worked closely with HFS and EFFIS, and established collaboration with the ITU AI for Good initiative. We participated in the ITU Webinar “Fighting wildfires with AI-powered insights”, in the ITU ML Workshop "The role of AI in tackling climate change and its impacts: from science to early warning" and in the AI for Good Global Summit 2023 in Geneva.
-The system Pluto, developed in UC4a, is in the UNESCO IRCAI Global Top 100 list of AI solutions to support the achievement of the UN SDGs and was rated as excellent by the IRCAI reviewers. In November 2023, the paper “Self-Supervised Contrastive Learning for Volcanic Unrest Detection”, was featured as “IEEE GRSS Article of the Week”, occasioned by the new volcanic activity of Fagradalsfjall in Iceland.
-The collaboration with TECNE in UC4b has paved the way for the development of a novel service addressing the monitoring of major infrastructure in Italy. The service has the potential for being replicated over similar infrastructure. UC4b also collaborated with the University of Florence.
-UC5 worked with several users and data providers in Brazil, France and Slovenia, including Terra Nordeste, Indexperience, Arctur, Segittur - an organisation attached to the Spanish Ministry of industry, commerce and tourism, Orange through its FluxVision service, and CRT Occitanie.
-UC1 worked with domain experts from GRC and WFP, and in October 2023 co-organized the AI for Climate Risk Mitigation Workshop at the WCRP Open Science Conference in Rwanda. It has inspired 3 follow-up works from groups at renowned scientific institutions (ETH Zürich, Uni Bern, DLR Munich).
-UC2 worked closely with iDMC, JRC KCDM, WFP Vulnerability Analysis Mapping, IDP Working Group, Danish Refugee Council, Adelphi, Potsdam Institute for Climate Research, Hugo Observatory, Institute for Peace and Security Studies, IFRC, IOM, UNHCR, and NGOs. UVEG's state of the art drought modeling report is among the top five downloads at iDMC’s website. In collaboration with iDMC, UC2 organized a workshop with the IDP Working group in Somalia to publish its findings in the GRID report.
-UC3 worked closely with HFS and EFFIS, and established collaboration with the ITU AI for Good initiative. We participated in the ITU Webinar “Fighting wildfires with AI-powered insights”, in the ITU ML Workshop "The role of AI in tackling climate change and its impacts: from science to early warning" and in the AI for Good Global Summit 2023 in Geneva.
-The system Pluto, developed in UC4a, is in the UNESCO IRCAI Global Top 100 list of AI solutions to support the achievement of the UN SDGs and was rated as excellent by the IRCAI reviewers. In November 2023, the paper “Self-Supervised Contrastive Learning for Volcanic Unrest Detection”, was featured as “IEEE GRSS Article of the Week”, occasioned by the new volcanic activity of Fagradalsfjall in Iceland.
-The collaboration with TECNE in UC4b has paved the way for the development of a novel service addressing the monitoring of major infrastructure in Italy. The service has the potential for being replicated over similar infrastructure. UC4b also collaborated with the University of Florence.
-UC5 worked with several users and data providers in Brazil, France and Slovenia, including Terra Nordeste, Indexperience, Arctur, Segittur - an organisation attached to the Spanish Ministry of industry, commerce and tourism, Orange through its FluxVision service, and CRT Occitanie.