Periodic Reporting for period 4 - BigEarth (Accurate and Scalable Processing of Big Data in Earth Observation)
Reporting period: 2022-10-01 to 2024-03-31
With the unprecedented advances in the satellite technology, recent years have led to a significant increase in the volume of Earth observation (EO) data archives. Thus, the development of accurate and scalable systems to discover crucial knowledge from massive EO data archives on the state of our planet Earth have recently emerged. Existing systems allow querying satellite images required for the considered EO applications based on keywords/tags in terms of sensor type, geographical location and data acquisition time of the satellite images stored in the archives. However, in the era of big data, the content of the satellite data is much more relevant than the keywords/tags. In order to keep up with the growing need of automatization, knowledge discovery systems and tools that operate on the content of the satellite images are necessary.
In the ERC BigEarth project, we develop cutting-edge methods for: i) large-scale image representation learning; and ii) large-scale image search and retrieval for an accurate and fast discovery of crucial information for observing Earth from big EO archives. The methods developed in the ERC BigEarth project provide the foundations for knowledge discovery systems that index and query the complex content of large-scale EO data in a scalable and accurate manner. In detail, the BigEarth project consists of five main Aims in total, from which four Aims are associated to the development of novel methodologies and tools on the main challenges of Big EO data and also one Aim is related to the benchmark archive construction to validate the algorithms and the software.
Aim 1: Development of novel methods and tools to characterize and exploit high level semantic and spectral information present in remote sensing (RS) images;
Aim 2: Development of novel feature extraction methods and tools to directly extract features from the compressed RS images;
Aim 3: Development of accurate and scalable RS image indexing and retrieval methods together with associated tools;
Aim 4: Development of methods and tools to integrate feature representations of different RS image sources into a unified form of feature representation;
Aim 5: Construction of a benchmark archive with high number of multi-source RS images.
The methods and algorithms developed in the BigEarth project: 1) address the challenges on knowledge discovery from big data archives for EO, which contributes to the EU’s Artificial Intelligence research and innovation agenda; and 2) ease the information discovery from massive archives based on efficient and effective modelling, indexing and querying the complex content of RS images (which go beyond the simple keywords/tags-based search).
In the ERC BigEarth project, we develop cutting-edge methods for: i) large-scale image representation learning; and ii) large-scale image search and retrieval for an accurate and fast discovery of crucial information for observing Earth from big EO archives. The methods developed in the ERC BigEarth project provide the foundations for knowledge discovery systems that index and query the complex content of large-scale EO data in a scalable and accurate manner. In detail, the BigEarth project consists of five main Aims in total, from which four Aims are associated to the development of novel methodologies and tools on the main challenges of Big EO data and also one Aim is related to the benchmark archive construction to validate the algorithms and the software.
Aim 1: Development of novel methods and tools to characterize and exploit high level semantic and spectral information present in remote sensing (RS) images;
Aim 2: Development of novel feature extraction methods and tools to directly extract features from the compressed RS images;
Aim 3: Development of accurate and scalable RS image indexing and retrieval methods together with associated tools;
Aim 4: Development of methods and tools to integrate feature representations of different RS image sources into a unified form of feature representation;
Aim 5: Construction of a benchmark archive with high number of multi-source RS images.
The methods and algorithms developed in the BigEarth project: 1) address the challenges on knowledge discovery from big data archives for EO, which contributes to the EU’s Artificial Intelligence research and innovation agenda; and 2) ease the information discovery from massive archives based on efficient and effective modelling, indexing and querying the complex content of RS images (which go beyond the simple keywords/tags-based search).
We developed several methods in framework of remote sensing (RS) image understanding, search and retrieval for fast and accurate information discovery from massive data archives. To achieve accurate remote sensing image representations, we introduced: i) a multi-attention driven approach; ii) a graph-theoretic deep representation learning method; iii) a plasticity-stability preserving multi-task learning method to jointly learn different learning tasks; and iv) several label-noise robust deep learning (DL) models to reduce the negative impact of noisy land-use and land-cover annotations. Due to the dramatically increased volume of RS image archives, images are usually stored in compressed format to reduce the storage size. Existing content based RS image retrieval and classification systems require as input fully decoded images, thus resulting in a computationally demanding task in the case of large-scale image retrieval problems. To overcome this limitation in retrieval problems, we developed novel systems, such as: 1) a system that achieves a coarse to fine progressive RS image description and retrieval in the partially decoded JPEG 2000 compressed domain; 2) a system that applies scene classification with deep neural networks in JPEG 2000 compressed domain; and 3) a system that achieves simultaneous deep learning-based image compression and hashing-based image indexing. The developed systems significantly reduce the computational time with similar retrieval and classification accuracies when compared to traditional approaches. To achieve high time-efficient search capability within huge data archives, we also researched on deep hashing methods that encode high-dimensional image descriptors into a low-dimensional Hamming space where the image descriptors are represented by binary hash codes. By this way, the (approximate) nearest neighbors among the images can be efficiently identified based on the the Hamming distance with simple bit-wise operations. One of the methods that we developed is the metric-learning based hashing network, which learns: 1) a semantic-based metric space for effective feature representation; and 2) compact binary hash codes for fast archive search. To integrate feature representations of different RS image modalities into a unified form of feature representation, we developed several multi-modal learning methods and tools. As an example, we introduced a self-supervised cross-modal RS image retrieval method that: i) models mutual-information between different modalities in a self-supervised manner; ii) retains the distributions of modal-specific feature spaces similar to each other; and iii) defines the most similar images within each modality without requiring any annotated training image. Moreover, we explored the effectiveness of masked autoencoders for sensor-agnostic (modality-agnostic) image search and retrieval in RS. We derived a guideline to exploit masked image modeling for uni-modal and cross-modal search and retrieval problems in RS. Most DL models require a huge amount of annotated images during training to optimize model parameters and reach a high performance during evaluation. The availability and quality of such data determine the feasibility of many DL models. To address this issue, we introduced benchmark datasets (e.g. BigEarthNet, HySpecNet-11k). BigEarthNet is a large-scale benchmark archive for RS image understanding (it is available at http://bigearth.net) and is the most impactful dataset that we developed. It is made up of 590,326 pairs of Sentinel-1 and Sentinel-2 image patches, enabling data-hungry DL algorithms in the context of multi-label RS image retrieval and classification tasks. Thus, it makes a significant advancement for the use of DL in RS, opening up promising directions to advance DL-based research in the framework of RS image scene classification and retrieval. All the data and the DL models are made publicly available, offering an important resource to guide future progress on image scene classification and retrieval problems in RS.
The BigEarth project addressed the emerging methods and tools that significantly improve the state-of-the-art for the content based image understanding, search and retrieval in remote sensing (RS). To this end, challenging and very important scientific, technical and practical problems were addressed by focusing on the main challenges of Big EO data, which are: RS image characterization, indexing and search from massive archives. The BigEarth team published more than 60 international journal and conference papers on these topics.