Final Report Summary - POPDAT (Problem-oriented Processing and Database Creation for Ionosphere Exploration)
Executive Summary:
In the frame of the FP7 POPDAT project the Ionosphere Waves Service (IWS) was developed by ionosphere experts to assist the ionosphere and Space Weather research, and to answer several questions, for instance: How make the data of old ionosphere missions more valuable? How to provide scientific community with a new insight into wave processes that take place in the ionosphere? The answer is a unique data mining service accessing a collection of topical catalogues that characterize a huge number of Whistlers-like Electromagnetic Wave Phenomena, Atmospheric Gravity Waves and Traveling Ionosphere Disturbances. Based on the detailed catalogue of ionosphere wave phenomena the Ionosphere Waves Service regroups databases of specific events extracted by specific software systems from records of selected ionosphere satellite missions which end users can access by applying specific searches and by using statistical analysis modules for their domain of interest. Therefore the IWS opens a new way in ionosphere and Space Weather research. The scientific applications covered by the IWS beside the Space Weather are related to earthquake precursors, ionosphere climatology, geomagnetic storms, troposphere-ionosphere energy transfer, and trans-ionosphere link perturbations.
Project Context and Objectives:
The Earth’s ionosphere is a substantial part of the solar-terrestrial interaction processes and (together with the magnetosphere) of the assistance of the existence of life on the Earth in the “neighborhood” of a star, the Sun. Therefore, ionosphere plasma dynamics influenced by the active Sun becomes an important element of the Space Weather. Thus, it exhibits satellite and ground based radio communication “break up’s”, disturbed operation of the global satellite positioning systems, as GPS, GLONASS, GALILEO, COMPASS, IRNSS, ground based power stations “switch off’s” in their normal operation, turbulent transients in ground based power lines, biomedical problems in the case of humans being in sensitive state, etc. Simultaneously, Earth’s ionosphere is strongly influenced by the large scale circulation in the troposphere (e.g. typhoons), earthquakes and tsunamis (perhaps in the earthquake preparatory phase), powerful nuclear or chemical explosions, larger rocket launches, etc. Beside these important effects, which appear in the ionosphere, the ionosphere is the coupling zone between the lower atmosphere including the troposphere, the biosphere and our civilization and the high atmosphere, i.e. exosphere, plasmasphere, magnetosphere; which is the coupling zone between the interplanetary, i.e. solar-influenced region and the Earth’s environment. Therefore the actual state of the ionosphere and the processes, transients inside the ionosphere has special importance not only in the research, however, in the managing of the operation of our civilization too.
The actual state of the ionosphere is important to the evolution and effects of Space Weather phenomena; the continuous monitoring of the state of these regions is a key element in Space Weather research and applications. One of the most important tools of monitoring is the recording and analysis of signals having natural or artificial origin and propagating through the interesting areas. However, the propagation of these signals through the ionosphere affects the signal shapes. These propagation effects could be tools of the monitoring e.g. the Space Weather effects, or must be taken account in the monitoring of other phenomena, e.g. seismic events.
In this context, the study of vertical coupling in the Earth’s ionosphere with adjacent zones is an actual subject with relatively high importance and practical application. In this sense, new techniques for ionosphere data evaluation, creation of integrated data bases containing different kinds of recorded ionosphere phenomena by different satellite missions, producing integrated data base during longer time period and applying new data mining techniques seems to be a successive solution for further studies carried out by the usage large amount of satellite data over prolonged periods of observations. As a part of this approach it is necessary to apply data processing procedures which are addressed to the recognition and collection of the various signatures appearing in the ionosphere, which conform to the specific physical phenomena already chosen by an operator or end-user.
The idea of the Ionosphere Waves Service (IWS)
In the earlier stage, satellite experiments were addressed to study the main morphological features in the global distribution of the atmosphere, of the ionosphere parameters (altitudinal, latitudinal, diurnal, etc.). At present large amount of satellite data gathered during these missions are successfully integrated in the various models of the upper atmosphere and ionosphere. Upcoming data from the current missions are used for the adjustment of existing models or to the development of alternative models. However, during this long research work a lot of transient, propagating wave-like phenomena was recorded and identified, and now the importance of these wave-like phenomena in the ionosphere investigation was verified. However, the systematic investigation of these phenomena has an accidental character and is limited by the existing structures of the raw satellite data sets.
The processing of the satellite experiments includes few important levels: The first level is the raw data conversion, decoding of the control parameters, removing erroneous data records, adjustment to the universal time, etc. This process transforms the telemetry data flow to the basic electrical signals relevant to the specific application of the onboard instrumentation as a chain of equidistant records in time. At the second level, these records of electrical values are transformed in the real physical one by the use of the next level (calibration, etc.) algorithms quite specific for different experiment.
The idea of the IWS is to apply a third level data processing and with this step to create a new data base, which is the goal of the EU FP7 POPDAT project. The main purpose of this third level of data processing is to offer an opportunity to the user to select specific “shapes” from the whole data set which correspond to the intriguing physical phenomena by use especially created software tools. Because the transients, the wave-like phenomena are one of the most informative “entities” in the ionosphere and related phenomena investigations, the IWS data base is the result of a third level data processing oriented to ionosphere wave-like phenomena. For example, it could be “wave-like structures” with different scales in the ion density or “double layer” structures in quasi DC electric field, “vortexes”, etc. This third level of data processing needs specific physical knowledge for algorithms and software tools to be accurately adjusted to the interesting cases for users. The current level of information technologies reveal realistic opportunities, third level of data mining to be used over sets of data taken from different satellites at different places and times, combined with sort of solar-geophysical data.
The creation and application of the Ionosphere Waves Service is a good tool and the first step toward the creation of third level set of physical knowledge about the ionosphere processes, the transients in the ionosphere, to assist the different investigations in Earth science and in Space Weather research, and to open the way for a wide range researcher certainly not included in the specific satellite project teams, working groups, etc, but interested in data mining and interpretation. For example, there is a cluster of scientists interested on the seismic effects in the ionosphere during the preparatory phase of earthquakes. Using this third level data base many scientists could study different kind of phenomena associated with earthquakes such as gravity waves in the ionosphere, changing the appearance density or the dispersion of whistlers appearing at the ionosphere, already included in the IWS data base. The IWS could be very helpful to the wide number of users in the space science community.
Satellite missions interesting in ionosphere investigations
From the beginning of the satellite investigations of the Earth and Earth’s environment more than 20 satellites were launched for the investigation of the ionosphere. The Table 1 contains the list of satellites which could be interest in the investigations of transient phenomena in the ionosphere and therefore it is necessary to investigate the possible application of the recorded data sets of these missions during the creation of the IWS.
[insert the table from the attachement]
Table 1. Main ionosphere satellite missions important in the creation of the IWS
To the creation of the IWS the first step was a critical review of this list of satellites to select the satellite missions which could have importance in this step. The broadening of the satellite list in the future after the start of the operation of the IWS is possible, however, in the first step we must use only the guaranteed data sets. This means that the incorporation the data of new satellite missions which are in operation it this time or will start in the near future is also possible.
The selection criteria of satellite mission having importance in the creation of the IWS are the following:
In general the availability of recorded data sets was one of important criteria. For example the APEX mission was considerable (USSR) space project; however, the recorded and transmitted to the Earth data of this satellite are not in digital electronic form in a open access data base. Therefore it is not possible to include this satellite into the creation phase of the IWS. The open (Internet) access to the recorded data, the original “raw” data of a given satellite without further data processing limitations was very successful for selection a satellite into this first creation step.
However, further selection criteria were applied in the case of electromagnetic wave (EMW) data. These are the following: The orbit can guarantee that the recorded electromagnetic signal is ionospheric one (≤ 800 – 900 km in the moment of or during the measurement) and it has no “magnetospheric” character. It is important the existence of the real broad-band ELF-VLF “burst”-type data in digital electronic form with the required accuracy, i.e. the preferred recorded frequency band is > 50-100 Hz to ≤ 20-40 kHz. In the case of older missions the 5 kHz upper frequency limit was accepted also. The required accuracy was the minimum 8 bit resolution of a sample, but the 10 to 16 bit resolution was preferred.
The additional selection criteria in the case of atmosphere gravity wave (AGW) and travelling ionosphere disturbance (TID) investigations were the “major” and “distinguished” missions target on in situ measurements of the ionosphere parameters. In details: Instrumental and orbital opportunities for simultaneous observations of AGW and TID, i.e. the existence of parallel onboard measurements of neutral and charged particle parameters at ionospheric altitudes (≤ 350 – 400 km or below 800 km depending on the instrumentation in the moment of or during the measurement). Beside this the sufficient time resolution along the orbit provided by the onboard instrumentation permitting AGW and TID registration over available data sets is also important. A criteria is the existence of prolonged observations (large data sets) with the comparable set of onboard instrumentation for various satellite projects.
By the application of the selection criteria for the satellite listed in Table 1 the following satellite missions were selected for the creation of the IWS: Atmosphere Explorer – C, Atmosphere Explorer – E, Dynamic Explorer 2, Active (Intercosmos – 24), GPS system, Variant (Sich – 1M), DEMETER and Compass – 2. By applying of the original (first and second level) data bases of these missions it was possible to produce the third level data sets and to create the IWS. After the completion of the basic IWS configuration it is possible to include other older or really new ionosphere satellite data bases, e.g. other Atmosphere Explorer data or the new Chibis-M data.
Project Results:
The main project outcome - Ionospheric Waves Service (IWS) is a powerful research tool made publicly accessible to various research communities, starting from hard-core ionosphere research to Space Weather, earthquake precursors, communication, space navigation, etc. There are some examples of the service application for solving practical scientific problems
Study on the through the ionosphere to the ground propagation of whistler-type EM phenomena
The IWS opens the way before some important research on a valuable level. Because the IWS contains the wave-like events with their meta-data system this new data base is an excellent tool in the statistical investigations of the event generation and / or appearing. An example of this type of applications is the statistics of the fractional hop whistler signal pairs (trapezoid whistler pairs) identified by AWDSat-WHPF software in DEMETER records and the events stored in IWS. In Fig. 1 the statistical distribution of the signal pairs dispersions are present as a function of the geomagnetic latitude containing the data of nearly 15 million (!) trapezoid whistler pairs. In Fig. 2 the geographical distribution is present of these signals confirming that these signals can propagate through the ionosphere only sporadic around a wider region of the geomagnetic equator and more often out of this zone, which zone is conform with the electron density distribution of the ionosphere.
[Figure 1]
[Figure 2]
The more effective investigation of the ionosphere (ionosphere parameters, equatorial anomaly, changing of the whistler occurrence depending on the solar activity, etc.) is possible using the IWS and this can produce new knowledge about the Space Weather effects.
Application example in the investigation of thunderstorm activity
Another possibility is e.g. the investigation of the electromagnetic manifestation of thunderstorms. In this case it is necessary the simultaneous application of satellite and ground based recorded data. The ground based data sets are not a part of the IWS. However, this example can confirm that in the future it will be extremely important to create a new high level data base in which the satellite and ground based data will be present in third (or an imagined fourth) level processed form with unified time base etc. (Therefore was elaborated a proposal in the FP7 to reach this goal, but it was not selected.) The relation between the thunderstorms and the ionosphere electromagnetic transient activity is also part of the Space Weather research.
Whistlers are electromagnetic waves that can be generated by lightning discharges and are observed at VLF range. In order to examine most standard way of analysis devoted to examination of electromagnetic manifestation of thunderstorm, DEMETER ICE data and ground based registration were utilized. Presented example serves as general approach to the started problem. Thunderstorm activity is monitored in Warsaw, where the cluster of 6 antennas is placed. They have been operating since 2009. Each antenna is measuring electric field changes that can be caused by thunderstorm. Discharge events are indicated by the rapid decrease in electric field, and we expect the signal registered to look as Fig. 29 presents. The limitations of this approach are due to restricted area of ground-based measurements and the nature of the recorded signals.
[Figure 3]
[Figure 4]
On the other hand there are satellites electric field registrations. For the purpose of this investigation we present two whistler events that were registered on DEMETER on 20-21 July 2007. At the same time SPARTAN Sprite-Watch Team form IMGW Meteorological Observatory located on Śnieżka Mountain in Poland recorded two sprites. Fig. 3 presents the two sprites. The distance between the location of the event and satellite position was around 1200 km. The wave forms were analyzed and as a final output the spectrogram for ELF and VLF ranges were computed. The Fig. 4 presents the ELF spectrogram of DEMETER records. [Błęcki et al., 2009] So we could say that the IWS opens the way in such kind of investigations and these type of Space Weather investigations are a strong argument to create a unified data base containing ionosphere satellite and ground based record.
IWS as Ionosphere Dynamics Virtual Observatory
IWS is powerful tool for the visualization and study of the ionosphere dynamics. To demonstrate IWS possibilities let us consider Fig. 5, which simply shows a few AGW waveforms taken from IWS database and superimposed on a geographical map. This figure visualizes the global picture of the wave activity of upper atmosphere. It turns out that AGW wave field has specific morphological structure, including: (1) areas of polar caps, where AGWs are continuously generated and their amplitude reaches maximum values; (2) mid-latitudes, where the intensity of AGWs decreases as the distance from the auroral ovals increases; and (3) latitudes below 40–50°, where the wave activity reaches minimum but does not fully disappear. Moreover, on this global wave back-ground the isolated wave bursts (distinctively localized wave packets) are sporadically observed.
[Figure 5]
The next Fig. 6 represents result of the application of IWS search opportunities to TID-catalogue. This visualization provided by ordinary IWS’ tool proves that, contrary to popular belief, there is no real physical border between middle scale and large scale TIDs, this common division is only conventional.
[Figure 6]
Statistical study of the seismo-ionosphere disturbances
Let us analyze possible relation of earthquakes to generation of localized wave packets of AGW (denoted in the AGW catalogue as ‘LAGW’). We would be only interested in the evidences of correlation between seismic and ionospheric events, regardless of any hypothetical mechanisms of seismo-ionospheric coupling. Fig. 7 shows geographical distribution of the earthquakes occurred during January-February 1983 together with locations of LAGWs detected by DE 2. This figure suggests that ionosphere disturbances and earthquakes are somehow connected (especially, if one takes into account that, during the propagation, AGW shifts from the source horizontally over thousands of kilometers). Following the approach [Skorokhod and Lizunov, 2012], we restrict the set of suitable earthquakes by intuitive criteria: (1) we consider only strong shocks with magnitudes of M > 5.0; (2) the hypocenter depth is < 100 km; (3) epicenters are located less than 4500 km away from AGWs.
Fig. 8 shows a causal diagram where all as described above earthquakes are placed in the reference origin using the method of superimposed epochs and the LAGWs are shown with respect to them in the time–distance coordinate system. This figure clearly separates a group of AGW as earthquake precursors at times from –8 to –4 h.
[Figure 7]
[Figure 8]
So the Ionosphere Waves Service, which is in operation, is an excellent tool in the different field of the Space Weather, earthquake and ionosphere dynamics research. It opens new possibilities in the space research and Earth sciences by the unification of different satellite recorded data sets.
In the future the creation a new high level data base containing the IWS and different ground recorded data sets is an important task to increase the effectiveness of the Space Weather and Earth research.
As for another important outcome of the project, it helped to shape a European research community working in the area of ionospheric wave-like phenomena, enabled information and idea exchange, made ionospheric datasets owned by different organizations more accessible and searchable.
Potential Impact:
The POP-DAT Project was aimed at developing and deploying new methods of processing and representation of the ionosphere data received from the series of the completed ionospheric satellite missions. This approach has brought an important body of new knowledge without requiring significant investments and will serve for scientific, technological and economic progress across several sectors.
Strategic impact
At the beginning of the space exploration era the main goal of rockets and satellites measurements was the investigation of morphological structure of the space environment and, on this basis, the development of the models of undisturbed atmosphere, ionosphere and magnetosphere. With this respect, the “old” missions have fulfilled their goals, although a considerable volume of measurement data is still unused and, even more, not processed.
In POPDAT, the “old” ionosphere measurements data becomes the source and “raw material” for obtaining new third level data. The magnitude of impact the Project delivers strongly depended on the success of the following project tasks’ implementation: development of the effective algorithms for the third level of data processing, as well as the open-source software implementing these algorithms; creation of the flexible database of wave catalogues (Ionosphere Wave Service) enabling effective and semantically reach search for and usage of data; support to functioning virtual observatories. The exploitation of these means of access to the satellite data for in interests of the wide community of specialists enables subsequent research and studies, including in those areas of physics of ionosphere, which are not addressed by the proposed Project and, even more, not considered at all by the POP-DAT Project proposers. For instance, this might include the development of the own algorithms and processing of the additional arrays of “old” data.
The Project radically widens the circles of specialists and scientists involved in and benefiting from processing “old” ionosphere observation data. This results in new research results and technologies, thus further stimulating the interest to the works in this area (so called, positive feedback).
The Project consortium is confident that the project implementation will have its positive effect on and complement the following activities:
1. “Blue-sky” research activities in the field of physics of ionosphere. A number of ongoing and planned initiatives supported by national and international space agencies and research programmes. Positive effect on the entire scientific ionosphere community.
2. Development of models for Sun-Earth interactions and Space Weather (see below)
3. Research on earthquakes forecast methods and technologies. National and international programmes in Europe and other regions. Examples of principle stakeholders: LPCE/CNES, France; IZMIRAN, Russia; University of Electro-Communication, Japan.
4. Development, preparation and more effective planning of new space projects in the field of Global Earth Observation (GEO Initiative) and space monitoring. Examples: Satellite mission “SWARM” coordinated by ESA’s Living Planet Programme; International satellite mission “Ionosat” (planned launch after 2013), Ukraine-Russia project; International satellite project “IONOZOND” coordinated by ROSKOSMOS, Russia.
The major impacts are also observed through the following positive developments:
• Significant widening of the user community interested in using ionosphere satellite data for solving various problems
• Better use of the available satellite data. The available data contains valuable information, which can generate new knowledge when processed on the basis of modern technologies and modern understanding of the physical processes in ionosphere. Thus, this is an opportunity to capitalize on the investments made decades ago and to bring more value with lesser costs.
• Significant increase in the efficiency of ongoing and planned satellite missions. The project results will help to avoid redundancies and duplications, investments into research, which can be done without making new measurements, thus saving resources for really innovative research efforts.
• The creation of the Ionosphere Waves Services contributes to promoting and improving the access to ionosphere observations from different space missions. The access is granted through the web portal and the promotion is ensured in the frame of scientific congresses, scientific press and by interacting with European Space Weather efforts. In that case, it seems important to point the synergy between the proposed project and the Space Situational Awareness programme of ESA. Indeed, the preparatory phase of SSA programme sets up space weather precursor services dedicated to several applications like satellite operating, satellite design, transionosphere communications, etc., which rely on expert groups such as ionosphere dynamics experts group. Therefore, by funding POP-DAT project, the FP7 Space programme contributed to the high level of expertise of European scientists on ionosphere dynamics and so engage complementary efforts to SSA programme.
Expected impact listed in the work programme
Impact 1. “…Projects are expected to add value to space missions and earth based observations by significantly contributing to the effective scientific exploitation of collected data. They are expected to enable space researchers to take full advantage of the potential value of data sets...“
In fact, this was exactly the main objective of POP-DAT. The Project developed the third-level data processing methods and tools, as well as the cutting-edge system for the information search and access in order to enable the use of the ionospheric observations data collected through more than 20 satellite missions implemented within almost a half of century. As highlighted above, the global scientific community working in the field of space exploration has collected huge volumes of such data, which in many instances still remains unprocessed. This data is a valuable source of new knowledge for several traditional and emerging research areas, knowledge to be extracted on the basis of modern understanding of the various ionospheric processes and by the application of the state-of-the-art technologies.
The Project results are open for the entire scientific community, as well as to other categories of users – all major methods are openly published, software solutions have an open source code, resulting web systems (e.g. Ionosphere Wave Service) has a public web interface. This approach will open up the large collections of the ionospheric data for public use and generate a significant added value for science, society and economy. The non-exhaustive list of ongoing and planned space research and exploration initiatives, other research endeavors of visible novelty and importance, which will benefit from the Project results are listed above. But, the impact of the Project is wider – the entire ionosphere research community gets an opportunity to take full advantage of the collected data.
Impact 2. “… Projects are expected to contribute to the much needed coordination of the exploitation of existing and future data collection, and thereby enhancing the possibility to base research on datasets providing comprehensive or full coverage, while at the same time addressing the potential need for further analysis of existing datasets…”
Clearly, providing the tools for extracting new and needed knowledge from the existing datasets of ionospheric measurements, the project contributes towards a better design of the future data collections. Having gained the effective means for the “old” data mining, the new ionospheric missions are able to avoid unnecessary experiments, measurements and data gathering aimed at obtaining knowledge, which can be extracted from already existing datasets. All this inevitably increases the efficiency of the future data collection efforts and will significantly reduce the associated costs. The project gave a “second life” to the existing datasets and, simultaneously, contributed to enriching the integral quality of the existing and future ionospheric data collections (by improving the coordination between them).
Impact 3. “… It is also expected that the projects will facilitate access to and appropriate use of data for those scientists who are not part of the team having obtained it…”
This was another key focal point of the project. In fact, problem-oriented ionospheric data processing can be implemented by specialists only (in the cases addressed by the project – specialists in the field of wave processes in ionosphere, actually the consortium accumulates best representatives of this community. Also, their organizations normally possess the respective data). But, the results of the third-level data processing enabled by the project represent a significant value for much broader circles of potential users. So, the project results are, above all, designed to serve the needs of external users – scientists and specialists working in many different fields of space research, exploration, communication, etc.
The project consortium made all principle outcomes open to the external world. Particularly, the following project features testify to that:
• All principle technical deliverables which concern with major project results are made public and easily accessible.
• The project invested significant efforts in building open, efficient and convenient search and access mechanisms (e.g. catalogues of data and Ionosphere Wave Service), thus enabling all interested parties to freely search for data, process and use it.
• The project dissemination strategy not only made the results publicly available (e.g. via the project website, publications, etc.), but focussed on active promotion of the results to external scientists (via Demo sessions at the workshops, targeted mailings, cross-linkage with other initiatives involving potential users)
Fundamentally, all major outcomes stay available for the entire scientific community and after the project end. Some limited efforts were invested in developing the viable model of Wave Service provision beyond the project framework. In fact, the service is hosted by SRC PAS - publicly funded research body, which is committed to host and maintain the service for at least 5 years after the project end.
Impact 4. “… Furthermore, projects are expected to add value to existing activities on European and national levels, and to raise the awareness of coordination and synergy efforts among stakeholders…”
The project contributed significantly to preparation and implementation of several national and international initiatives (e.g. expert input for the MIME initiative, ULISSE, SSA, a number of satellite missions – see above). In order to strengthen this influence the project undertook the special actions, such as (1) cross-linkage of the Ionosphere Wave Service with the ULISSE information system, (2) develop and provide the expect input to MIME, (3) communicate the project findings to the national space agencies and ESA, etc. The project’s Dissemination activitiesl implemented the system of actions aimed at raising awareness among major stakeholders (by different means).
The project results have been published in various high-level scientific journals, including Space Weather and Space Climate International Journal, presented at major relevant conferences, such as the Space Conference of the Berlin International Aerospace Exhibition (ILA 2012) and EGU General Congress 2013 in Vienna.
List of Websites:
The project website: www.popdat.org
Main project contact:
Prof. Klaus Briess
Technical University Berlin, Aerospace Institute, Germany
klaus.briess@ilr.tu-berlin.de
