Periodic Reporting for period 2 - ComPat (Computing Patterns for High Performance Multiscale Computing)
Période du rapport: 2017-04-01 au 2018-09-30
Our Objectives:
Our main objective is to develop generic and reusable High Performance Multiscale Computing algorithms that will enable us to tackle scientific grand challenges at the exascale. The algorithms will provide scalability, robustness, resiliency, and efficiency of multiscale applications with extreme data requirements. We will formalise three multiscale computing patterns, all of them incorporating customized algorithms for load balancing, data handling, fault tolerance and energy consumption under generic exascale application scenarios, as well as performance prediction models. We will develop and implement all algorithms required by the three multiscale computing patterns, co- designing with the anticipated characteristics of exascale machines and providing strategies for optimisation with respect to such characteristics. We will develop an application toolkit needed to instantiate the computing patterns which in turn will allow multiscale simulations to reach exascale performance. Adopting selected middleware, we will realise an Experimental Execution Environment on HPC resources available to the project. We will implement nine grand challenge applications as instantiations of the multiscale computing patterns. Their scalability and performance will be measured on available high performance computing systems and will be predicted for future exascale systems. The added value of our approach for software engineering for extreme parallelism will be demonstrated.
Main Results of ComPat:
ComPat has delivered a design of three Multiscale Computing Patterns (MCP), implemented MCP algorithms and software, and tested this software on a Pan-European Experimental Execution Environment operated by the project. ComPat delivered an integrated software stack, relying on QCG, to execute a range of multiscale applications on the EEE, using the MCP algorithms and software. This resulted both in a proof of concept of the vision of MCPs, as well as detailed performance measurement of multiscale applications executed with the MCP algorithms and software. ComPat has strongly disseminated these results in papers, conferences, workshops, schools, and a webinar, as well as by creating a set of videos to explain the major developments in ComPat.
Our main objective is to develop generic and reusable High Performance Multiscale Computing algorithms that will enable us to tackle scientific grand challenges at the exascale. The algorithms will provide scalability, robustness, resiliency, and efficiency of multiscale applications with extreme data requirements. We will formalise three multiscale computing patterns, all of them incorporating customized algorithms for load balancing, data handling, fault tolerance and energy consumption under generic exascale application scenarios, as well as performance prediction models. We will develop and implement all algorithms required by the three multiscale computing patterns, co- designing with the anticipated characteristics of exascale machines and providing strategies for optimisation with respect to such characteristics. We will develop an application toolkit needed to instantiate the computing patterns which in turn will allow multiscale simulations to reach exascale performance. Adopting selected middleware, we will realise an Experimental Execution Environment on HPC resources available to the project. We will implement nine grand challenge applications as instantiations of the multiscale computing patterns. Their scalability and performance will be measured on available high performance computing systems and will be predicted for future exascale systems. The added value of our approach for software engineering for extreme parallelism will be demonstrated.
Main Results of ComPat:
ComPat has delivered a design of three Multiscale Computing Patterns (MCP), implemented MCP algorithms and software, and tested this software on a Pan-European Experimental Execution Environment operated by the project. ComPat delivered an integrated software stack, relying on QCG, to execute a range of multiscale applications on the EEE, using the MCP algorithms and software. This resulted both in a proof of concept of the vision of MCPs, as well as detailed performance measurement of multiscale applications executed with the MCP algorithms and software. ComPat has strongly disseminated these results in papers, conferences, workshops, schools, and a webinar, as well as by creating a set of videos to explain the major developments in ComPat.
The main result is the design and implementation of multiscale computing patterns software and services, operational on the ComPat Experimental Execution Environment, with three multiscale computing patterns being realized and used by nine different applications.
To reach this point the project has achieved a large number of results, that can be summarized as follows:
1) We have an application portfolio containing nine different applications from four domains, and all nine applications have been successfully deployed and executed on the most powerful supercomputers using ComPat tools. These applications drive forward the research and developments within ComPat, and are representative for a wide range of multiscale applications that need HPC.
2) We have laid theoretical foundations for the concept of Multiscale Computing Patterns, in context of the Multiscale Modelling and Simulation Framework, as being motifs in task graphs. For all three patterns we have identified these generic MCP task graphs.
3) We have designed and implemented Multiscale Computing Patterns and algorithms software, and interfaced it with the European-made QCG middleware.
4) We have instantiated all nine applications (from materials science, fusion, biomedicine and astrophysics) as three Multiscale Computing Patterns (Extreme Scaling, Replica Computing, Heterogeneous Multiscale Computing) relying on three different coupling environments (MUSCLE2, file-based data exchange/FabSim, and AMUSE) and a newly developed pilot job manager available via QCG. We demonstrated their operation and performance.
5) We have delivered tools to automatically measure performance of multiscale applications as well as their single scale components running in HPC environments, both for MUSCLE2 Custom Metric and Partial Report and for Energy profiling. These measurements are routinely stored in a database, and the results are again used by ComPat services to find the most optimal and efficient deployment of a multiscale simulation of a range of HPC resources.
6) We have designed the ComPat System Architecture and released ComPat specific QCG middleware services and tools.
7) We realized the ComPat Experimental Execution Environment, with all 3 sites (LRZ, PSNC, STFC) fully integrated.
To reach this point the project has achieved a large number of results, that can be summarized as follows:
1) We have an application portfolio containing nine different applications from four domains, and all nine applications have been successfully deployed and executed on the most powerful supercomputers using ComPat tools. These applications drive forward the research and developments within ComPat, and are representative for a wide range of multiscale applications that need HPC.
2) We have laid theoretical foundations for the concept of Multiscale Computing Patterns, in context of the Multiscale Modelling and Simulation Framework, as being motifs in task graphs. For all three patterns we have identified these generic MCP task graphs.
3) We have designed and implemented Multiscale Computing Patterns and algorithms software, and interfaced it with the European-made QCG middleware.
4) We have instantiated all nine applications (from materials science, fusion, biomedicine and astrophysics) as three Multiscale Computing Patterns (Extreme Scaling, Replica Computing, Heterogeneous Multiscale Computing) relying on three different coupling environments (MUSCLE2, file-based data exchange/FabSim, and AMUSE) and a newly developed pilot job manager available via QCG. We demonstrated their operation and performance.
5) We have delivered tools to automatically measure performance of multiscale applications as well as their single scale components running in HPC environments, both for MUSCLE2 Custom Metric and Partial Report and for Energy profiling. These measurements are routinely stored in a database, and the results are again used by ComPat services to find the most optimal and efficient deployment of a multiscale simulation of a range of HPC resources.
6) We have designed the ComPat System Architecture and released ComPat specific QCG middleware services and tools.
7) We realized the ComPat Experimental Execution Environment, with all 3 sites (LRZ, PSNC, STFC) fully integrated.
ComPat strongly contributes to the HPC strategy in Europe, by demonstrating new avenues for effective and energy efficient use of HPC for a potentially very wide range of multiscale applications. This will help in realization of exascale performance levels by applications developers, will influence next generation compute architectures, and will influence policies of funding organisations and service providers to create the infrastructure needed for exascale deployment and new use cases for HPC. Its current progress beyond state of the art, as summarized above, already leads to such potential impact as ComPat actively joins discussion on next HPC in Europe, ‘towards the Exascale’.
In the specific application domains, scientific results will arise from improved fidelity of simulations in applications areas. We have already seen this in the results obtained in the extreme scale simulations carried out with the Binding Affinity Calculator on SuperMuc.
ComPat is educating and training a new generation of researchers and developers with enhanced skills in what we call High Performance Multiscale Computing. We are pushing this paradigm forward, as we believe that this will be very beneficial for use of future European Exascale HPC infrastructures.
In the specific application domains, scientific results will arise from improved fidelity of simulations in applications areas. We have already seen this in the results obtained in the extreme scale simulations carried out with the Binding Affinity Calculator on SuperMuc.
ComPat is educating and training a new generation of researchers and developers with enhanced skills in what we call High Performance Multiscale Computing. We are pushing this paradigm forward, as we believe that this will be very beneficial for use of future European Exascale HPC infrastructures.