Periodic Reporting for period 2 - REGALE (An open architecture to equip next generation HPC applications with exascale capabilities)
Période du rapport: 2022-10-01 au 2024-03-31
An exascale supercomputer is not “yet another big machine”. With a cost of hundreds of million euros, power consumption in the order of tens of megawatts and a lifetime that reaches a decade at most, judicious management of those resources is of utmost importance. Quite frequently, a machine of this size is not able to operate at full power 24/7, and energy consumption is now a primary concern to keep its environmental footprint and operational costs at acceptable levels without neglecting its ultimate purpose: to equip highly critical applications with the computational capacity to solve extremely resource hungry problems.
Focusing on the application developer side, achieving scalable performance and high system throughput has always been a cumbersome task. To make things even more challenging, next-generation HPC applications can no longer be considered as simple, monolithic blocks. The revolution of Big Data and Machine Learning, the emerging Edge Computing and IoT, with the scale of modern HPC systems and cloud datacentres, are rapidly changing the way we solve scientific problems which are now based on the composition to what we call workflows. Existing solutions may render the execution of such workflows in a large-scale supercomputer either impossible, or extremely time consuming both in terms of development and execution time.
The ultimate goal of REGALE was to pave the way of next generation HPC applications to exascale, energy-efficient systems. To accomplish this, we defined an open architecture, built a number of instantiations of this architecture and incorporated in this system appropriate sophistication in order to equip supercomputing systems with the mechanisms and policies for effective resource utilization and execution of complex applications. Additionally, we promoted five pilots for key sectors by integrating them with appropriate workflow engines, alleviating in this way the developer from the burden to manage computational resources and achieve better performance and scalability.
Focusing on the application developer side, achieving scalable performance and high system throughput has always been a cumbersome task. To make things even more challenging, next-generation HPC applications can no longer be considered as simple, monolithic blocks. The revolution of Big Data and Machine Learning, the emerging Edge Computing and IoT, with the scale of modern HPC systems and cloud datacentres, are rapidly changing the way we solve scientific problems which are now based on the composition to what we call workflows. Existing solutions may render the execution of such workflows in a large-scale supercomputer either impossible, or extremely time consuming both in terms of development and execution time.
The ultimate goal of REGALE was to pave the way of next generation HPC applications to exascale, energy-efficient systems. To accomplish this, we defined an open architecture, built a number of instantiations of this architecture and incorporated in this system appropriate sophistication in order to equip supercomputing systems with the mechanisms and policies for effective resource utilization and execution of complex applications. Additionally, we promoted five pilots for key sectors by integrating them with appropriate workflow engines, alleviating in this way the developer from the burden to manage computational resources and achieve better performance and scalability.
Throughout the course of the project, REGALE progressively defined three versions of the REGALE architecture which guided the design and implementation of several prototypes in a feedback loop. The prototypes involved tools that adhere to the architecture and contribute to an energy-efficient operation of a supercomputing system. Additionally, to promote openness, molecularity and extensibility, in REGALE we also designed an intermediate layer, the REGALE library that is able to glue together alternative tools and technologies. Moreover, five pilots were integrated with workflow engines to increase performance and programmability.
The results of REGALE can be summarized as follows:
Result 1: The REGALE architecture. An open architecture aligned to the activities of the HPC PowerStack with specific components, roles, and a complete set of interfaces.
Result 2: The REGALE prototype system and the REGALE library. An end-to-end instantiation of the REGALE architecture using the components brought in the project by the REGALE partners, interfacing with major components of the HPC ecosystem and incorporating the sophistication sought by the project.
Result 3: The REGALE-compatible software modules. The evolved components of the REGALE prototype system, aligned to the REGALE architecture, i.e. OAR, DCDB, BEO, BDBO, EAR, Melissa, RYAX, EXAMON, COUNTDOWN, ULPController and BeBiDa.
Result 4: The REGALE pilots integrated with workflow engines. The evolved REGALE pilots enhanced with capabilities to align with the REGALE architecture and made ready to execute in large-scale exascale environments and beyond, i.e. Pilot #1: Industrial-Scale Unsteady Adjoint-based Shape Optimization of Hydraulic Turbines (Result 4.1) Pilot #2: In-Transit Workflow for Ubiquitous Sensitivity Analysis and MetaModel Training. Application to Infrastructure Safety (Result 4.2) Pilot #3: Enterprise Risk Assessment (Result 4.3) Pilot #4: Complex geomorphometric models executed over earth observation data for groundwater estimation and management (Result 4.4) and Pilot #5: Design of car bumper made of carbon reinforced polymers (Result 4.5).
Result 1: The REGALE architecture. An open architecture aligned to the activities of the HPC PowerStack with specific components, roles, and a complete set of interfaces.
Result 2: The REGALE prototype system and the REGALE library. An end-to-end instantiation of the REGALE architecture using the components brought in the project by the REGALE partners, interfacing with major components of the HPC ecosystem and incorporating the sophistication sought by the project.
Result 3: The REGALE-compatible software modules. The evolved components of the REGALE prototype system, aligned to the REGALE architecture, i.e. OAR, DCDB, BEO, BDBO, EAR, Melissa, RYAX, EXAMON, COUNTDOWN, ULPController and BeBiDa.
Result 4: The REGALE pilots integrated with workflow engines. The evolved REGALE pilots enhanced with capabilities to align with the REGALE architecture and made ready to execute in large-scale exascale environments and beyond, i.e. Pilot #1: Industrial-Scale Unsteady Adjoint-based Shape Optimization of Hydraulic Turbines (Result 4.1) Pilot #2: In-Transit Workflow for Ubiquitous Sensitivity Analysis and MetaModel Training. Application to Infrastructure Safety (Result 4.2) Pilot #3: Enterprise Risk Assessment (Result 4.3) Pilot #4: Complex geomorphometric models executed over earth observation data for groundwater estimation and management (Result 4.4) and Pilot #5: Design of car bumper made of carbon reinforced polymers (Result 4.5).