Periodic Reporting for period 2 - ACROSS (HPC BIG DATA ARTIFICIAL INTELLIGENCE CROSS STACK PLATFORM TOWARDS EXASCALE)
Reporting period: 2022-09-01 to 2024-02-29
The ACROSS project will co-design and develop an HPC, BD, and Artificial Intelligence convergent platform, supporting applications in the Aeronautics, Climate and Weather, and Energy domains. To this end, ACROSS will leverage on next generation of pre- exascale infrastructures and on effective mechanisms to easily describe and manage complex workflows. Energy efficiency will be achieved by massive use of specialized hardware accelerators, and applying smart mechanisms of scheduling jobs.
Overall Platform Objectives:
-Foundation and co-design of energy efficient and HPC/BD/AI cross-stack platform.
-Validation of ACROSS platform through industrial applications and fostering societal applications adoption.
-Ensuring ACROSS platform interoperability, adaptability and security.
Pilots specific Objectives:
-Aeronautics: Enhancing effectiveness in designing key aeronautical components (engine components, combustors, turbines) by adopting new workflows, Multi-scale/Multiphysics unsteady approach, and artificial intelligence.
-Weather and Climate: Enhancing global numerical weather prediction by means of hardware acceleration, low-latency exploitation of climate simulations, and enabling HPDA on large datasets.
-Energy and Carbon Sequestration: Improving capability of performing large-scale carbon geologic sequestration simulations; enable direct subsurface flow simulations on processed seismic data.
Technology-Specific Objectives:
-Co-designing HW/SW integration in collaboration with pilots, for creating exascale-ready services and ensuring compliance with the future EPI initiative.
-Building a platform supporting the execution of large-scale HPC simulations along with exploitation of big data and deep learning techniques and with seamless exploitation of hardware heterogeneity.
-Efficient execution of mixed HPC/BD/DL workflows through hardware accelerators.
The results will enable or drastically improve performance of a wide range of computationally intensive tasks, such as large-scale complex simulations and advanced AI-enabled analysis aim of generating innovation and value creation in key societal and industrial key sectors in Europe as (Greener aero engine, Weather and climate and C02 seqyestration
Overall Platform Objectives:
-Foundation and co-design of energy efficient and HPC/BD/AI cross-stack platform.
-Validation of ACROSS platform through industrial applications and fostering societal applications adoption.
-Ensuring ACROSS platform interoperability, adaptability and security.
Pilots specific Objectives:
-Aeronautics: Enhancing effectiveness in designing key aeronautical components (engine components, combustors, turbines) by adopting new workflows, Multi-scale/Multiphysics unsteady approach, and artificial intelligence.
-Weather and Climate: Enhancing global numerical weather prediction by means of hardware acceleration, low-latency exploitation of climate simulations, and enabling HPDA on large datasets.
-Energy and Carbon Sequestration: Improving capability of performing large-scale carbon geologic sequestration simulations; enable direct subsurface flow simulations on processed seismic data.
Technology-Specific Objectives:
-Co-designing HW/SW integration in collaboration with pilots, for creating exascale-ready services and ensuring compliance with the future EPI initiative.
-Building a platform supporting the execution of large-scale HPC simulations along with exploitation of big data and deep learning techniques and with seamless exploitation of hardware heterogeneity.
-Efficient execution of mixed HPC/BD/DL workflows through hardware accelerators.
The results will enable or drastically improve performance of a wide range of computationally intensive tasks, such as large-scale complex simulations and advanced AI-enabled analysis aim of generating innovation and value creation in key societal and industrial key sectors in Europe as (Greener aero engine, Weather and climate and C02 seqyestration
All pilots workflows are integrated on the ACROSS software stack, the functional and technical Objectives are completed specifically
Foundation and co-design of energy efficient and HPC/BD/AI cross-stack platform: finak version of ACROSS software stack ha been delivered and validate including the the workflows deployment
Validation of ACROSS platform through industrial applications and fostering societal applications adoption: Validation was performed including KPI analysis
Ensuring ACROSS platform interoperability, adaptability and security: The final version of the ACROSS software stack (we remark again here that, the ACROSS platform refers to the instantiation of the software stack on a given infrastructure, considering all the constraints brought by the specific infrastructure -e.g. the inability to install certain components, the specific management policies for computing resources, etc.) have been demonstrated
AERONAUTICS: the two Aeronautics use-cases have been fully developed according to the plan with good results achieving the goals fixed at the beginning of the ACROSS project.
WEATHER AND CLIMATE: Enhancing global numerical weather prediction by means of hardware acceleration, low-latency exploitation of climate simulations, and enabling HPDA on large datasets has been reached
ENERGY AND CARBON SEQUESTRATION: we have achieved further improvements in the areas of performance, I/O and scalability to reach the project goal, as indicated by the KPI-3.1 and KPI-3.2. Furthermore, substantial advances in GPU acceleration have been made beyond the project goals.
Co-designing HW/SW integration in strict collaboration with pilots, for creating exascale-ready services and ensuring compliance with the future EPI initiative: the activities targeting the exploitation of hardware acceleration technologies and the tight integration of all the software components (from the higher-level of the orchestration stack to the low-level, domain specific libraries and frameworks, to the libraries and tools enabling the use of hardware accelerators) have been carried out
Building a platform supporting the execution of large-scale HPC simulations along with exploitation of Big Data and deep learning techniques and with seamless exploitation of hardware heterogeneity: he ACROSS software stack has been implemented. This activity was complemented by the co-design and development of an advanced execution infrastructure (HPC cluster) incorporating a wide range of computing and acceleration technologies (including FPGAs and GPUs), along with their respective software libraries and frameworks
Efficient execution of mixed HPC/BD/DL workflows through hardware accelerators, productivity maximization using ad-hoc low-level libraries, and improving overall energy efficiency: The high degree of modularity given by the co-designed ACROSS software stack makes possible its instantiation on different computing infrastructures, without compromising resource management policies that are put in place by system administrators of the infrastructures
Optimizing workflow execution through the development of improved orchestration and workflow management strategies: the development and integration of the key orchestration components reached a maturity milestone, which allowed to enable the deterministic execution of workflows and the (fine-grain) efficient exploitation of the node compute resources
Enhancing HPC performance for data stream processing, using in-situ/in-transit techniques, in support of extreme-scale applications: The in-situ post-processing introduced in global-scale NWP has been successfully demonstrated as part of WP6 activities. The technology has been adopted in other large-scale projects (i.e. Destination Earth) and is currently assessed for inclusion in operational forecasts.
Publications in journals, conferences, and workshops; 29 accepted (1 edited-book, 1 book chapter, 9 journal papers, 6 conference papers, 5 posters, 1 PhD Thesis & 6 Workshops), 8 under review and two for submission.
A comprehensive list of pilots’ objectives and related KPIs has been evaluated and achieved.
outcomes of the validation path for the different pilots’ achievements, related to Software and adopted methodologies, were finally achieved.
Foundation and co-design of energy efficient and HPC/BD/AI cross-stack platform: finak version of ACROSS software stack ha been delivered and validate including the the workflows deployment
Validation of ACROSS platform through industrial applications and fostering societal applications adoption: Validation was performed including KPI analysis
Ensuring ACROSS platform interoperability, adaptability and security: The final version of the ACROSS software stack (we remark again here that, the ACROSS platform refers to the instantiation of the software stack on a given infrastructure, considering all the constraints brought by the specific infrastructure -e.g. the inability to install certain components, the specific management policies for computing resources, etc.) have been demonstrated
AERONAUTICS: the two Aeronautics use-cases have been fully developed according to the plan with good results achieving the goals fixed at the beginning of the ACROSS project.
WEATHER AND CLIMATE: Enhancing global numerical weather prediction by means of hardware acceleration, low-latency exploitation of climate simulations, and enabling HPDA on large datasets has been reached
ENERGY AND CARBON SEQUESTRATION: we have achieved further improvements in the areas of performance, I/O and scalability to reach the project goal, as indicated by the KPI-3.1 and KPI-3.2. Furthermore, substantial advances in GPU acceleration have been made beyond the project goals.
Co-designing HW/SW integration in strict collaboration with pilots, for creating exascale-ready services and ensuring compliance with the future EPI initiative: the activities targeting the exploitation of hardware acceleration technologies and the tight integration of all the software components (from the higher-level of the orchestration stack to the low-level, domain specific libraries and frameworks, to the libraries and tools enabling the use of hardware accelerators) have been carried out
Building a platform supporting the execution of large-scale HPC simulations along with exploitation of Big Data and deep learning techniques and with seamless exploitation of hardware heterogeneity: he ACROSS software stack has been implemented. This activity was complemented by the co-design and development of an advanced execution infrastructure (HPC cluster) incorporating a wide range of computing and acceleration technologies (including FPGAs and GPUs), along with their respective software libraries and frameworks
Efficient execution of mixed HPC/BD/DL workflows through hardware accelerators, productivity maximization using ad-hoc low-level libraries, and improving overall energy efficiency: The high degree of modularity given by the co-designed ACROSS software stack makes possible its instantiation on different computing infrastructures, without compromising resource management policies that are put in place by system administrators of the infrastructures
Optimizing workflow execution through the development of improved orchestration and workflow management strategies: the development and integration of the key orchestration components reached a maturity milestone, which allowed to enable the deterministic execution of workflows and the (fine-grain) efficient exploitation of the node compute resources
Enhancing HPC performance for data stream processing, using in-situ/in-transit techniques, in support of extreme-scale applications: The in-situ post-processing introduced in global-scale NWP has been successfully demonstrated as part of WP6 activities. The technology has been adopted in other large-scale projects (i.e. Destination Earth) and is currently assessed for inclusion in operational forecasts.
Publications in journals, conferences, and workshops; 29 accepted (1 edited-book, 1 book chapter, 9 journal papers, 6 conference papers, 5 posters, 1 PhD Thesis & 6 Workshops), 8 under review and two for submission.
A comprehensive list of pilots’ objectives and related KPIs has been evaluated and achieved.
outcomes of the validation path for the different pilots’ achievements, related to Software and adopted methodologies, were finally achieved.
The aim of the ACROSS societal and industrial impact maximization activities was to promote the project’s vision to specific target communities which are expected to further increase its visibility and accelerate its uptake.
• 38 external stakeholders have been identified in total by consortium partners in an ecosystem mapping activity.
• Four training activities associated with technical aspects of the project have been reported
• Eight unique newsletters have been issued
•49 blog posts (i.e. 20 specialized and 29 general posts) has been created by consortium partners and published.
• More than 8000 unique visitors have visited the project website (3100 during the first period), while 617 social media followers (360 on first period) have been recorded.
• A Wikipedia page dedicated to ACROSS will soon be available
• Three videos (two technical and one pilot-related) were created and uploaded
• 38 external stakeholders have been identified in total by consortium partners in an ecosystem mapping activity.
• Four training activities associated with technical aspects of the project have been reported
• Eight unique newsletters have been issued
•49 blog posts (i.e. 20 specialized and 29 general posts) has been created by consortium partners and published.
• More than 8000 unique visitors have visited the project website (3100 during the first period), while 617 social media followers (360 on first period) have been recorded.
• A Wikipedia page dedicated to ACROSS will soon be available
• Three videos (two technical and one pilot-related) were created and uploaded