Skip to main content
European Commission logo
English English
CORDIS - EU research results
CORDIS
CORDIS Web 30th anniversary CORDIS Web 30th anniversary

Hybrid Eco Responsible Optimized European Solution

Article Category

Article available in the following languages:

Democratising access to high performance computing

The HEROES software platform matches high performance computing resources with digital pioneers who could benefit from emerging digital technologies – helping launch a new era of European innovation.

If industry or scientific communities are to take advantage of fast evolving advances in artificial intelligence (AI) and machine learning (ML), they need access to high performance computing (HPC) facilities to process large volumes of data. But these are costly and time-consuming to set up and run, and their energy use raises environmental concerns. The EU-funded HEROES project has developed software which matches HPC resources to users, to enable them to run simulations and ML workflows based on criteria such as time, budget and energy efficiency. “We are entering an era of complex digital choices where computational cost per second is vital. Our solution not only helps users navigate the options but democratises HPC, as users don’t need specialist IT knowledge to benefit,” says Philippe Bricard, HEROES project coordinator from UCIT, the project host. HEROES developed and tested a demonstrator trialling workflows from the renewable energy and manufacturing sectors.

Simplifying complex infrastructure options

For users of data-heavy technologies such as AI or ML, selecting the best HPC option to run simulations means not only understanding specialist IT jargon, but also weighing up multiple requirements related to budgets, turnaround times, data security and environmental footprint. The HEROES solution simplifies the process by finding choices for users. In the project’s prototype, once logged in, users select a workflow or job template, then input their job parameters (including constraints, such as cost). The software then offers a choice between four strategies, linked to HPC platforms. Once selected, the HPC resource runs the job, with the results returned back to the platform. In some cases the best solution will be hybrid, with some computing resources allocated in-house, others assigned externally, depending on the security restraints and business needs of the user. “In the future, our platform will be adapted to various contexts, from private organisations with on-premises HPC clusters and some public cloud resources, to more elaborate arrangements with external facilities once authentication issues are resolved,” explains Bricard. “Our ambition is to enable clients to use our platform to build and operate their own bespoke HPC marketplace.”

From architectural prototype to commercial solutions

The project’s prototype comprises a variety of task-specific modules, principally for: workflow and job orchestration; data transfer and storage; pricing and cost management; and energy monitoring and optimisation. At the solution’s core was the decision-making module which communicates with services providers, such as cloud service providers and HPC centres, centralising workflow information and suitable HPC options. “Most projects like this result in proof of concepts. We wanted to go further and actually start implementing, to start iterating upgrades based on actual use,” notes Bricard. The result was an integration of the decision-making module into the pre-existing data science platform for HPC environments, OKA, improving the latter’s effectiveness. EAR, the system software for data centre energy management, has also now been integrated into the HEROES platform. “A number of HEROES’s components are already available for integration into current computing infrastructures through our commercial venture ‘Do it Now’,” adds Bricard. The project’s results, along with its partners’ expertise, are in prime position to contribute to the EuroHPC Joint Undertaking initiative, set up to develop a world-class supercomputing ecosystem in Europe.

Keywords

HEROES, artificial intelligence, machine learning, high performance computing, workflows, energy efficiency, supercomputing

Discover other articles in the same domain of application