Expanding HPC to cloud computing and big data
An increasing number of High-Performance Computing (HPC) users are combining HPC with big data and cloud infrastructure to apply common management platforms and paradigms across them. In response to this demand, the EU-funded BRIGHT-BEYOND HPC project expanded Bright Computing’s capabilities. This Dutch company provides comprehensive software solutions for the provision and management of HPC clusters and OpenStack private clouds. “We recognise that the deployment and use of these advanced technologies is not only expanding, but converging, and this project allowed Bright Computing to deliver a powerful, integrated, single-pane-of-glass management platform that helps IT organisations gain visibility, control and peace of mind,” says Dr Matthijs van Leeuwen, Chief Strategy Officer at Bright Computing. The project aimed to solve the cluster management problem in HPC, big data/Hadoop and OpenStack cloud computing by developing and bringing Bright Beyond HPC to market readiness. “Bright Computing’s Bright Cluster Manager product has a very strong presence in the HPC market,” explains Dr van Leeuwen. “The goal of this project was to extend the Bright Cluster Manager and make it usable in two important emerging markets, namely, cloud computing and big data.” Cloud computing As to the first, the Bright Cluster Manager was used to manage large clusters of computers that, together, solve complex scientific, technical, engineering or financial problems. However, the product’s functionality was limited as the clusters managed by the product consisted of computers physically located inside the company or organisation. With the emergence of cloud computing, stakeholders were doing more and more of their computing ‘in the cloud’. OpenStack is a standard cloud management software that is widely supported by industry for setting up public and private clouds. ‘Being able to extend an on-premise Bright cluster to an OpenStack cloud, both internal to an organisation and external, is a major achievement of this project,’ says Dr van Leeuwen. According to him, this allows a user to combine clusters of computers inside an organisation with computers available in a cloud. Although the user primarily does this using their own computers, in instances of high computing needs, they also have access to cloud computers, including Amazon Web Services (AWS) and Microsoft Azure Clouds (in addition to private and selected public OpenStack clouds) Big data Likewise, with big data, large amounts of data have traditionally been analysed by clusters of computers. However, the management and application software for big data is different from what is used for HPC clusters in that it is geared towards moving large amounts of data and being able to search through this data (as opposed to doing lots of calculations). The main tool used for managing big data clusters is Apache Hadoop, an Open Source software. By integrating Hadoop into the Bright Cluster Manager, organisations can now manage clusters of systems that include both HPC- and big data-oriented computers. This can be used, for instance, by research organisations that analyse experimental data in a Hadoop cluster and compare that to theoretical models calculated on HPC computers. “Industry is often reluctant to use open source software due to the lack of professional support behind it,” says Dr van Leeuwen. “BRIGHT-BEYOND HPC adds industrial strength support to this software, so the Bright Cluster Manager can now be used by organisations that want to manage big data clusters but also require a certain amount of support.” The next level As a partner in the European Technology Platform for High Performance Computing (ETP4HPC), the BRIGHT-BEYOND HPC project plays an important role in taking HPC to the next level in Europe. Bright Cluster Manager software version 8.1 is that next level and is currently being rolled out to existing and new customers. In the meantime, work goes on, as project researchers turn to new developments, including Artificial Intelligence.
Keywords
BRIGHT-BEYOND HPC, Bright Computing, cloud computing, big data, high performance computing, European Union EU
