The Portland Group
CSCS Top Right Frontpage

Since 1986 - Covering the Fastest Computers
in the World and the People Who Run Them

Language Flags

Visit additional Tabor Communication Publications

Enterprise Tech
HPCwire Japan

When Is Cloud the Right Choice for HPC?

Cloud technologies have become integral to a number of services including video streaming, file sharing and social media to name a few. But when it comes to HPC applications, benefits shared by the previous examples don't always translate. This leaves some HPC users preferring private clusters sans virtualization, or straight-up supercomputers. Yet there have been quite a few high-profile accounts of users running HPC workloads on public clouds like Amazon Web Services and Microsoft Azure. Does this mean it's a good idea to run HPC applications in the cloud? Cluster Monkey's Thomas Eadline addresses this question, and points out which HPC applications are best suited for today's multi-purpose clouds.

In the HPC space, users prefer the following features:

  • Bare metal infrastructure
  • Tuned hardware and storage
  • Batch Scheduling
  • Userspace Communication (Applications bypassing the OS kernel)

While cloud systems are based on a different set of characteristics:

  • Immediate Access
  • Large Capacity
  • Available Software
  • Virtualization
  • Service Level Agreements

Eadline notes that where the two spaces overlap is in the need for scalability and redundancy though hardware independence, but otherwise these are very distinct worlds. HPC users have limited choice regarding what hardware they can use, and if they decide to use a service like Amazon, their resources will be virtualized. Furthermore, applications that lean heavily on a system's interconnect and call for high levels of I/O would suffer serious performance degradation.

Despite these differences, some HPC applications are good candidates for cloud. Also known as embarrassingly parallel (EP), these programs do not require high interconnect performance, and can run on 10 Gigabit Ethernet or even Gigabit Ethernet. Of these EP applications, there is a subset of programs that don't require especially high I/O rates. This group is typically well-suited for traditional cloud implementations.

Earlier this year, Cycle Computing ran one such application on Amazon's EC2 infrastructure. The program, called Glide, was used to test various potential cancer drugs for pharmaceutical firm Schrödinger and their research partner Nimbus Discovery. A 50,000-core virtual cluster named "Naga" was spun up for three hours and used the application to test 21 million drug compounds. The project processed over twelve-and-a-half years of compute work in just three hours. Total cost? $4,900. By comparison, had the end users built their own facility and purchased a comparable supercomputer, it would have cost between $20-30 million.

While Cycle's example demonstrates the extreme savings an HPC user can experience with cloud services, not all applications can be handled in the same fashion on a service like Amazon, but AWS is not the only cloud in town.

Some infrastructure providers like Penguin Computing, Softlayer and Zunicore focus on creating HPC-friendly cloud environments. These companies offer bare metal services, which allow end users to access resources without virtualization layers. Although these features are attractive for HPC applications, users need to analyze their requirements carefully before migrating to the cloud.

HPCwire on Twitter


There is 1 discussion item posted.

KU Leuven benchmarks Machine Learning in the cloud
Submitted by eerola on Aug 10, 2012 @ 3:01 PM EDT

I agree that not all workloads/ applications are suitable for loosely coupled environments, which 99% of current cloud services are.

On the other hand, even if cloud as an infrastructure is a great platform for EP problems, cloud itself does not make the computing too much easier compared to a conventional infrastructure.

In my opinion, cloud provides economics of scale advantages, but if you want to do efficient EP, the platform needs to be highly scalable, and this is missing form more-or-less all platforms (physical or virtualized). If developing and using the platform requires a degree in computer scienes, of if the environment requires a full-time system administrator to maintain the nodes, it does not make too much difference if the environment is on-premise or in the cloud. The complexity will eat other benefits.

Katholieke Universiteit Leuven published a nice paper paper last week, where they had benchmarked Machine Learning algorithms in the cloud.

Post #1

Join the Discussion

Most Read Features

Most Read Around the Web

Most Read This Just In

Most Read Blogs

Sponsored Whitepapers

Breaking I/O Bottlenecks

10/30/2013 | Cray, DDN, Mellanox, NetApp, ScaleMP, Supermicro, Xyratex | Creating data is easy… the challenge is getting it to the right place to make use of it. This paper discusses fresh solutions that can directly increase I/O efficiency, and the applications of these solutions to current, and new technology infrastructures.

A New Ultra-Dense Hyper-Scale x86 Server Design

10/01/2013 | IBM | A new trend is developing in the HPC space that is also affecting enterprise computing productivity with the arrival of “ultra-dense” hyper-scale servers.

Sponsored Multimedia

Xyratex, presents ClusterStor at the Vendor Showdown at ISC13

Ken Claffey, SVP and General Manager at Xyratex, presents ClusterStor at the Vendor Showdown at ISC13 in Leipzig, Germany.

HPCwire Live! Atlanta's Big Data Kick Off Week Meets HPC

Join HPCwire Editor Nicole Hemsoth and Dr. David Bader from Georgia Tech as they take center stage on opening night at Atlanta's first Big Data Kick Off Week, filmed in front of a live audience. Nicole and David look at the evolution of HPC, today's big data challenges, discuss real world solutions, and reveal their predictions. Exactly what does the future holds for HPC?


Stay informed! Subscribe to HPCwire email Newsletters.

HPCwire Weekly Update
HPC in the Cloud Update
Digital Manufacturing Report
HPCwire Conferences & Events
Job Bank
HPCwire Product Showcases


HPC Job Bank

Featured Events

HPCwire Events