August 09, 2012
Cloud technologies have become integral to a number of services including video streaming, file sharing and social media to name a few. But when it comes to HPC applications, benefits shared by the previous examples don't always translate. This leaves some HPC users preferring private clusters sans virtualization, or straight-up supercomputers. Yet there have been quite a few high-profile accounts of users running HPC workloads on public clouds like Amazon Web Services and Microsoft Azure. Does this mean it's a good idea to run HPC applications in the cloud? Cluster Monkey's Thomas Eadline addresses this question, and points out which HPC applications are best suited for today's multi-purpose clouds.
In the HPC space, users prefer the following features:
While cloud systems are based on a different set of characteristics:
Eadline notes that where the two spaces overlap is in the need for scalability and redundancy though hardware independence, but otherwise these are very distinct worlds. HPC users have limited choice regarding what hardware they can use, and if they decide to use a service like Amazon, their resources will be virtualized. Furthermore, applications that lean heavily on a system's interconnect and call for high levels of I/O would suffer serious performance degradation.
Despite these differences, some HPC applications are good candidates for cloud. Also known as embarrassingly parallel (EP), these programs do not require high interconnect performance, and can run on 10 Gigabit Ethernet or even Gigabit Ethernet. Of these EP applications, there is a subset of programs that don't require especially high I/O rates. This group is typically well-suited for traditional cloud implementations.
Earlier this year, Cycle Computing ran one such application on Amazon's EC2 infrastructure. The program, called Glide, was used to test various potential cancer drugs for pharmaceutical firm Schrödinger and their research partner Nimbus Discovery. A 50,000-core virtual cluster named "Naga" was spun up for three hours and used the application to test 21 million drug compounds. The project processed over twelve-and-a-half years of compute work in just three hours. Total cost? $4,900. By comparison, had the end users built their own facility and purchased a comparable supercomputer, it would have cost between $20-30 million.
While Cycle's example demonstrates the extreme savings an HPC user can experience with cloud services, not all applications can be handled in the same fashion on a service like Amazon, but AWS is not the only cloud in town.
Some infrastructure providers like Penguin Computing, Softlayer and Zunicore focus on creating HPC-friendly cloud environments. These companies offer bare metal services, which allow end users to access resources without virtualization layers. Although these features are attractive for HPC applications, users need to analyze their requirements carefully before migrating to the cloud.
10/30/2013 | Cray, DDN, Mellanox, NetApp, ScaleMP, Supermicro, Xyratex | Creating data is easy… the challenge is getting it to the right place to make use of it. This paper discusses fresh solutions that can directly increase I/O efficiency, and the applications of these solutions to current, and new technology infrastructures.
10/01/2013 | IBM | A new trend is developing in the HPC space that is also affecting enterprise computing productivity with the arrival of “ultra-dense” hyper-scale servers.
Ken Claffey, SVP and General Manager at Xyratex, presents ClusterStor at the Vendor Showdown at ISC13 in Leipzig, Germany.
Join HPCwire Editor Nicole Hemsoth and Dr. David Bader from Georgia Tech as they take center stage on opening night at Atlanta's first Big Data Kick Off Week, filmed in front of a live audience. Nicole and David look at the evolution of HPC, today's big data challenges, discuss real world solutions, and reveal their predictions. Exactly what does the future holds for HPC?