On Thursday, June 29th, Stephen Wheat, Senior Director of Intel's High Performance Computing Platform Office will be participating in the ISC Hot Seat session (Part II). He will be presenting his thoughts on the relevance of the Top500 list to the HPC community, from the perspective of the system vendor. His talk will be titled 'On the Relevance of Top500 to Product Planning.'

HPCwire recently got the opportunity to talk with Wheat about some of the issues involved with this topic, including the significance of the MP-Linpack metric for procurements, its future relevance to HPC, and the challenges of creating the next-generation HPC single-metric benchmark.

The MP-Linpack benchmark is probably the most widely cited metric for HPC performance. It is based on the Linpack software library that is used to calculate numerical linear equations for matrix and vector operations. But from a hardware perspective, what precisely is the benchmark measuring? And what kinds of real world HPC applications does it map to?

"Basically, there are two things it measures," says Wheat. "Primarily, it measures a single node's floating point performance at a sustained rate. So from a node perspective, it's all about the floating point engine. The second thing that's being measured -- and it's actually way, way second -- is the efficiency in communication of your interprocessor communication fabric -- InfiniBand, Myricom's Myrinet, Gigabit Ethernet, 10 Gigabit Ethernet, or whatever."

Wheat says if you just wanted to build a platform that maximized Linpack performance, you would construct a machine that does fast floating point arithmetic with a sufficient cache and memory configuration to keep the floating-point engine well fed with data.
 
While there are many applications for which Linpack performance can be a predictor, there are many more whose performance is much more complex and for which Linpack is not a suitable performance predictor. As the use of high performance computing spreads beyond its original big-science origins, Linpack's relevancy further diminishes.

This doesn't prevent HPC vendors from citing the Linpack results in their literature. But are system designers making decisions based on this metric?

"Not any more," says Wheat. "Historically, if we look back when this all started, an embedded floating point unit was a novelty. A floating-point co-processor was the closest thing you got, and that was optional. The benchmark encouraged the industry to provide ever-increasing floating point performance at ever-decreasing cost. Since that time, floating point capability has become ubiquitous."

As historically significant as the Linpack has been, its real popularity can be attributed to its use in the list of the Top500 fastest computers. The list is updated twice a year. But within the last several years, and especially within the last couple of years, the Top500 list has undergone a rapid turnover.

This is the result of a number of trends -- the commodization of multiprocessor systems (e.g., 64-bit x86 architecture), the pervasiveness of 64-bit floating point hardware in all high-end systems, and the rise of multi-core microprocessors.

 "Let's talk about the bottom 450 of the list and leave the top 50 as a special case," says Wheat. "In the bottom 450, you will find two kinds of systems listed. First are systems that have trickled down from the top 50 as new top 50 systems have come on line -- as long they're still operational. Second are predominately systems that would not have necessarily been designed for the high performance computing market segment. These systems are mostly made of industry-standard building blocks common to the enterprise market segment -- servers, workstations, and to some extent, desktop machines."

So the bottom 450, ninety percent of the list, will be dominated by commodity cluster systems. Some clusters will show up in the top 50 as well, but as you examine the top of the list, you will see a more varied mix of systems, including:

(1) Very large commodity cluster clusters (for example, Dell's PowerEdge Thunderbird system at Sandia National Labs),
(2) High-end SMP clusters, based on a variety of processor architectures (for example, SGI's Altix Columbia system at NASA/Ames), and
(3) Additional machines based on a variety of proprietary architectural elements (for example, IBM's Blue Gene/L at LLNL).

Wheat observes that the top ten systems, in particular, are in some way architecturally differentiated from the rest of the machines on the list. These machines may possess a specialized interconnect fabric, a highly scaled-out processor/memory architecture, a proprietary microprocessor technology or all three.
 
"At any particular point in time, we might see a temporary dominant representation of one type of system architecture or another, but at all times the top ten systems will be truly unique systems, " says Wheat. "As the top 10 of the Top500 continues to be viewed as leadership statements by vendors and end-users alike, there remains motivation to demonstrate the best HPC solutions that are likely to be system architectures specifically targeted at the HPC market segment. "

Not only is the Top500 undergoing rapid transformation, the relevance of the list to buyers of HPC systems is changing as well

"Procurements still find value in positioning on the Top500 list," says Wheat, "But I think it's certainly becoming less of a factor. We're seeing less and less reference to it in RFPs. There was a time when people said: 'Tell us how you perform on MP-Linpack.' That would be part of an RFP. While the direct references to Top500 positioning in RFP's is diminishing rapidly, the indirect references are decreasing a bit slower -- end-users still consider their positioning on the Top500 list."

The HPC Challenge (HPCC) suite has been offered as an alternative to the single Linpack metric. HPCC consists of seven different types of benchmarks that exercise a wider variety of system attributes than just Linpack performance.

Wheat believes the HPCC benchmark has the most value for high-end capability platforms or capacity platforms with capability features -- systems that would be subject to a relatively high-touch sales process.

"We're talking about relatively serious acquisitions," says Wheat. "And getting a good perspective of the utility of that platform is reasonably measured by the new [HPCC] suite of benchmarks."

But according to Wheat, it's much more difficult to apply HPCC to the volume HPC space, the 8-, 16-, and 32-processor systems. These systems are very price-sensitive and price/performance is the dominant metric. The customer is usually left with the task of interpreting the HPCC results for their application. And there is no agreed-upon weighting of the seven benchmarks in the HPCC suite.

"So for the large platforms, it's a major step forward, but for the high-volume systems it leaves much to be desired," says Wheat.

Application-based benchmarks, such as standard CFD or FEA codes, hold much promise for providing accurate performance metrics, since they map closely (or even exactly) to the workloads that the end-user will be running. Wheat says he sees RFPs specifying application performance in far greater numbers than he's seen in the past. In doing so he thinks users are making a statement that the synthetic benchmarks are too complex to map to their actual workloads.

To be sure, there are challenges here as well. What is a representative application code for a given market? How much effort will be required to run all these codes across an ever-growing number of platforms?

But as high performance computing is expanding into new uses, different metrics will have to be developed. These HPC systems are being deployed into market segments where an HPC solution didn't exist five years ago and where it wasn't even conceived of ten years ago. As a result, HPC is taking on many more personalities than any one metric can measure. For vendors to offer good solutions to their customers, they need to use far more inclusive measurements than the single Linpack benchmark. This will be the thrust of Wheat's Hot Seat presentation at ISC.

"With this growing, multi-faceted market for high performance computing, the challenges are huge," concludes Wheat. "We've gone beyond the proud and the few. We're all still proud, but we're not few anymore."