July 21, 2011
June 21, 2011 -- What good are supercomputers if you have to spend your whole time getting data in and out? Researchers at Oak Ridge National Laboratory (ORNL) are working to say goodbye to input/output (I/O) problems with their most recent upgrade of the Adaptive Input/Output System (ADIOS).
ADIOS grew out of a 2008 collaboration between the Oak Ridge Leadership Computing Facility (OLCF) and researchers from academia, industry, and national laboratories. Their goal was a system to get information in and out of a supercomputer efficiently.
“The measurement of success for us has always been ‘what percentage of time do you spend in I/O?’” said OLCF computational scientist Scott Klasky. ADIOS was inspired by Klasky’s work at the Princeton Plasma Physics Laboratory, where he noticed up to 30 percent of researchers’ computational time was spent reading and writing analysis files.
The open-source middleware is designed to help researchers maximize their allocations on leadership-class computing resources from wherever they may be. In essence, it creates more time for research by minimizing the time needed to read and write data to files, even if researchers are sending those files from thousands of miles away.
The previous release—version 1.2—improved usability by allowing users to construct new variables in their simulations while they run, simplifying the interface users work with and improving I/O performance. In fact, researchers using the middleware found the writing process substantially expedited, consuming only .06 percent of their computation time on average.
The biggest challenge for the newest release—version 1.3—was to improve reading efficiency, according to Qing Liu, a scientific researcher working in the Remote Data Analysis and Visualization branch at the National Institute for Computational Sciences and the leading developer of ADIOS.
“Our users were very happy that the writing was greatly improved [with version 1.2]. They could get 30 gigabytes per second for the writing performance, but when they tried to read, the performance was much lower,” he said. “There was a huge gap between writing and reading performance.”
Ray Grout, a researcher at the National Renewable Energy Laboratory, was one of the first people to test the latest update, using the S3D combustion code to study turbulent flows reacting to one another. Grout noted a huge increase in reading performance.
Klasky also added that ADIOS 1.4 will continue grass-root collaboration from computer science researchers to greatly reduce the problem of coping with large data on high-performance machines.
In addition to Klasky, Liu, and Grout, ADIOS 1.3 collaborators include William Tang, C.S. Chang, Weixing Wang, Stephane Ethier, and Zhihong Lin of the Princeton Plasma Physics Laboratory; Norbert Podhorszki, Jeremy Logan, Sean Ahern, Luis Chacon, Roselyne Tchoua, and Ricky Kendall of ORNL; Jackie Chen, Kenneth Moreland, Jay Lofstead, Ron Oldfield, and Todd Kordenbrock of Sandia National Laboratories; Xiaosong Ma, Nagiza Samatova, and Sriram Lakshminarasimh of North Carolina State University; Garth Gibson and Milo Polte of Carnegie Mellon University; Karsten Schwan and Greg Eisenhauer of Georgia Tech; Hasan Abbasi of the University of Tennessee–Knoxville; Seung-Hoe Ku of New York University; John Wu, Arie Shoshani, and Jinoh Kim of Lawrence Berkeley National Laboratory; Weikuan Yu and Yuan Tian of Auburn University; Manish Parashar, Ciprian Docan, and Fan Zhang of Rutgers University; and Julian Cummings of the California Institute of Technology.
For more information on ADIOS or to download the source, please visit http://www.olcf.ornl.gov/center-projects/adios/ —By Eric Gedenk
Source: Oak Ridge National Laboratory
10/30/2013 | Cray, DDN, Mellanox, NetApp, ScaleMP, Supermicro, Xyratex | Creating data is easy… the challenge is getting it to the right place to make use of it. This paper discusses fresh solutions that can directly increase I/O efficiency, and the applications of these solutions to current, and new technology infrastructures.
10/01/2013 | IBM | A new trend is developing in the HPC space that is also affecting enterprise computing productivity with the arrival of “ultra-dense” hyper-scale servers.
Ken Claffey, SVP and General Manager at Xyratex, presents ClusterStor at the Vendor Showdown at ISC13 in Leipzig, Germany.
Join HPCwire Editor Nicole Hemsoth and Dr. David Bader from Georgia Tech as they take center stage on opening night at Atlanta's first Big Data Kick Off Week, filmed in front of a live audience. Nicole and David look at the evolution of HPC, today's big data challenges, discuss real world solutions, and reveal their predictions. Exactly what does the future holds for HPC?