November 27, 2006
The National Center for Data Mining at UIC has won the HPC Bandwidth Challenge at SC06 in Tampa, FL, sponsored by Qwest. Nine institutions participated in the competition. NCDM won by sustaining a data transfer rate of 8 Gb/s over a 10 Gb/s link, with a peak rate of 9.18 Gb/s during the competition window. NCDM uses its own open source software products, UDT and SECTOR, to transfer large datasets efficiently at high speeds on optical networks.
This year the Bandwidth Challenge focused on a specific facet of networking: End-to-End achievement. Competitors were asked to fully utilize one 10 Gig path, end-to-end, disk-to-disk, from SC06 in Tampa back to their home institution, using the actual production network back home. Participants were required to realize, demonstrate and publish all the configuration, troubleshooting, tuning and policies used. The SC06 show floor was connected with the major U.S. research networks, specifically: Abilene, ESnet, NLR PacketNet, NLR FrameNet, and HOPI. The U.S. research networks provided transit for the international networks with which they peer.
NCDM transferred Sloan Digital Sky Survey Data (SDSS) between the SC06 show floor in Tampa, Fl and its lab in 4223 SEL at the University of Illinois in Chicago. It used SECTOR, the newly developed distributed data space management system. SECTOR transparently manages the file locating and data moving, while the NCDM developed UDT software is used for the actual data transfer. The data transfer was disk to disk over one 10 Gb/s shared routed link between SC06 and UIC, via StarLight.
SDSS is systematically mapping a quarter of the entire sky to produce a detailed image of it and determine the positions and absolute brightness of more than 100 million celestial objects. When completed, it will provide a three-dimensional map of nearly a million galaxies and quasars. The total survey information is about 15 terabytes and rivals the information content of the Library of Congress. As the survey progresses, the data are released to the scientific community and the general public in annual increments.
Bandwidth Challenge Competitors: Winner
National Center for Data Mining (NCDM) at UIC, Northwestern Univ., Johns Hopkins Univ., "Transporting Sloan Digital Sky Survey Data Using SECTOR."
1) CalTech, CERN, Univ. of Florida, Univ. of Michigan, "High Speed Data Gathering, Distribution and Analysis for Physics Discoveries at the Large Hadron Collider."
2) Indiana Univ., Pittsburgh SuperComputing Center, Oak Ridge National Laboratory, "All in a Day's Work: Advancing Data Intensive Research with the Data Capacitor."
Japanese Aerospace Exploration Agency; Pacific Northwest National Laboratory; Purdue University; Internet2, Univ. of Washington; Texas A&M Univ., Univ. of Delaware; Univ. of Tokyo.
10/30/2013 | Cray, DDN, Mellanox, NetApp, ScaleMP, Supermicro, Xyratex | Creating data is easy… the challenge is getting it to the right place to make use of it. This paper discusses fresh solutions that can directly increase I/O efficiency, and the applications of these solutions to current, and new technology infrastructures.
10/01/2013 | IBM | A new trend is developing in the HPC space that is also affecting enterprise computing productivity with the arrival of “ultra-dense” hyper-scale servers.
Ken Claffey, SVP and General Manager at Xyratex, presents ClusterStor at the Vendor Showdown at ISC13 in Leipzig, Germany.
Join HPCwire Editor Nicole Hemsoth and Dr. David Bader from Georgia Tech as they take center stage on opening night at Atlanta's first Big Data Kick Off Week, filmed in front of a live audience. Nicole and David look at the evolution of HPC, today's big data challenges, discuss real world solutions, and reveal their predictions. Exactly what does the future holds for HPC?