March 28, 2013
MARKHAM, Ontario, March 28 — Univa, the Data Center Automation Company and home of Grid Engine, today announced the release of Univa Grid Engine Version 8.1.4 with 46 new updates versus the previous version. Univa Grid Engine, the most widely deployed, distributed resource management software platform, has gained in excess of 525 updates since Univa took over the management of Grid Engine.
“Our latest Univa Grid Engine version 8.1.4 has been completely customer driven and is the largest update of the last 10 months,” said Fritz Ferstl, CTO Univa Corporation and father of Grid Engine. “We are leading the industry right now in converged infrastructures supporting big data and big compute, and our customers rely on Univa Grid Engine to manage mission-critical applications – so we make sure to always stay close to them in order to support their needs.”
Univa supports the new breed of enterprise and big data applications its customers depend on. Commercial enterprises in the industries of Industrial Manufacturing, Oil and Gas, Energy, Life Sciences, Biology and Semiconductors have increasingly turned to Univa because they need enterprise-grade support and functionality for their mission-critical computing capacity.
Of the 46 Key features updates, the major updates include:
Univa Grid Engine is engineered, tested and globally supported by Univa – with an average of one fix per day – and has been proven to perform in production environments such as:
Use Case - Archimedes, Inc.
Archimedes, Inc. is a healthcare modeling organization. Its core technology - the Archimedes Model - is a clinically realistic, mathematical model of human physiology, diseases, interventions and healthcare systems. The organization’s founding mission is to revolutionize the quality and efficiency of the global healthcare industry by using mathematical data.
The journey began in 1992 when David Eddy, M.D., Ph.D., approached the Southern California region of Kaiser Permanente (KPSC) clinical and health plan leaders with a vision. Eddy wanted to build a clinically realistic, large scale simulation model of human physiology, populations, and health care systems. The objective was to provide decision makers at KPSC with quantitative information about the outcomes they could expect from different clinical and administrative policies and programs.
After 20 years and acclaim in multiple medical journals, Archimedes is now a privately held subsidiary of Kaiser serving a global healthcare clientele. The Archimedes Model is relied upon to answer complex, real world questions for health plans, health systems, medical groups, pharmaceutical companies, researchers, and other organizations in the United States and Europe.
Use cases for the simulated data have expanded to wider applications such as award winning products including IndiGO, which delivers customized risk assessment and clinical-decision support information outlining health options directly to individual patients.
Scaling resources to meet growing demands
Building the infrastructure needed to simulate, produce, and quantify the big data processed by the Archimedes Model was a resource and scaling challenge.
When Archimedes Model was built and housed within Kaiser Permanente, it ran on Univa’s Grid MP, a distributed computing software. This allowed Archimedes to optimize their existing infrastructure of thousands of idle PCs throughout Kaiser Permanente that were capable of processing the load in the early years. All simulations at that time were run by Archimedes’ scientists for healthcare and life science clients, completed as consulting projects. The results were delivered to clients via an Excel spreadsheet at the end of the engagement, which typically lasted several weeks.
As the demand for more simulations increased, the cost-effectiveness of this system diminished. Archimedes invested in a dedicated cluster of approximately 50, multi-core, rack mounted servers. The cluster still worked seamlessly with Grid MP, however, the pain point came when Archimedes began to productize their offering. At that point, the amount of time it took to quantify the data for external clients became an issue.
To efficiently service a broader clientele, Archimedes’ developers created a web interface for the Archimedes Model, named ARCHeS, that allowed clients to define a simulation themselves, run it, and view the results without the need to engage in a consulting project with Archimedes.
Up until this time, the simulator results would be loaded into an internal Postgres database called DataDyve. Then the data would be pulled out and aggregated serially before being loaded into the Archimedes Outcomes Analyzer (AOA) Postgres DB. The ARCHeS simulation results are large: approximately 1 Gig. They contain hundreds of data points for eleven thousand patients per year, over a 20 year period.
“We had done a lot of work to speed up the simulator, so the bottleneck was aggregating and loading the data that came out of the simulator into the analysis tool,” says Katrina Montinola, Archimedes’ vice president of engineering. “When the data comes out of the simulator you have to aggregate it in different ways. There are about 100 biomarkers, outcomes, and treatments per patient that need to be aggregated for each trial arm and each subpopulation. A simple load would take many, many hours, so we quickly realized we needed a better solution.”
Archimedes then developed new software called Aggregator, which aggregates the simulator data and makes calculations at a faster pace. Aggregator was built using the open-source big data solution, Hadoop, a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.
This approach solved the data loading and analysis time constraints, and enabled Archimedes to deliver results of ARCHeS simulations automatically – within 24 hours – instead of the weeks or months,it took when clients could only complete simulations via a consulting project. This advancement, however, led to another challenge.
“We already had a dedicated cluster for our simulator that used Grid MP. In order to take Aggregator live, we were looking at creating a second dedicated cluster for Hadoop,” says Montinola. “That would mean doubling our costs for hardware, housing space, IT support, and power supplies.”
Consulting with a trusted partner
After several years of successfully partnering with Univa using Grid MP, the first step towards a solution was to engage consultants from Univa. The Univa consultants were tasked with the challenge of taking Aggregator live with Hadoop without requiring a second cluster that would double investment and operational costs.
The fact that the same hardware works for both Aggregator and the Simulator, made it logical that a solution using one cluster should be possible. Additionally, the programs did not require the full capacity of two designated clusters, so doubling up was a waste of resources.
In order to add the new components and equip the system to continue to scale, Univa consultants suggested Univa Grid Engine. Univa Grid Engine is a distributed resource management (DRM) system used to build large computer infrastructures for processing massive volumes of workload. It harnesses the combined computing power of desktops, servers and clouds in an easy-to-administer environment.
A key benefit for Archimedes migrating to Univa Grid Engine is that scheduling policies can be applied to all computing jobs – including Hadoop – submitted to the cluster; ensuring high-priority jobs are completed on time while simultaneously maximizing utilization of all cluster machines to handle both programs. Additionally, Grid Engine is quality controlled and completely supported by Univa – eliminating the risk attached to open-source Grid Engine.
“With Grid Engine, it’s just a much cleaner way to migrate,” says Montinola. “If we didn’t have Grid Engine it would be a major investment to go live with Aggregator and Hadoop. Migrating to Univa Grid Engine removed all the pain and costs.”
The historical ease of use with Grid MP also contributed to the decision. During the years of using Grid MP, all internal IT support to the system was handled by one employee as a fraction of his job. Montinola felt confident that moving to another Univa supported solution would assure internal IT costs remained low.
Cost Savings Up to 50% realized in Phase One of Migration
Archimedes began the migration to Univa Grid Engine in September 2012. Currently the simulator continues to run on Grid MP-based in-house cluster. The results of the simulator are aggregated via Hadoop-based Aggregator using Univa Grid Engine and then loaded into a Postgres database for viewing and analyzing.
Archimedes estimates they have already cut costs on hardware and software up to 50% through adopting Univa Grid Engine in the first step of the migration, without any compromise to overall performance. One reason for the initial savings is that Aggregator now runs on smaller, commodity servers with Univa Grid Engine instead of big, multi-core servers necessary to run DataDyve.
“By integrating Univa Grid Engine, adopting a Hadoop-based solution was much more cost effective, therefore maximizing the existing infrastructure to its full potential,” says Montinola. “So for us, it’s a perfect solution.”
The ultimate goal is a full migration of the Simulator to Univa Grid Engine so that both can use the same cluster. This first step sets the foundation for a more scalable solution that can handle the expected growth of traffic Archimedes anticipates.
Montinola projects from this point forward that the continued implementation of Univa Grid Engine will allow the company to easily scale without any additional cost. Eliminating a second dedicated cluster for Aggregator eradicates significant costs in hardware, storage space, IT support, and electricity that would have been necessary without integrating the entire system to Univa Grid Engine.
“With Univa Grid Engine, we can continue to scale and decrease deliverable time to our clients,” says Montinola. “Univa has supported us through the migration and will continue to do so into the future as we continue to grow. If I went to another company that was using purely an open-source Grid Engine, I would take Univa with me to assure this kind of flexibility and security. I know Univa has my back.”
About Archimedes Inc.
Archimedes Inc. is a healthcare modeling organization. Its core technology - the Archimedes Model - is a clinically realistic, mathematical model of human physiology, diseases, interventions and healthcare systems. The Model is continually validated by comparing the results of simulated trials to the results of real multi-national clinical trials and cohort studies.
Through innovations such as IndiGO and ARCHeS, Archimedes helps people understand the implications of their decisions and for the last 15 years has been relied upon to answer complex, real world questions for health plans, health systems, medical groups, pharmaceutical companies, researchers, and other organizations in the United States and Europe. Archimedes, a Kaiser Permanente Innovation, is based in San Francisco, California.
About Univa Corporation
Univa, the Data Center Automation Company, is the leading provider of automation and management software for computational and big data infrastructures. Our products and global enterprise support give our customers the power to manage their entire computational space, no matter how big or where it is deployed. Many of the leading brands in the world depend on Univa’s unsurpassed expertise, and premier services and support. Univa is headquartered in Hoffman Estates, Ill., USA, with offices in Markham, Ontario, Canada; Austin, Texas, USA and Munich, Germany.
Source: Univa Corporation
10/30/2013 | Cray, DDN, Mellanox, NetApp, ScaleMP, Supermicro, Xyratex | Creating data is easy… the challenge is getting it to the right place to make use of it. This paper discusses fresh solutions that can directly increase I/O efficiency, and the applications of these solutions to current, and new technology infrastructures.
10/01/2013 | IBM | A new trend is developing in the HPC space that is also affecting enterprise computing productivity with the arrival of “ultra-dense” hyper-scale servers.
Ken Claffey, SVP and General Manager at Xyratex, presents ClusterStor at the Vendor Showdown at ISC13 in Leipzig, Germany.
Join HPCwire Editor Nicole Hemsoth and Dr. David Bader from Georgia Tech as they take center stage on opening night at Atlanta's first Big Data Kick Off Week, filmed in front of a live audience. Nicole and David look at the evolution of HPC, today's big data challenges, discuss real world solutions, and reveal their predictions. Exactly what does the future holds for HPC?