Supercomputers Unleash Big Data's Power
Here are six reasons why established companies, and even startups, are using supercomputing resources, and why your IT organization may want to consider such options to meet your big data and business analytics needs.
![](https://eu-images.contentstack.com/v3/assets/blt69509c9116440be8/blt20cbe03c39a88522/64cb5386e5c09458baefadf7/data-bright-66479703_thumbnail.jpg?width=700&auto=webp&quality=80&disable=upscale)
Manufacturers, logistics companies, pharmaceutical companies, and energy companies have something in common: They're using supercomputers to push the limits of research and discovery, as well as to answer questions that may not be possible or practical to answer using other means.
Organizations are using the cloud and PCs to solve the problems yesterday's supercomputers once solved. Cloud computing has also grown to encompass High Performance Computing or HPC in the cloud, and the providers of those products, services, and solutions are targeting research and scientific communities that have traditionally used supercomputers. As cloud solutions and supercomputers continue to advance, their use is not necessarily mutually exclusive. Some companies are working with universities and national labs to access the most powerful resources available, including companies that own their own supercomputers.
"We've got to get [companies] from where they are today to where we are in the massively parallel, multi tens of petaflop capabilities," said Jeff Nichols, associate laboratory director for Oakridge National Lab's (ORNL) computing and computational sciences, in an interview. "Companies in the automotive industry, the airline industry, the energy space, and science space want to work with us to solve their big science questions."
ORNL and the Joint Institute of Computational Sciences (JICS) -- which is a joint partnership between ORNL and the University of Tennessee -- each have a number of national class and leadership class computing resources with different architectures that allow them to solve different kinds and scales of problems. Titan is ORNL's big gun and the second most powerful supercomputer on the planet today. It's a Cray XK7 27 petaflop machine with 299,008 16-core AMD Opteron CPUs, 18,688 NVIDIA Tesla K20 GPU accelerators and 710 terabytes of total system memory.
In addition to having a wide range of hardware at its disposal, JICS optimizes packaged and proprietary software so it can run efficiently on supercomputers or in the cloud. In addition, JICS has a staff of 20 Ph.D.s in physics, chemistry, computer science, math and other fields who speak machine language. They help organizations understand what is possible to achieve computationally. And, because they're scientists, they also help companies advance the state of the art.
"As you begin to use computing, you start asking harder questions and need more computing. Novices hit a wall right away," said Tony Mezzacappa, JICS director and chair of theoretical and computational astrophysics and astronomy at the University of Tennessee, in an interview. "Knowledgeable folks [who] know what they're doing know how to scale across the full set of nodes on their machine, but eventually they hit a wall too. They want to solve a problem that requires more memory or more compute power to execute in a reasonable amount of time."
Here are six reasons why established companies and even startups are using supercomputing resources, and why your IT organization may want to consider it to meet your big data and business analytics needs.
If a dataset is terascale or you're combining multiple terascale datasets, the data may too large to fit into memory. One way to deal with it is to break it down into smaller pieces and analyze the pieces individually. While it's possible to do many kinds of analysis on the fragments of data, more types of analysis can be done when all of the data is available in memory and the analyses can be executed faster. It is also possible to ask more types of questions, expand the scope of discovery, and do more correlations.
"If I want the right answer to a question, bringing in all the needed data, bringing it in fully, being able to query it fully, best enables me to come to the right conclusions," said JICS director Tony Mezzacappa in an interview.
One example is identifying fraudulent healthcare claims, which involves massive amounts of data. By bringing all the data into memory, it's possible to discover patterns that are not apparent if the claims are viewed in isolation, according to Ken Gilbert, director of Business Analytics at the University of Tennessee office of research and economic development, in an interview.
Companies manufacturing everything from candy to tires are using supercomputers, and the computational scientists at JICS, to help fine-tune their modeling capabilities. For a tire company, that means modeling the manufacturing process as well as the individual components within each phase of the process, including the rubber and polymers and other things required to produce a tire. It's a very layered and complex process that leads to several business decisions including the products that will be used to produce the tire, the process developed to make the tire, how safe and reliable the tire is and how well the tire will sell. Uncertainty quantification is critical since lives depend on safe, reliable tires.
"There are different types of data, including simulation data and conclusions of the analyses based on all the data you have [which might include] some experimental data," said JICS director Tony Mezzacappa. "You bring experimental data into the models as input, and then your models will generate further output that tells you what happens under certain circumstances given that experimental input and that model of the tires. So you have error bars around your experimental data, around your simulation data, and you have methods in uncertainty quantification where you purposely vary the inputs and outputs, and certainly how you conduct the simulation, and see how the output varies. Do you end up with a tire that is equally safe or not?"
Large, established companies have been using supercomputers for decades but some startups also want to use them. For example, Atomwise, a startup determined to change the way drugs are discovered and developed, used an IBM supercomputer to screen 7,000 drugs that might be effective in treating Ebola. Four months after initiating its virtual search, it discovered evidence of two possibilities. JICS and ORNL are being tapped by startups.
Large, established companies have been using supercomputers for decades but some startups also want to use them. For example, Atomwise, a startup determined to change the way drugs are discovered and developed, used an IBM supercomputer to screen 7,000 drugs that might be effective in treating Ebola. Four months after initiating its virtual search, it discovered evidence of two possibilities. JICS and ORNL are being tapped by startups.
-
About the Author(s)
You May Also Like