Facebook Open Compute Project Shapes Big Data Hardware
Big data practitioners like Facebook, Goldman Sachs and Fidelity are setting the standards for cheaper, more efficient servers and systems from the likes of Applied Micro, AMD, Dell and Intel.
Big Data's Surprising Uses:
From Lady Gaga To CIA
(click image for larger view and for slideshow)
Facebook and the Open Compute Project (OCP) announced Wednesday that they've made huge strides toward the goal of setting standards for the most efficient server, storage and data center hardware available for scalable computing.
Facebook launched OCP 18 months ago hoping to crowdsource the problem of creating better hardware for high-scale computing. From its start with one member, Facebook, and 200 participants, the group now has more than 50 member companies and saw more than 2,000 participants attend this week's Open Compute Platform Summit in Santa Clara, Calif.
OCP says its standards promise to deliver hardware that is 24% more energy efficient and 38% more cost efficient, on average, than so-called commodity hardware. The group is working on specs for storage, motherboard and server design, racks, interoperability, hardware management, and data center design.
"We know what we need and we see the current and future challenges that the deluge of data, analytics, compute power and these massively scaled data centers are bringing forward," said Frank Frankovsky, VP of hardware design and supply chain at Facebook and chairman of OCP's Open Compute Foundation, which is modeled after the Apache Software Foundation.
Hardware vendors have also joined OCP and are designing hardware to its specs. AMD and Intel announced in May collaborative work with OCP members Fidelity and Goldman Sachs to develop new boards to suit their financial processing workloads, and both chip makers presented finished products at this week's event.
Intel has contributed specs to OCP for Silicon Photonic interconnect technologies that already surpass 100 gigabits per second -- nearly twice the speed of the fastest interconnect technologies currently available. Presenting on the technology, Intel CTO Justin Rattner said the products will soon be commercially available from Intel and would solve data I/O constraints both inside servers and in connecting servers within racks and racks within data centers.
"It has been a decade-long effort to create the technology, and it has enormous potential due to its combination of speed, low power consumption and low cost," Rattner said.
Dell also presented at the event, showing off Intel- and ARM-based OCP-compliant servers that will soon be commercially available. Other manufacturers working on or delivering OCP-compliant hardware and components include Applied Micro, Calxeda, Delta, Emerson and Hyve Solutions.
Many of the cost savings in OCP-standard products come down to two design themes: commonality and disaggregation. In an example of commonality, OCP unveiled a "common slot architecture" that will enable ARM and X86 chips to coexist on the same motherboard. The design will give end-user organizations flexibility to test and configure servers for best possible performance and then reconfigure as workloads change without facing proprietary design constraints. The spec is being supported by AMD, Applied Micro, Calxeda and Intel.
Disaggregation is about separating and modularizing storage, compute, interconnects, power, cooling and other components so companies can custom configure to their workload requirements.
This approach also supports smarter technology refreshes, so companies can swap out and replace quickly evolving components, such as CPUs, while keeping in service slowly evolving components, such as memory and network interface cards.
"You might be five generations behind on the processor while 80% of the other components of the server are still good," Frankovsky said, noting that monolithic designs don't let you replace dated components such as CPUs. "Smarter technology refreshes not only make sense from a financial perspective; think about how much less IT equipment is going to go into the waste stream."
While many OCP members are hyper-scale Internet companies, the point was made Wednesday that the group's efforts will benefit companies with mainstream computing requirements within a few short years.
"Hadoop, Hive, Hbase and other new platforms are accelerating the growth of big data adoption, so our big data challenges of today are everyone's big data challenges of tomorrow," said Jay Parikh, Facebook's VP of infrastructure engineering.
The Agile ArchiveWhen it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
2014 Analytics, BI, and Information Management SurveyITís tried for years to simplify data analytics and business intelligence efforts. Have visual analysis tools and Hadoop and NoSQL databases helped? Respondents to our 2014 InformationWeek Analytics, Business Intelligence, and Information Management Survey have a mixed outlook.