'The Hobbit' Creates Big Data Challenge For Moviemaker

Big Data's Surprising Uses: From Lady Gaga To CIA

When it comes to manipulating massive amounts of digital information, few creative industries can match the data-intensive workloads of movie and TV production. And the advent of cutting-edge motion picture and HDTV technologies, including high frame rate (HFR) 3-D and 4K TV, will generate even more data.

"High-definition cameras and the new wave in which people create effects and try to enthrall the audience have resulted in a pretty big explosion in the amount of content produced per movie," said Jeff Denworth, vice president of marketing for DataDirect Networks (DDN), in a phone interview with InformationWeek.

DDN sells high-performance, scalable storage systems for a variety of industries, including film and post production. "We say that we've been in big data long before the term was even invented," said Denworth, who noted that 2013 will mark the company's 15th year in business.

[ What do you expect to see in big data this year? Read 5 Big Data Predictions For 2013. ]

The amount of data generated by movie and TV productions continues to rise. In 2009, director James Cameron's 3-D sci-fi epic "Avatar" was one of the first movies to generate about a petabyte of information, according to Denworth. (One petabyte equals 1,048,576 gigabytes.)

"Obviously, 3-D films weren’t anything new -- people had been releasing 3-D (movies) for a long time," said Denworth. "But Avatar was the first film to show that you could commercialize 3-D in a big way. So the market really rallied around the concept of 3-D technology."

And that meant a lot more data. "The 3-D effect on filmmaking will create anywhere from 100% to 200% more data per frame," Denworth said.

While the allure of 3-D movies has certainly cooled among moviegoers, filmmakers continue to try new technologies designed to heighten the cinematic experience.

For instance, "The Hobbit: An Unexpected Journey," which is currently in theaters, was shot in a new digital format called High Frame Rate 3-D (HFR 3-D). The format shows the movie at 48 frames per second (fps), twice the standard 24-fps rate that's been in place for more than 80 years. (Only select theaters with HFR equipment can show "The Hobbit" in HFR 3-D.) For viewers, HDR delivers a more realistic, immersive experience, according to "Hobbit" director Peter Jackson.

"The thing around 'The Hobbit' is that we're now crossing over a new threshold into the next wave of filmmaking, which will ultimately create even more data in the film-production process," said Denworth.

The Hobbit - An Unexpected Journey movie poster

And 48-fps may be just the beginning. Filmmaker Cameron, for instance, reportedly plans to use 60-fps cameras for his upcoming "Avatar" sequels. On the TV side of things, television manufacturers are starting to beat the drum for 4K TV -- or nearly four times the resolution of 1080p -- as the next big thing in home entertainment.

"Within the last five years, we've seen something in the order of an 8x increase in the amount of content being generated per every two-hour cinematic piece," Denworth said.

As every data aficionado knows, the term "big data" isn’t defined by volume alone. The velocity and variety of data are equally important, particularly when you're managing massive amounts of information.

"The data sizes are growing, but an 8x increase in (content) can't result in an 8x increase in the amount of time it takes to create a movie," said Denworth. As a result, filmmakers will need extremely high-frame-rate processing that grows with their data sizes.

"The commercial success of 'The Hobbit' could certainly wake up the whole industry to this new way of filmmaking, and that will create a lot of havoc for people who have lesser-capable storage platforms," added Denworth.

Cloud computing, virtualization and the mobile explosion create computing demands that today’s servers may not meet. Join Dell executives to get an in-depth look at how next-generation servers meet the evolving demands of enterprise computing, while adapting to the next wave of IT challenges. Register for this Dell-sponsored webcast now.