Big Data // Big Data Analytics
Commentary
11/8/2012
11:06 AM
Eric  Lundquist
Eric Lundquist
Commentary
Connect Directly
Twitter
RSS
E-Mail
50%
50%

Nate Silver's Big Data Lessons For The Enterprise

Big data analytics have proven their mettle in predicting baseball and election success. How can you make them work for your business?

 Big Data Talent War: 7 Ways To Win
Big Data Talent War: 7 Ways To Win
(click image for larger view and for slideshow)
Only one number mattered to the data analysis aficionados watching the presidential election results on Tuesday night. That number, 538, is Nate Silver's blog (now under the New York Times auspices). Silver, by using predictive analytics applied against a range of polling and related data, hit a perfect 50 for 50 in his state-by-state predictions.

Remember, this was in a race where the pundits for the losing side were confident in their landslide-win predictions, and pundits on the winning side were predicting a razor-thin victory margin where vote counts and recounts could stretch for weeks. While Silver was spot-on, the pundit stars were overwhelmingly wrong.

Silver is now a media star with sales of his recent book, The Signal and the Noise: Why So Many Predictions Fail -- But Some Don't, up 850%. Does Silver's success mean that comments about aggregated analytics, gamma distributions and sum-of-squares formulas will now become de rigueur on the Washington cocktail circuit? That all those math teachers telling toiling high school students that statistics really will be useful in real life will finally be vindicated? That all the talk about "big data as the next big thing" will actually prove to be more than passing buzz?

Let's talk about that third item. In my opinion, Silver's success is less about big data (which is quickly being overused into meaninglessness) and is more about rigor, innovation and looking outside your business confines to find inspiration. And baseball -- baseball is important also.

[ We're not lacking facts, figures or tools -- so why are we having such trouble wrangling big data? Learn 6 Lies About Big Data. ]

Silver is not secretive about his methodology. You can read it here. (By the way, the Times has a paywall which allows for 10 free articles; if you're worried about hitting it, follow these tips for an end run.) The methodology's seven steps are laid out in detail, but just following them will not make you a political-prediction superstar. Silver provides the recipe, but not the measure of ingredients used in each step.

His weighting to individual polls is part of the secret sauce, or more precisely the end result of the scientific method applied to statistics. That secret sauce is derived through trial, error and adjustment: the same steps scientists have used for years to conduct experiments. Gut checks are replaced by rigor, thinking outside the normal confines and fine tuning. The process is not fast, but measured in years, and supports the concept that computers can aid in prediction, but cannot totally usurp the intelligent, curious human at the keyboard.

Global CIO
Global CIOs: A Site Just For You
Visit InformationWeek's Global CIO -- our online community and information resource for CIOs operating in the global economy.

I've found one of the best explanations of Silver's methods to be in the Wikipedia entry about his role as a "practical statistician." "The practical statistician first needs a sound understanding of how baseball, poker, elections or other uncertain processes work, what measures are reliable and which not, what scales of aggregation are useful, and then to utilize the statistical tool kit as well as possible."

And that brings me to baseball. Silver's first rise to fame came in 2003 when he developed the PECOTA (player empirical comparison and optimization test algorithm) system for predicting hitters' and pitchers' future outlook. Is baseball the same as politics? Yes and no. Yes, in that there is an old-guard baseball scouting system that relies on gut feel, and a new guard melding statistics and new ways of player measurement. This old guard/new guard division was best laid out in the book and movie Moneyball.

So what do politics and baseball have to do with your business? The goal of all the discussion around big data and data analysis is, as I've argued, not to make the wrong decision faster, but to develop the best decision at the right time and deliver the information to the people that most need the information. In an Information Week column Wednesday, Tony Byrne argued small data beat big data in the presidential election.

Call it business intelligence, data analysis or predictive analytics, IT's role here is to provide a foundation for your company to make the right decisions. Those decisions might be what to charge passengers for seats on a flight, how much to charge to for a season ticket or how many widgets to create to strike the right balance among manufacturing costs, inventory and availability. These decisions are fundamental to business success.

There is no magic to Silver's methods. There is hard work, a willingness to make mistakes and adjust, and a realization that the common wisdom is sometimes not wisdom at all. Innovation can happen in strange places and Silver has shown that the buttoned-down world of statistics doesn't have to be that buttoned down at all.

Here's my advice: Pay attention to Silver's process, but be equally assertive in looking at how your company operates. You will probably find lots of silos of activity where each group tends to use the same measures and methods year after year. Your job is to think outside the box, think like a customer and consider all the influences that would go into a purchasing decision. Understand the influences and you will be on your way to developing a prediction model that actually works for your business.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Page 1 / 2   >   >>
Cindi Howson, BIScorecard
50%
50%
Cindi Howson, BIScorecard,
User Rank: Apprentice
11/12/2012 | 3:12:51 PM
re: Nate Silver's Big Data Lessons For The Enterprise
Great article, Eric.

That Silver got it right while many political pundits did not does remind me of Moneyball. Your points about "rigor, innovation and looking outside your business confines to find inspiration" are important. So much about BI has been about the basics of data access and looking at numbers the same way. I suspect the next phase of business success will be based on those willing to look at data in new ways ... and sometimes new data.
Regards,
Cindi
MedicalQuack
50%
50%
MedicalQuack,
User Rank: Moderator
11/12/2012 | 5:12:44 AM
re: Nate Silver's Big Data Lessons For The Enterprise
Let's tie in a bit of luck here too. He does admit and rightly so to the huge error factors in polls and some of predicting is keeping up on credible news...and sometimes we don't get enough of that. We are a naive society with Algorithms and Formulas and I've been writing about for around 3 years now I do research a lot of this in healthcare, plus I used to write code which is very closely related to math.

Actually on Wall Street the Quants write the formulas and the technicians do the code work after words to query and sometime bury algorithms in code modules as software is nothing but a bunch of algorithms working together, in the words of Bill Gates. We get "Algo Duped" as I call it and have had a few professors agree along with folks from the National Institute of Statistical Science that keep telling me to keep making noise.

Here's a series of 4 videos at the link below done folks smarter than me and if you watch all 4 you will soon see that behavioral predictive analytics is going to end up being one huge are for fraud and again gaining money, algorithms move money and they are not very stable. It is what it is. Take the case in Italy where the scientists were guilty of manslaughter for not predicting the earthquakes, naive public put way too much value in their mathematics and formulas as we are not there...yet..and who's to say that someday it could get closer but not for a long time on that item. I live in southern California and believe me if someone could have figured out how to sue Cal Tech for not predicting an earthquake, it would have been done here with all the smart folks that live here <grin>.

I see it all the time and 3 years ago I wrote an article asking if the US needs a Department of Algorithms, nobody checks the math and just like there are 50 ways to leave your lover there's at least that if not more than 50 ways to snow the public:)

Here's Algo Duping 101...

http://www.ducknet.net/attack-...

On a related topic, and some of the same videos are also here, I am up to 46 chapters now on the Attack of the Killer Algorithms as well. In healthcare, United is a good example of using code and mathematics to short pay doctors and hospitals going back 15 years, long time and the formulas worked well for them and made tons of money. They worked so well that Blue Cross, Health Net and the other big guys licensed the customary fee data base for their claim payments...it's been around a long time but is now accelerated.

Again, predictive analytics if not regulated and checked for accurate code and formulas stands to be one of the largest technology rips on the American consumer for sure.

http://ducknetweb.blogspot.com...

I might also add I was not really happy with the Nobel prize for the algorithms used for the medical students finding where they should serve their internship either. It's the same as what's used for dating sites so I didn't see any awards for Match.com or Plenty of Fish handed out:) What he did was a good thing of course but there were better choices I felt instead of just throwing an award out for an algorithm as programmers write them day in and day out and ones of bigger value than that one. The Clinton Foundation or the Clinton's themselves, a much better choice as it also adds some humanism along the line, which we need with everyone getting Algo Duped which leads to some very skewed perceived realities when they are really not there. </grin>
Canamjay
50%
50%
Canamjay,
User Rank: Apprentice
11/10/2012 | 7:36:45 PM
re: Nate Silver's Big Data Lessons For The Enterprise
Great article and telling on so many levels. One of my great peeves is the totally predictable phrase always accompanying any public announcement of virtually any monitored stat today: 'surprising economists', 'confounding economic watch-dogs',
'much to the surprise of experts'... etcetera. These folks with Ph.D.'s , one would assume constantly monitored models, access to all the data and computer power they want/need, ALWAYS register 'surprise' when unemployment, GDP, CPI, various market performance indexes, don't match their predictions. This is always much more discouraging to me than the figures. Because, these are the 'experts' advising leadership on actions to effect results. As a society, I often feel we are vastly over-educated and woefully under skilled. Nate simply shows us: the tools are available, we just fail to use them correctly. What a revelation. Thanks for this.
Patrick Taylor
50%
50%
Patrick Taylor,
User Rank: Apprentice
11/10/2012 | 6:23:12 PM
re: Nate Silver's Big Data Lessons For The Enterprise
It's all about practically using analysis to all of the detailed information to make smarter decisions - whether calling a presidential election or winning baseball games. As pointed out in he article the key is "develop the best decision at the right time and deliver the information to the people that most need the information."

We've found that an important dimension for leveraging analytics is putting them to work at the frontline of business. In the Moneyball story Oakland goes on the winning streak when the players use the statistical insights to play the game smarter - you should take the walk in the 8th inning.

The thing we have to recognize in business is that it's more than executives making smarter decisions, it's everyone making smarter decisions by analyzing all the information we have available.
amywohl
50%
50%
amywohl,
User Rank: Apprentice
11/9/2012 | 6:38:34 PM
re: Nate Silver's Big Data Lessons For The Enterprise
Excellent article. I agree that knowing your business and taking a customer point of view is much more important than exactly following Nate Silver's process.
jries921
50%
50%
jries921,
User Rank: Ninja
11/9/2012 | 5:32:14 PM
re: Nate Silver's Big Data Lessons For The Enterprise
I'm starting to gain great appreciation for the policy adopted by Hari Seldon and his followers in Isaac Azimov's "Foundation" series of being very careful about what predictions they made when to keep people from reacting to them. It appears that Nate Silver did some great work, but I wonder if it might be so good that if it gets too much publicity it might affect future election results.
Magzilla
50%
50%
Magzilla,
User Rank: Apprentice
11/9/2012 | 2:11:29 PM
re: Nate Silver's Big Data Lessons For The Enterprise
Great article! We see creativity entering the picture of wrangling big data early on as a way to form a hypothesis, like most science experiments. And like all science experiments, data analytics are seeking out facts. Businesses ignoring this intelligence and basing decisions off of "gut" reactions will soon fall behind. Here is another article discussing this: < a href= " http://themoreyouknowbandb.wor..." _blank " > The More You Know < /a >
cbabcock
50%
50%
cbabcock,
User Rank: Strategist
11/8/2012 | 9:30:21 PM
re: Nate Silver's Big Data Lessons For The Enterprise
Silver's willingness to examine polling data in individual states and compare it to national polls gave him great insight into how the swing states might go. Outcomes in Iowa, Fla., Ohio, Virginia are mercurial, sometimes red, sometimes blue, but can be predicted for a given election if you've established a past relationship between those two sets of data. I think he knows how to spot trends in the national data, then winnows it among eight or nine swing states--how much will the trend apply in this setting? Hard to believe but Silver was the topic of discussion following our AARP-league basketball game last night. Charlie Babcock
EricLundquist
50%
50%
EricLundquist,
User Rank: Apprentice
11/8/2012 | 8:08:10 PM
re: Nate Silver's Big Data Lessons For The Enterprise
Thanks for the comment. I don't think this sounds ike an ad for Big Data, but you are entitled to your opinion. I think it is clear that I said Silver's success is not about big data but about an innovative use of data big and small.
Ellis Booker
50%
50%
Ellis Booker,
User Rank: Moderator
11/8/2012 | 5:49:09 PM
re: Nate Silver's Big Data Lessons For The Enterprise
This bit right here should be printed out and hung up at every water cooler and coffee machine in corporate America: "There is no magic to Silver's methods. There is hard work, a willingness to make mistakes and adjust, and a realization that the common wisdom is sometimes not wisdom at all." --Ellis Booker, InformationWeek Community Editor.
Page 1 / 2   >   >>
6 Tools to Protect Big Data
6 Tools to Protect Big Data
Most IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest, Dec. 9, 2014
Apps will make or break the tablet as a work device, but don't shortchange critical factors related to hardware, security, peripherals, and integration.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of December 7, 2014. Be here for the show and for the incredible Friday Afternoon Conversation that runs beside the program!
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.