April 29, 2012 | WSJ | By BEN ROONEY

Big Data's Big Problem: Little Talent

"A significant constraint on realizing value from Big Data will be a shortage of talent, particularly of people with deep expertise in statistics and machine learning, and the managers and analysts who know how to operate companies by using insights from Big Data," the report said. "We project a need for 1.5 million additional managers and analysts in the United States who can ask the right questions and consume the results of the analysis of Big Data effectively." What the industry needs is a new type of person: the data scientist.

One of the earliest tests for biggish data was applying it to the battlefield. The Pentagon ran a number of field exercises of its Force XXI—a device that allows commanders to track forces on the battlefield—around the turn of the century. The hope was that giving generals "exquisite situational awareness" (i.e. knowing everything about everyone on the battlefield) would turn the art of warfare into a science. What they found was that just giving bad generals more information didn't make them good generals; they were still bad generals, just better informed.

Hilary Mason, chief scientist for the URL shortening service bit.ly, says a data scientist must have three key skills. "They can take a data set and model it mathematically and understand the math required to build those models; they can actually do that, which means they have the engineering skills…and finally they are someone who can find insights and tell stories from their data. That means asking the right questions, and that is usually the hardest piece."

"job of the data scientist isn't simply to uncover lost nuggets, but discover … & turn them into actions"

It seems that the markets are as much in love with "Big Data"—the ability to acquire, process and sort vast quantities of data in real time—as the technology industry.

Big Data's Big Problem: Little Talent http://t.co/7groLocm

HP's @autonomycorp says algorithms mitigate Big Data's talent shortage. If so, good for #ML cos like @skytreehq

OMG, Big data needs sociological theory to make sense of it. Who would have thunk it?

Big Data's Big Problem: Little Talent ()

Big Data's Big Problem: Little Talent

"A significant constraint on realizing value from Big Data will be a shortage of talent, particularly of people with deep expertise in statistics and machine learning, and the managers and analysts who know how to operate companies by using insights from Big Data," the report said. "We project a need for 1.5 million additional managers and analysts in the United States who can ask the right questions and consume the results of the analysis of Big Data effectively." What the industry needs is a new type of person: the data scientist.

It seems that the markets are as much in love with "Big Data"—the ability to acquire, process and sort vast quantities of data in real time—as the technology industry. Hilary Mason,…

Talent Shortage Looms Over Big Data - http://t.co/SLUmUKI0 http://t.co/Wzgciasm

Big Data's Big Problem: Little Talent

RT @myoung: WSJ article, starring @hmason, on the lack of data scientists: Big Data's Big Problem: Little Talent

WSJ article, starring @hmason, on the lack of data scientists: Big Data's Big Problem: Little Talent

WSJ article, starring @hmason, on the lack of data scientists: Big Data's Big Problem: Little Talent