Why big data might be more about automation than insights

Despite all the talk about companies using big data to uncover insights, maybe automation is the real reason the world is so excited about big data. What makes the big data era so significant isn’t that people are using data to inform their decisions, but that there’s just too much data of too many different types. In many cases, keeping up isn’t so much a matter of changing mindsets as it is about getting better tools.

Last week, New York Times reporter Steve Lohr wrote about the possibility of a big data bubble forming because people rely too much on data at the expense of experience and intuition. It got me thinking about all the technologies and algorithms I’ve covered, about all the discussions I’ve had about why a data scientist is more than just a statistician who can write MapReduce jobs. Nearly everywhere, it seems to me (save for, as Lohr cites, unique uses such as algorithmic trading), big data really is less about replacing human intuition than it is about augmenting the human experience by making it easier, faster and more efficient.

Like the purpose-built robots that have revolutionized manufacturing, today’s methods for processing and analyzing data are fast, scalable and precise, but they don’t yet (in most cases) make our decisions. Big data can make life and business a lot more efficient, but for the time being, human judgment and willpower are still very much in control.

Read Full Story