"It's tough to make predictions, especially about the future." So said Yogi Berra, baseball great and amateur philosopher.
Sensible (and amusing) as it sounds, his dictum no longer rings true. The Age of Big Data has arrived — and, with it, the ability to predict the future is increasingly a part of a new business reality. Whatever your discipline, doing business today means immersing yourself, and your organization, in a wealth of messy, unstructured, real time data from customers, competitors, and markets — and finding ways to use such data visibility to see what's coming.
Advantage lies in a capacity to predict the future before your rivals can — whether they're companies or criminals. Consider how the New York Police Department is using Big Data to fight crime in Manhattan. According to a series on Big Data in The New York Times, the NYPD and other big city police departments are using data-crunching technology to geo-locate and analyze "historical arrest patterns," while cross-tabbing them with sporting events, paydays, rainfall, traffic flows, and Federal holidays to identify what NYPD calls likely crime "hot spots." As immortalized in a "Smarter Planet" commercial from IBM, such insight can help deploy officers to locations where crimes are likely to occur before they are actually committed.
The beauty of such Big Data applications is that they can process Web-based text, digital images, and online video. They can also glean intelligence from the exploding social media sphere, whether it consists of blogs, chat forums, Twitter trends, or Facebook commentary. Traditional market research generally involves unnatural acts, such as surveys, mall-intercept interviews, and focus groups. Big Data examines what people say about what they have done or will do. That's in addition to tracking what people are actually doing about everything from crime to weather to shopping to brands. It is only Big Data's capacity for dealing with vast quantities of real-time unstructured data that makes this possible.
For example, retailers like Wal-Mart and Kohl's are making use of sales, pricing, and economic data, combined with demographic and weather data, to fine-tune merchandising store by store and anticipate appropriate timing of store sales. Similarly, online data services like eHarmony and Match.com are constantly observing activity on their sites to optimize their matching algorithms to predict who will hit it off with whom. The same logic is being applied to economic forecasting. For example, the number of Google queries about housing and real estate from one quarter to the next turns out to predict more accurately what's going to happen in the housing market than any team of expert real estate forecasters. Similarly, Google search queries on flu symptoms and treatments reveal weeks in advance what flu-related volumes hospital emergency departments can expect.
Much of the data organizations are crunching is human-generated. But machine sensors — what GE people like CMO Beth Comstock called "machine whispering" when I talked with her this past summer — are creating a second tsunami of data. Digital sensors on industrial hardware like aircraft engines, electric turbines, automobiles, consumer packaged goods, and shipping crates can communicate "location, movement, vibration, temperature, humidity, and even chemical changes in the air." As the volume of both human and machine data grows exponentially, so too will organizations' ability to see the future.
The net of all this is hardly a cold quantitative world. Rather, as marketers and machine systems learn more about our attitudes and behaviors, they're likely to achieve greater intimacy with consumers and customers than ever before. Yes, there is the risk of an Orwellian nightmare, if the inferences from Big Data become too intimate and too intrusive — and end up in the wrong hands. But there is also the opportunity to deliver services and marketing with unprecedented precision and accuracy, meeting and exceeding customer expectations in preternatural ways at every turn. Knowing the right time to deliver the right message (or action) in the right place before the time has come will bestow extraordinary power to those who wield such intelligence with intelligence. Use prediction wisely, and Big Data has the potential to make the world small again. That is every marketer's dream: getting closer to customers.
Using Market Research Just for Marketing Is a Missed Opportunity Can Big Data Smoke Out the Silent Majority? Retailers Turn to "Soft Surveillance" to Fight Customer Anonymity What Data Can't Tell You About Customers
It's coming, faster than you can crunch.