The following is a slightly edited cross-posting from anthrodesign.
As I read this thread, I find myself wondering if some of the critical remarks about big data aren't a bit behind the curve. I am thinking in particular of Sam's remark that,
Without some working theory of what social behaviour is, we have nothing but reams of meaningless data.
There is, of course, a sense in which this statement is obviously true. I suspect, however, that those who read it will imagine some degree of theoretical sophistication that only an anthropologist could provide. The fact of the matter is that relatively commonplace rules of thumb may result in delivering the value that businesses are looking for. One example is the proposition that someone who has already purchased a product is more likely to purchase one similar to it than a randomly selected prospect. This is the basic rule behind Amazon's you-might-also-be-interested-in suggestions. It is also the basic rule of thumb for political and charitable fundraisers, the reason why one donation leads to solicitations for repeated or similar donations.
I recall a bit of history. Back in the early 1990s, I was working on the Coca-Cola account for a Japanese advertising agency. I recall talking to someone about how the business was changing. Then the issue was data from point-of-sale (POS) systems. In "the good old days," the agency would pitch the client using research that was often conducted months, even years, before Coca-Cola made marketing decisions for the coming fiscal year. Then, when Coca-Cola had made its decisions, a big annual meeting was held at which Coca-Cola revealed its marketing plans to the bottlers. The plans would be executed starting a few weeks later. POS data created a totally different world. Convenience store (CVS) chains, in particular, were now able to track product sales day-by-day, and data collected months or years previously was hopelessly out of date. Suddenly the world in which researchers developed complex theories, basically good stories, to sell annual marketing strategies was a thing of the past. Now everybody, at the bottlers, at Coca-Cola, at the agency, was scrambling to keep up with the latest POS data and agile response became more important than carefully constructed theories.
Now the Internet has shifted data collection from day-by-day through POS systems to second-by-second data collected on every online transaction. And that isn't all that has changed. In Bursts: The Hidden Pattern Behind Everything We Do, Albert-Laszlo Barabasi suggests that the proposition that people are predictable en masse but not as individuals is, in fact, nearly totally backwards. As individuals we are mostly creatures of habit, and now that data on our habits can be collected on an individual level, we are very highly predictable in what we do. Stories that explain the means and medians revealed by normal curves are pointless in a world where the behavior or people on the tails of the curve can be predicted as easily and precisely as that of the "average" person.
Does this mean that theory is dead? Perhaps not. But theories developed for the world in which the aim was to explain the norm, and the norm was the average of some population? They are increasingly dead in the water. One suspects that Tom Kelley has it right in The Ten Faces of Innovation when he describes the anthropologist as a person of infinite curiosity and meticulous note taking who notices things that other people miss and says nothing at all about theory. The question is whether the anthropologist will be equipped to notice things that other people miss IN big data or will be left standing on the sidelines moaning about the fact that what we used to think was valuable isn't seen that way any more.
Might be worth thinking about.