Clayton Christensen, author of the only business book Steve Jobs ever read, The Innovators Dilemma, had some pretty interesting comments on big data and how ideas emerge in an interview with the economist.
In response to a Christensen suggestion that firms are too beholden to data, the economist asked him how that fits with the “age of big data?”
It is truly scary to me. By definition, big data cannot yield complicated descriptions of causality. Especially in healthcare. Almost all of our diseases occur in the intersections of systems in the body. For example, there is a drug that is marketed by Elan BioNeurology called TYSABRI. It was developed for MS [multiple sclerosis]. It turns out that of the people who have MS a proportion respond magnificently to TYSABRI. And others don't. So what do you conclude from this? Is it just a mediocre drug? No. It is that there is one disease but it manifests itself in different ways. How does big data figure out what is the core of what is going on?
Christensen also comments on how ideas emerge.
Almost always great new ideas don't emerge from within a single person or function, but at the intersection of functions or people that have never met before.
Most companies are organized so you don't have those intersections because they are too siloed.
And most of our education is organized the same way – silos. One way for individuals to improve decisions and ideas is to intersect the main ideas from the big disciplines.