How Scientific Advancement Happens

The progress of science is commonly perceived of as a continuous, incremental advance, with new discoveries added to the existing body of scientific knowledge. However, Thomas Kuhn argues that the history of science tells a different story, in which discontinuities are crucial. He argues that science proceeds with a serious of revolutions. “A prevailing theory or paradigm is not overthrown by the accumulation of contrary evidence,” Richard Zeckhauser wrote, “but rather by a new paradigm that, for whatever reasons, begins to be accepted by scientists.” Between revolutions old ideas and beliefs persist and form barriers of resistance to alternative explanations.

Zeckhauser continues “In this view, scientific scholars are subject to status quo persistence. Far from being objective decoders of the empirical evidence, scientists have decided preferences about the scientific beliefs they hold. From a psychological perspective, this preference for beliefs can be seen as a reaction to the tensions caused by cognitive dissonance. ”

* * *

Gary Taubes posted an excellent blog post discussing how paradigm shifts come about in science. But Taubes explanation wasn’t enough to satisfy my curiosity.

Taubes wrote:

…as Kuhn explained in The Structure of Scientific Revolutions, his seminal thesis on paradigm shifts, the people who invariably do manage to shift scientific paradigms are “either very young or very new to the field whose paradigm they change… for obviously these are the men [or women, of course] who, being little committed by prior practice to the traditional rules of normal science, are particularly likely to see that those rules no longer define a playable game and to conceive another set that can replace them.”

So when a shift does happen, it’s almost invariably the case that an outsider or a newcomer, at least, is going to be the one who pulls it off. This is one thing that makes this endeavor of figuring out who’s right or what’s right such a tricky one. Insiders are highly unlikely to shift a paradigm and history tells us they won’t do it. And if outsiders or newcomers take on the task, they not only suffer from the charge that they lack credentials and so credibility, but their work de facto implies that they know something that the insiders don’t – hence, the idiocy implication.

…This leads to a second major problem with making these assessments – who’s right or what’s right. As Kuhn explained, shifting a paradigm includes not just providing a solution to the outstanding problems in the field, but a rethinking of the questions that are asked, the observations that are considered and how those observations are interpreted, and even the technologies that are used to answer the questions. In fact, often the problems that the new paradigm solves, the questions it answers, are not the problems and the questions that practitioners living in the old paradigm would have recognized as useful.

“Paradigms provide scientists not only with a map but also with some of the direction essential for map-making,” wrote Kuhn. “In learning a paradigm the scientist acquires theory, methods, and standards together, usually in an inextricable mixture. Therefore, when paradigms change, there are usually significant shifts in the criteria determining the legitimacy both of problems and of proposed solutions.”

As a result, Kuhn said, researchers on different sides of conflicting paradigms can barely discuss their differences in any meaningful way: “They will inevitably talk through each other when debating the relative merits of their respective paradigms. In the partially circular arguments that regularly result, each paradigm will be shown to satisfy more or less the criteria that it dictates for itself and to fall short of a few of those dictated by its opponent.”

To learn more on how paradigm shifts happen, I purchased Kuhn’s book, The Structure of Scientific Revolutions, and started to investigate.

Kuhn writes:

“The decision to reject one paradigm is always simultaneously the decisions to accept another, and the judgment leading to that decision involves the comparison of both paradigms with nature and with each other.”

Anomalies are not all bad. Any scientist who pauses to examine and refute every anomaly will seldom get any work done.

…during the sixty years after Newton’s original computation, the predicted motion of the moon’s perigee remained only half of that observed. As Europe’s best mathematical physicists continued to wrestle unsuccessfully with the well-known discrepancy, there were occasional proposals for a modification of Newton’s inverse square law. But no one took these proposals very seriously, and in practice this patience with a major anomaly proved justified. Clairaut in 1750 was able to show that only the mathematics of the application had been wrong and that Newtonian theory could stand as before. … persistent and recognized anomaly does not always induce crisis. … It follows that if an anomaly is to evoke crisis, it must usually be more than just an anomaly.

So what is it that makes an anomaly worth the effort of investigation? To that question Kuhn responds, “there is probably no fully general answer.”

When the anomaly comes to be recognized as more than another puzzle of science the transition has begun.

The anomaly itself now comes to be more generally recognized as such by the profession. More and more attention is devoted to it by more and more of the field’s most eminent men. If it still continues to resist, as it usually does not, many of them may come to view its resolution as the subject matter of their discipline. …

Early attacks on the anomaly will have followed the paradigm rules closely. As time passes and scrutiny increases, more of the attacks will start to diverge from the existing paradigm. It is “through this proliferation of divergent articulations,” Kuhn argues, “the rules of normal science become increasing blurred. Though there still is a paradigm, few practitioners prove to be entirely agreed about what it is. Even formally standard solutions of solved problems are called into question.”

Einstein explained this transition phase best: “It was as if the ground had been pulled out from under one, with no firm foundation to be seen anywhere, upon which one could have built.

All scientific crises begin with the blurring of a paradigm.

In this respect research during crisis very much resembles research during the per-paradigm period, except that in the former the locus of difference is both smaller and more clearly defined. And all crises close in one of three ways. Sometimes normal science ultimately proves able to handle the crisis—provoking problem despite the despair of those who have seen it as the end of an existing paradigm. On other occasions the problem resists even apparently radical new approaches. Then scientists may conclude that no solution will be forthcoming in the present state of their field. The problem is labelled and set aside for a future generation with more developed tools. Or, finally, the case that will most concern us here, a crisis may end up with the emergence of a new candidate for paradigm and with the ensuing battle over its acceptance.

But this isn’t easy.

The transition from a paradigm in crisis to a new one from which a new tradition of normal science can emerge is far from a cumulative process, one achieved by an articulation or extension of the old paradigm. Rather it is a reconstruction of the field from new fundamentals, a reconstruction that changes some of the field’s most elementary theoretical generalizations as well as many of its paradigm methods and applications.

Who solves these problems? Do the men and women who have invested a large portion of their lives in a field or theory suddenly confront evidence and change their mind? Sadly, no.

Almost always the men who achieve these fundamental inventions of a new paradigm have been either very young, or very new to the field whose paradigm they change. And perhaps that point need not have been made explicit, for obviously these are men who, being little committed by prior practice to the traditional rules of normal science, are particularly likely to see that those rules no longer define a playable game and to conceive another set that can replace them.

And

Therefore, when paradigms change, there are usually significant shifts in the criteria determining the legitimacy both of problems and of proposed solutions.

That observation returns us to the point from which this section began, for it provides our first explicit indication of why the choice between competing paradigms regularly raises questions that cannot be resolved by the criteria of normal science. To the extent, as significant as it is incomplete, that two scientific schools disagree about what is a problem and what is a solution, they will inevitably talk through each other when debating the relative merits of their respective paradigms. In the partially circular arguments that regularly result, each paradigm will be shown to satisfy more or less the criteria that it dictates for itself and to fall short of a few of those dictated by its opponent. There are other reasons, too, for the incompleteness of logical contact that consistently characterizes paradigm debates. For example, since no paradigm ever solves all the problems it defines and since no two paradigms leave all the same problems unsolved, paradigm debates always involve the question: Which problems is it more significant to have solved? Like the issue of competing standards, that questions of values can be answered only in terms of criteria that lie outside of normal science altogether.

Many years ago Max Planck offered this insight: “A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.”

If you’re interested in learning more about how paradigm shifts happen, read The Structure of Scientific Revolutions.

* * *

You can read the rest of Taubes post here.

Gary Taubes is the author of Good Calories, Bad Calories and most recently, Why We Get Fat And What To Do About It. See the FS notes on Why We Get Fat.