How to Measure What We Don’t Know
The search for mathematical precision in everything (physics envy) continues. There is no way for people to learn everything about a complex process and some information always remains hidden and difficult to extract.
In their recent work, the researchers adopt a thorough-going informational view: All of Nature is a communication channel that transmits the past to the future by storing information in the present. The information that the past and future share can be quantified using the “excess entropy” – the mutual information between the past and the future.
“The basic idea is that a process can appear to not transmit much information from its past to its future, but still require a large amount of hardware to keep the internal machine going,” Crutchfield said. “For example, imagine that you have two coins: Coin A is a fair coin and Coin B is slightly biased. Now the output of this process is a series of heads and tails. That’s all the observer gets to see. The observer doesn’t know when A is used or B is used. To an observer this process is very close to a fair coin – the heads and tails from B just don’t differ much in their statistics from the heads and tails from A. So, the observed process has little mutual information (the heads and tails are pretty much independent of the past). That is, the process has very low excess entropy. Nonetheless, there is one bit of internal stored information: Which coin, A or B, is flipped at each step? You can take this example to an extreme where you have hundreds of internal coins, all slightly biased, all slightly different in their bias, and therefore distinct coins. The large number of coins gives you an arbitrarily large statistical complexity. But the small biases mean the excess entropy is as close to zero as you like.”
These fundamental results should impact research across a wide range of disciplines, from statistical modeling to novel forms of computing. As the researchers explain, when a process contains hidden information, the process cannot be directly represented using only raw measurement data. Rather, a model must be build to account for the degree of hidden information that is encrypted within the process’s observed behavior. Otherwise, analyzing a process only in terms of observed information overlooks the process’s structure, making it appear more random than it actually is.
If you like this subscribe.