Beyond Bayes: on the need for a unified and Jaynesian definition of probability and information within neuroscience | by Fiorillo, Information 3, 175-203.
@entropy @MED
These notes are specifically from chapter 3: Information and probability (179-188)
Also see freq. vs. Bayesian statistics.
This paper section mostly addresses the question: when we calculate with probabilities, where do these numbers come from?
Logic? Common sense?
A main point is that in order to correctly use probability theory and also Bayes' Theorem etc., that the method of Jaynes insists "on a very careful specification of the information upon which probabilities are conditional" (188)
Using Jaynes method of maximizing entropy, we can derive probability distributions from the available information (whereas BT only derives probabilities from other probabilities).
For a meaningful comparison, researchers should assume the same background knowledge (again, using maximum entropy). We can only meaningfully specify information (gain) against the backdrop of shared information. This is typical for what is usually known objective Bayesianism (but the term is not used in the paper, I think).
Historical notes
@Laplace 'referred to probability theory as "nothing but common sense reduced to calculation"' (179).
Laplace suggested the principle of indifference for the derivation of probabilities from knowledge.
Frequentist statistics (init: Ronald Fisher) departs from the attempt to give probabilities a "objective" status: we can meaningfully speak of probabilities when observing repeated events and measuring their frequency.
Still, you need to have rules for transforming frequencies into probabilities (generalized to general population etc.). Fiorillo calls these rules ad-hoc, i.e. frequentist statistics has no general unified principle to guide this transformation.
He argues that Bayesian or, he says, Jaynesian statistics does have such a principle, namely the Maximum Entropy Principle.
History:
- Laplace, indifference principle (TODO review)
- @Shannon's entropy
Shannon's entropy is a more general expression of Laplace's "indifference," and common sense tells us that in making inferences, we should not assume any information that we do not actually have.
- @Jaynes 1957: "Jaynes demonstrated that the maximization of entropy provided an objective basis for deriving probabilities from information" (180)
Probability and Maximum Entropy Principle
"According to Jaynes, probability theory is a natural extension of earlier work on logic." (180)
More specifically: "Jaynesian probability theory is a formal expression of logic (or reason)" (180). Why?
Logic is a restricted case where knowledge is certain so propositions can be said to be true or false. Probability is the same but with the introduction of certainty; introduces degree of belief or strength of belief for given proposition in [0-1]
Logic vs information
Logic is universal and indisputable, whereas information is localized in space and time to an observer, and it typically differs over time between observers. (180)
The relation between knowledge and probabilities then is, first of all, depend on available information to some person:
The probability of a particular proposition (or state) is always entirely conditional on a particular set of knowledge or information.
And this relation between probability and information (i.e. how do we go from information to probability?) is guided by the Maximum Entropy Principle:
The maximum entropy principle requires that we fully acknowledge our ignorance by considering all possibilities equally probable unless we have evidence to the contrary.
You have the following information about some problem, what can we then say about probability? Examples:
- there are 4 mutually exclusive outcomes
- uniform distribution has max entropy, assigns equal probability to all outcomes
- has to be 0.25 simply because the sum must be 1.
- assumes no further information
- We do not known the number of possible states, but we do know location and scale, e.g. mean and variance
- Gaussian probability distribution has max entropy!
- We may have observed high frequency in the past
- Probability distribution resembles the observed frequency distribution
- N.B. so frequency is special case of Jaynesian logic
In short:
The bridge between information and a probability distribution is the principle of maximum entropy (181)
Quantifying the amount of information (and information gain)
The amount of entropy H is, in this view, dependent on available information XY: H(x|XY) , where X is background or prior information of the observer and Y is the information we wish to quantify.
So information determines both probabilities and entropy.
Amount of information gain can be expressed as the difference between entropy in the old situation and entropy in the new situation after learning Y, i.e.:
I(Y|X)=H(x|X)-H(x|XY)
Underlying idea of Shannon's notion of information gain.
Main observation here: Shannon's notion of information quantity is relativistic! (even though he does not mention this himself).
I.e. the information gain is conditioned upon background information X.
People may find it confusing that 1) information quantity is relative to the observer's background knowledge and 2) that information gain, in Shannon's definition, may be negative.
Can we quantify information more "absolutely" in a meaningful way?
If two observers want to speak of the same information quantity, they must share the background X. One suggestion is thus to define a shared background between them; and in the general case, between all possible observers. This would be a universal prior!
The next best thing would be to identify a primitive state of minimal information that is universal (shared by all observers). If all observers quantify the amount of information (in states A, B, C, etc.) relative to the same "universal prior" information X, then they will all measure information on a common scale and they will agree on the amount of information. (182)
Also solves the negative information quantity:
Furthermore, if X corresponds to information that is minimal (very high entropy), then any additional information (XA, XB, etc.) will reduce entropy and thus have a positive value. (182)
Cf. in Shannon's entropy learning something new might increase entropy.
In other words, if the prior represents the largest possible ignorance (which is the point: do not assume information you do not have), then the Maximum Entropy Principle can be rephrased as the "minimum information principle". Minimizing I(Y|X) implies maximizing H(x|XY).
Repetition of "the point":
any distribution that has less than the maximum entropy assumes information that is not actually available (and any distribution that has higher entropy ignores the available information). (182)
Bayes' theorem
All probability theory can be seen in terms of MEP, but in practice this is often hard to compute. BT allows manipulating of probabilities without having to directly maximize entropy. The idea is rather, I think, that if your prior is appropriate, and your updating rule as well because it only takes into account the newly observed information (BT), the final result still follows MEP.
Nice quote:
Like all probability theory, BT is simply an expression of logic action on information.
This does mean however that Bayes' theorem is not strictly essential (185); the main principle is entropy maximization.
General method of Jaynes
- State precisely and explicitly on what information the probabilities are conditional
- Maximize entropy
- Use MED and Bayes' Theorem to derive new MEDs.
- Quantify the amount of information
"Agreement and understanding requires that all observers are utilizing the same information (a fact which is universally apparent to anyone above a certain mental age, and which is not at all unique to probability theory)." (183-184)
Just a thought: Jaynesianism describes how to update degrees of beliefs in terms of probabilities after an observation. To meaningfully talk about this process and the related information gain (XY vs X), you need to make sure you have the same background information X. But how do you make sure you have the same background information?
The solution of above: assume as little information as absolutely possible, given some scenario. This could very well describe how a good teacher leaves home all background knowledge, to match the mindset of the student. This allows the teacher to more appropriately conceptualize the information need and gain of the student. But this is a quite limited way of describing the process of coming to agreement after having and initial disagreement or lack of understanding due to different backgrounds. Gadamer's hermeneutics instead proposed that different backgrounds ("horizons") can merge in light of the "Sache selbst", i.e. that what is at stake in some dialogue. "Die Sache Selbst" thus seem to play some sort of analogous role to evidence, but reverse the logic. It is not the case that we need to come down to the same background with minimum assumptions and only the absolute minimum information that is given and shared between us, in order to express the information gain in light of new evidence. Conversely, in order to address the "evidence", what is in front of us as the matter to be discussed, "Die Sache Selbst", there is the need to merge our backgrounds into a new shared background. In the one case, the background information becomes as poor as possible, in the latter rich. The former describes rational belief updating; the latter describes knowledge in light of a tension between different traditions, cultures, and prejudices.
TODO defense of Jaynesianism and attack on frequentist statistics
These are great, but I don't feel like writing them out right now.
Only quickly, some great burns:
The most general point to be made here is that those maintaining a frequentist view are in reality basing their probabilities on their own information. Frequentist probabilities are not actually properties of an observed system, as frequentists claim. (186)
W.r.t. neuroscience:
A more glaring illustration of this fault, with respect to neuroscience, is that a frequentist view has no means to address the plain fact that different brains have different information and therefore place different probabilities on the same event. Thus, frequentist methods are frequently inapplicable.
Ha!
Bayesian Brain
Fiorillo is a neuroscientists and especially his third response is interesting in this regard.
It concerns the apparent contradiction between our brain being probabilistic, and humans being unable to accurately express probabilities in terms of a number when asked about it.
A third criticism stems from confusion of the subjective aspect of Jaynesian probabilities with our conscious attempts at quantifying the strength of our beliefs. Jaynesian probabilities are not simply a number that a person verbally reports based upon introspection. (185)
To solve any particular problem, the brain must have information about some relevant aspect of the world (with a corresponding state space). If the brain is then asked to verbally specify a probability, the relevant aspect of the world (and its state space) now corresponds to the abstract notion of "probabilities" themselves, and the problem facing the brain has radically changed. (185)
In other words, what Fiorilli is saying is that the "brain may have substantial information about some aspect of the world, but may have little information about the probabilities. In other words, *information about 'X' is not the same as information about the probability of 'X'. The former concerns the probability of 'X', whereas the latter concerns the probability of a probability of 'X'". (186)