Frequentist versus subjective interpretation of probability
@Bayes @probability @frequentist
Based on TFCA lecture notes, appendix B. Also see other TFCA notes.
Statement: P(A)=0.4
Frequentist interpretation:
- There is some underlying process generating A
- In approximately 4 out of 10 cases this process will generate A
- Alternative expression: the relative frequency of A = true is 0.4
- "Note that this is a constrained view on probabilities: it is limited to processes that can be repeated in principle infinitely often." (74)
N.B. relative frequency (i.e. objective chance) is relative to the set of all outcomes.
Bayesian or subjective view:
- includes the frequentist view and is itself broader
- "The subjective degree of belief, based on all the available information (including, but not limited to knowledge about relative frequencies), that one has in the event that A is true, is 0.4" (74)
- a "within subject" view on probability, rather than related to objective frequencies in the world
- the assignment of objective frequencies in the world in turn depends (e.g. for a die) depends on subjective information available; perhaps I see the die from a angle that obscures some outcomes, leading to different subjective probabilities for the same objective outcomes.
How is Bayesianism broader?
The subjective interpretation is said to include the frequentist interpretation. This implies that the Bayesian interpretation can attribute meaning to remarks that are not expressible in a strictly frequentist view.
Examples from page 74:
- Throwing heads with a fair coin
- Throwing heads with a coin which is not known to be fair
- Throwing heads with a coin which is known not to be fair
- Reverend Green killed Dr. Black with the dagger
- The sun will rise tomorrow
- It will rain next weekend
All these statements are meaningful in a Bayesian interpretation of probability, but only the first one is meaningful in a frequentist interpretation. Why?
We only have enough experience with throwing fair coins to know their relative frequency.
In case 2 and 3, we cannot assign a frequentist probability due to ignorance. In case of 2, we simply do not know anything; in case 3, we know the coin is not fair, but we have no experience with the specific unfairness of this coin. But this problem does not arise with subjective probabilities: p(A)=0.5 expresses this subjective ignorance.
If you do not already have plenty of experience, you would need to set up an experiment to sample from the process generating the observation of interest.
But 4,5,6 are all unique one time events, so we cannot assign a (frequentist) probability to them. That requires, by definition, a repeatable event.
(The syllabus does not mention this, but for 4,5,6 we can just use prior beliefs for expressing subjective probability).
Another example from Information 2012, 3, p. 179:
You tell someone there are four possible outcomes A, B,C, D that are all mutually exclusive. What is the probability of each outcome?
The Jaynesian answer is that, in the absence of any additional information, logic requires that all outcomes be considered equally likely, and thus the probability of each is 1/4 or 0.25 (179)
However, from a strictly frequentist perspective the question cannot be answered. You'd first need to see many repeated observations, i.e. samples from the underlying generative event, to be able to make a claim about the objective / relative frequencies of this event (and to be certain, infinite observations).
Frequentist:
- Probabilities are unknown and undefined in this case
- Because no frequency has been measured
- There is no notion of a prior, so this is only dealt with ad-hoc.
- Consequently, we cannot estimate the true probability ...
- ... of the underlying physical system/process.
Jaynesian:
- In this case, the prior belief of 0.25
- this is logically justified (or: common sense); these probabilities precisely express a state of ignorance (max entropy).
- If you say something else you assume information that is not there.
- 0.25 still expresses information, namely that we know the events are mutually exclusive, that there are four outcomes etc.
- All of this is completely disregarded in the frequentist approach
- The probabilities are thus a property of the observer of this system.
Note however that within the Bayesian framework, there is a further distinction between subjective and objective Bayesians.
Objective Bayesians impose additional constraints on the probabilities used in Bayesian conditionalization, such as Jaynes' max entropy principle mentioned above. Knowledge about frequentist probabilities (or the absence thereof, in the above example) can thus also be included. This only further supports the general point that frequentist probability is included in the more general Bayesian understanding of probability as degrees of belief.