Lecture BN 7: Conditional Independence Testing III
Conditional independence is also useful when the model is finished.
- Quantify the strength of each edge
- Use to prune down too complicated models
- NEW: conditional independence testing for discrete data without stratification (other than chi-squared)
Testing with continuous data
What we did is look at regression models. But this means our tests now use standards assumptions of linear regression. This assumption is that the residuals of the data are normally distributed, i.e. linear line with Gaussian noise.
Slide 8-5: linear, monotone, non-monotone.
For monotone increasing function, we see coefficient decreased with factor 2, but this does not reflect that there is in fact a clear relationship in the data. This is even more extreme for non-monotone functions; linear regression coefficient does not capture that there is in fact a strong (non-monotonic) correlation between the data.
So you may miss essential (non-linear) elements of the data.
The chi-squared test does not make this assumption. This is why it is a non-parametric test. But as a downside, it has statistical instability, esp. with higher dimensionality.
5-min exercise
See slide 8-6.
- YES
- NO
- YES
- NO
Modelling non-linear relations
We can also try free-form regression instead of linear regression.
- free-form regression is local polynomial regression (LOWESS/LOESS)
- or "locally weighted scatterplot smoothing"
- Does regression on sliding windows
- Window width can be estimated automatically
- Random forest
Testing independence for linear and non-linear regression
Have regression predictions E[Y|X] and correlate the prediction to the actual data.
If the correlation is high, then the regression is suitable for the data. We can assess the statistical significance of these correlations using a permutation test.
Permutation test
- Make a dataset where the null hypothesis is true
- So when we test for independence we make a uncorrelated dataset by randomly shuffling the data -Test correlation several times between prediction on different shuffles/permutations and data
- We then have a pretty good picture of what correlation to expect if the null hypothesis were true
- We can reject the null hypothesis if the correlation of the prediction on X with X is really different from the distribution found with the random permutations.
Modern tests like t-tests can be seen as approximating permutation tests. But in many cases we mathematically/analytically know what the null-hypothesis distribution should look like, so we don't need to "simulate" that distribution like using this permutation approach.
Permutation testing always works, no matter how complicated your data is. In well-defined cases it corresponds to the results achieved from a specific test, that is only applicable under certain circumstances.
To estimate a p-value, we can count the number of times your correlation was larger than what is expected under the null (see 8-9).
Residiualization approach for conditional independence
We can also approach testing conditional independence using residuals. This is useful when the assumptions for linear regression do not hold. I.e. it is more generalizable.
- Regress X on Z and determine residuals
- Regress Y on Z and determine residuals
- rx and ry correlate, then X ind. Y given Z does not hold.
rx and ry express "unique" information that's already captured by Z.
But residual correlatedness can also be misleading
Linear regressions can lead to false positives of independence, but not to false positives.
But using residuals we can also have false positives!
8-12; this is also an example of a scenario where we should not just add an edge between X and Z.
The problem is not necessaarily the missing edge, but may just be that we are using linear regression on data that is simply not linear. Think about the data! (e.g. scatterplot of two involved variables)
We can also just use free-form regression (e.g. LOESS) and see if that changes the relation.
localTests( data=df, type=cis.loess, reps)
Determining edge coefficients
Any test of conditional independence is not only useful for testing network structure, but also for quantifying edge strengths.
This is because of the Frisch-Waugh-Lovell theorem: we can also do two separate regressions and then correlate the residuals. These three regressions can be used to determine the coefficient of the single residual.
REVIEW 8-15.
Ordinal + discrete + continuous data
I paid attention and didn't make many notes. Very useful half.
Hotelling's t-square.
When using dummy coding, we have negative correlations between the dummy variables. So if we correlate some other variable with these dummy encoded columns, all these correlations will themselves be correlated.
How do we then combine these correlations into a correlation estimate for the whole variable, since the assumption of independence does not hold?
- Transform data in new space where vector x is not auto-correlated
- Inverse of correlation matrix can be used for de-correlation
- "Inertia matrix method" in other contexts
See slide 8-20: you can see correlation as a mean. And Hotelling's test is about checking whether a mean is 0. SMART MOTHERFUCKER.
8-27: can also be used for pruning edges where p is large.
Some weaker correlations survive the pruning in slide 8-27 anyways; this is because p-values are a monotone function of ...