Bayesian Networks Lecture 7: Structural Equation Models
Topic of today: dealing with continuous data.
- Review Gaussian Distributions
- Structure Equation Models (SEMs)
- Testing SEMs
Continuous Probability Densities
Cf. 202006041217-continuous_probability_distribution.md
The probability that a variable has an exact value is 0, because on a continuous scale we have infinity many possible values Probability is only defined over an interval i.e. a "bin"; the probability that the value falls in this interval.
If we replace summing with integration, all equations we are used to are equivalent, including the conditions for (conditional) independence.
Uniform Density
Rarely occurs in nature.
For a <= X <= b, P(x)=1/(b-a).
Variance and covariance
Var(X)=E[(X-E[X])^2] = Cov(X,X)
Cov(X,Y) = E[ (X - E[X])(Y - E[Y])]
Covariance is not the same as correlation! Correlation is basically scaled/normalized covariance.
Variables that are not predicting each other well may still have a very high covariance. Uncorrelated variables may still have high covariance.
Covariance can be seen as a measure of shared information.
Correlation is more easy to see in a plot. In our example, C had higher covariance mainly because of higher variance on the Y-axis (this matters because again, it's not normalized like correlation).
Univariate Gaussian
Gaussian is not necessarily the most common distribution, but very important nevertheless.
Parametric distribution with mean and variance parameters.
Multivariate
The multivariate Gaussian contains the covariance matrix as a parameter.
Gaussian properties
REVIEW slide 6-11; use these properties.
The sum of Gaussians is another Gaussian.
Exercise: let X ~ N(0,1) and Y ~ N(0,1) where Cov(X, Y)=0
Var(2X)=4. (one sample multiplied by two)
Var(X+X)=2 (two independent samples from the same variable; so variance stays the same)
Var(X+Y)=Var(X+X)
Var(X+2Y)=Var(2Y)= 5
Conditional Covariance
Variance of variables in the conditional distribution. For Gaussian this is a single number for the whole distribution; independent of the specific value of Z. This is a very nice property of Gaussians.
This is very different for discrete distributions: when you condition upon a value there, the distribution may really change.
Can be computed using a recursive formula:
$Cov( A, B | X ) = Cov( A, B ) - Cov( A, X ) * Cov( B, X ) / Var( X )$
See 6-13.
You subtract the shared covariance with a third variable.
Exercise 6-14, Cov(A,B|X): by taking into account X, we take away all shared information between A and B. We see that X is strongly correlated both with A and B.
Cov(A,B|X)= (0.4-0.5*0.8)/1 = 0
This implies that A and B are conditionally independent given X!
So for continuous values, we can use covariance as a test of conditional independence!
Structural Equation Models
We have been talking about BNs in terms of models, using graphical structures. Each node has a conditional probability table conditioned on its parents. But this only works for discrete data. We can't do probability tables for continuous data; here we need a parametric distribution.
SEMs come from psychometrics, but the concepts you use are very similar to BNs. But if you study SEMs the world "Bayesian Network" will probably not be mentioned at all.
A SEM is a Bayesian network where each variable X is a linear function of its parents and a Gaussian residual variable $\eta_x$.
We usually display SEMs as a directed path diagram labelled with the linear coefficients (beta's) and residual variances.
REVIEW 6-15; didnt pay attention. Remarks about exponential complexity vs linear equation representation.
We also in principle have residual means, but we assume here they are zero and that we have preprocessed our data to be centered around mean zero.
6-16; we are not simply doing prediction. Is this an example of a SEM? Not sure. We can fit parameters to data, and then try to extract information from the parameters itself. They are meaningful; in this graph they indicate strength between nodes.
SEM is a generative model.
But what data are we generated with SEM models? The generated data is itself a multivariate normal, because each datum is a linear combination of the residuals, ultimately.
- Each linear combination of variables is therefore also a linear combination of residuals
- Residuals are normally distributed.
When we have a generative model, we can produce data and compare it to our original data set. This can be a check for the correctness of your parameters.
If you have a wrong model, you'll never be able to generate the data that looks like the training data.
Fitting SEMs locally
Usually we draw the structures, and then try to estimate the parameters such that this fits the data.
Again, we do not just want to predict your skill, but we care about the whole network. We want the whole picture and have an idea of the relative strengths between variables.
Normalizing the data is important, because then we cannot compare the covariances at the edges directly; when scaling covariance is going to be the same thing as correlation and then we can properly compare the strength of connections.
Packages that include SEMs:
- lavaan (latent variable analysis)
In Lavaan syntax:
sem( "science ~ read+write+math + math ~ read + write", ... ) (didn't write out the whole command)
Includes standardization.
Applying d-separation to SEMs
First we want to test our structure, again using conditional independence relationships.
We saw that in multivariate Gaussians, conditional independence entails vanishing conditional covariance (cf. example with conditional covariance going to 0).
We can in general test this using linear regression as well.
There is a deep relation between linear regression and conditional covariance!
The regression coefficient is conditional covariance divided by ...
This means that we can use standard regression packages to test for conditional independence! By inspecting the weights. This is handy because these packages immediately provide confidence intervals and/or different tests on regression weights.
See 6-22 for example.
- Bin your continuous data
- Treat your categorical data as binary and treat 1 and 0 just as numbers.
Preview
Non-ordinal categorical variables remain problematic! Only work well if they don't have parents. (Review 7-30)
Check lavaan's lavCor.