PDL Discussion of assignment 4

  • We observe that the error calibration of the variational model is actually way worse than that of the regular ML model. How is that possible?
    • A: Well, looks like the prior is not well-calibrated!
    • You can try cross-validation your prior choices for various choices of parameters. Very expensive!
    • What are better choices for priors then? We only used independent Gaussians now (i.e. mean field approximation)
    • But how would we e.g. find a good correlation for priors? Where do we get this info from?
    • Transfer learning / meta learning approach
  • It is also still possible that the variational model becomes overconfident
  • The measure of error calibration is also quite sensitive to outliers
    • REVIEW In the case of logistic regressions we can sometimes get a log likelihood of -Inf (when data is linearly separable, I think?)
    • This problem should be remedied when you have more data, as this is then less likely to be linearly separable.