Amortized inference

This is the last step we discuss before introducing variational autoencoders.

The original autoencoder model is attached as well. This is also (exam) material.

Amortized optimization

The loss function is parameterized with some extra parameter w. The idea is that we can use this single parameterized expression for a whole family of optimization problems.

Trade-off: return of assets and the volatility, and correlation between stocks. These can be parameters in the loss function. We strive for some optimal ratio, maximizing profit while reducing risk. Depending on these values, our optimization problem changes.

The idea of amortized inference is to do this optimization for all possible parameters of the parameterized loss function. So we do not train a separate model for each configuration, but a single overarching model.

To do this we define a new amortized loss function by intregrating over individual loss functions.

It can happen that the range is infinite, causing the integral to be infinite (and the loss becoming meaningless). To solve this issue, we multiply by some density function that integrates to 1 and captures the interesting regions from the loss. I.e. the interesting parts should be in the high density parts of your density function.

This basically means that you can express the amortized loss as an expectation over all parameterized loss functions.

We have global optimima over individual loss functions.

If $f$ is infinitely flexible, you expect that for each $w$ it will return the global optimum for that specific loss function.

This does assume that all density at values of w are non-zero. If it's zero, the contribution of the specific loss function to the amortized loss can be any arbitrary value; this is a degenerate case. So: the support of density needs to be equal to whole range of parameters.

In practice, we don't reach the global optimum. In this case, we do want to reach the global optimum in this case.

Why amortized inference?

  • Need to solve optimization problems very fast
  • Train and apply on the fly

Amortized variational inference

Amortized inference is important for how variational inference is done in practice.

The most common form of amortization in VI is data amortization.

In what we saw before, we need to redo the whole optimization by gradient descent each time we process a data point, in order to recompute/approximate the posterior.

So instead we parameterize the variational posterior (e.g. Gaussian mean field approximation) with functions of the data instead of trainable params like means and sigma's. I.e. in the case of Gaussians, mean and sigma's are now functions of dataset X.

If we use deep learning, we now have two outputs of an inference network:

  1. mean
  2. variance

In this case, the parameter space is the space of possible data sets. We usually do this empirically by sampling and averaging.

We need to apply reparameterization trick.

See eq. 8 for amortized loss expression. variational posterior q is dependent on x.

Normally you fix the data and infer the latent. So with new data, you need to retrain and find the new latents.

Joint-contrastive training

In ML it's e.g. common to have generative models.

Eq. 9; phi, you often don't care about uncertainty here and just do ML. But we have z, so we need ELBO anyways.

There is a nice method for jointly training the likelihood of (generative) model and variational posterior at the same time.

Two different representations of joint model.

  1. Likelihood and prior
  2. Dataset and variational distribution over latents

If we can move these closer, we can do multiple things at the same time (which two things?).

By reducing the distance between 1 and 2 we approximate the sampling distribution of the data with the marginal likelihood of the model.

This is going way to fast for me.

So as loss function we use the KL-divergence between these two.

Eq. 11; we had the problem that the KL-divergence before had a marginal likelihood term. In this case not!

But... we now have the new problem that we need to know the density of the data set. But because this is not parameterized, this will add a constant contribution. So we can actually ignore this term. By doing that, we retrieved the amortized ELBO (ref. eq 8).

So in eq. 11 $\rho(x)$ drops out.

Variational autoencoders

  • Take high-dimensional data (e.g. image)
  • Compress in latent features
    • These ideally are uncorrelated, because uncorrelated features are more informative
    • Cf. in image space information is highly redundant, because pixels are highly correlated
    • So by compression we ... entropy ...
  • Decompress and reconstruct from latents

Common model for variational autoencoders is a Gaussian for each of the pixels. This is essentially a simple noise model. Mean is given by some (complex) linear function of latents. Conditional independence assumption: Given latent, each pixel is assumed to be statistically independent of other pixels.

Luca had a whole story about the noise model, but I was too tired to follow. You encode image noise but not additional white noise.

Encoder-decoder structure can be used for denoising. Guess the point is that once you capture the latents,

Depends on your distinction between signal and noise. Typically, all image info is signal, unless you know you e.g. have blurred images as input and you want to remove the blur.

If you marginalize out the latent, you reintroduce the correlation between pixels.

$\mu$ is usually called the decoder, because it takes the latent vector as input and synthesizes an output from that (by sampling?). Decoder is usually a neural network.

We usually pick simple priors. All complexity is already captured in the decoder network.

Trade-off between z in prior, value of x acquired by transforming z. If we keep the prior over z simple, we can model most complexity with the decoder model. You can also make z more complex and capture some image statistics. But then you have to acquire a good prior before.

We usually apply some simplifications to make the estimator more reliable.

ELBO can be expressed as sum of two terms.

TODO: Review last part. Luca lost connection, and afterwards I couldn't be bothered anymore.