Variational Timeseries Analysis and structured variational inference

We want to account for (observational) noise.

$$p(y_t | x_t) = N(y_t; x_t, \sigma_{lk}^2)$$

MLE solution:

$$\hat{x_{t}} = argmax\ log p(y_t | x_t)$$

The problem with this solution is that it cannot "denoise" the measurements and thus will be unreliable when the variance $\sigma_{lk}^2$ is higher.

We can reduce the effect of measurement noise by choosing an appropriate prior.

Because with time series we don't have a static mean, it is very important to define the prior with respect to the previous position. I.e. how likely is your current location given your past location?

$$p(x_{t} | x_{t-1})$$

If we can assume some form of continuity in the movements (i.e. no sudden large teletransportations) then we can model the prior probabilities of a particular move with a Gaussian random walk model.

$$p(x_{t} | x){t-1} = N(x{t} | x_{t-1}, dx^2)$$

The meaning of the variance/scale parameter $dx$ is the average step size per timestep, i.e. it indicates an average velocity $dx/dt$.

This assumption of continuity is extremely basic, but this already helps denoising observations a lot.

We can now define the joint model that captures the probability of 1) your actual position (captured by prior) and 2) the measured positions (with observation noise captured by the likelihood).

$$p(x_{1:T}, y_{1:T} ) = \prod_{t=1}^T N(y_t; x_t, \sigma_{lk}^2) N(x_{t} | x_{t-1}, dx^2)$$

Because the prior links the current timepoint with the previous timepoint, it has a smoothing effect.

Ultimately, the goal is to predict the "true" or "denoised" position $x_{t}$ based on a noisy measurement $y_{t}$. I.e. once we have the joint $$p(x_{1:T}, y_{1:T} )$$ we want to compute the posterior $$p(x_{1:T} | y_{1:T} )$$. Because of the smoothing/denoising effect of the prior, this predictive posterior is also called the smoothing distribution.

(Notice that $y_t$ is here an observed data point... usually it's reserved for targets, but OK.)

ELBO

As a variational distribution, we can just use the mean field approximation and define a separate independent Gaussian for each timepoint.

$$q(x_{1:T} | \mu_{1:T}, \sigma_{1:T}) = \prod_{t=1}^T N(x_t | \mu_{t}, \sigma_{t}^2)$$

  • Write out ELBO as usual
  • Then apply reparameterization as usual.

Gradients and closed form solution

Normally, the reparameterized ELBO is all we need because we can use autograd software. In case of the mean field assumption, we can also do easily find the closed form solution.

The prior leads to an underestimation of variance, because assuming to know the average dx at a particular time implies that you also have exact data from $x_{t+1}$. But you don't. In reality you only have measurements that come with the additional noise.

Latent Hidden Markov timeseries models

The previous random walk model was an example of an hidden markov model.

Nice insight: what makes a Brownian model a random walk is that the beste possible prediction we can make for its future values given the current value is the current value itself.

This is an example of a linear autoregressive model.

$$p(x_{1:T}; x_0) = \prod_{t=1}^T N(x_{t-1}; \alpha( \beta - x_t); \sigma^2)$$

For $\alpha >0$ the process tends to equilibrium point $\beta$. For $\alpha < 0$ the process will show exponential movement to $+\infty$.

However, in this case we assumed the variance of the process to be stable. For timeseries in the real world this is very unlikely to hold!

Instead, we can also model the time dependency of the variance itself (i.e. model $\sigma_t$ instead of $x_t$)!

$$p(x_{1:T}, \sigma_{1:T}; x_0, \sigma_0) = prod_{t=1}^T N(\sigma_{t}; \alpha( \beta - \sigma_{t-1} v^2) \cdot N(x_{t}; x_{t-1}, \sigma_{t-1}^2)$$

i.e. "the variance of the volatility process is scaled by the previous volatility value. This ensures that the process is scaled by the previous volatility value."

So what we say is essentially that we can define a complex time-dependent dynamics both on the mean and variance of the transition probabilities of a HMM. We can learn this dynamics with deep learning, based on observed data.

$$p(x_{1:T}; x_0) = \prod_{t=1}^T N(x_t; f_{\mu}(x_{t-1}), f_{\sigma}^2(x_{t-1})$$

Observation models

In the previous we were talking about the latent dynamical process underlying time series. In practice, we don't have access to this underlying model, but instead to noisy observations of data points that are emitted by the underlying process and measured by some means.

To capture this, we can have a observation model or emission model.

$$p(y_{1:T}|x_{1:T}) = \prod_{t=1}^T p_t(y_t|x_t)$$

If our measurement device is unbiased, we can take the mean of the latent process as the mean of the observation model. The variance then captures the measurement error of the device.

This allows us to write the joint of the latent variables and the observations.

Structured variational distribution and inference

When using the variational approach, we define the variational distribution $q$ again. This captures statistical dependencies in the variables in the posterior.

$$q(x_{1:T}; \theta_{1:T}) = \prod_{t=1}^T q(x_t | x_{t-1}; \theta_t)$$

"Using structured distributions is particularly important in timeseries models since the prior usually has a strong auto-correlation structure, which would result in a severe underestimation of the posterior variance if the mean field approach is used. The loss function of the resulting variational optimization is the following evidence."

When reparameterizing, the time series model is slightly more difficult because we have the additional dependency on $x_{t-1}$.

We have seen that the mean field approximation leads to a severe underestimation of the variance. However, captures more complex dynamics in a more structured variational distribution is hard.

This section however states that many of the statistical dependencies in the posterior are determined by the prior anyways, so in practice it is also possible to define a fixed parameterized density $\pi(x_t;\phi)$ with $\phi$ being a linking function that determined the transition probabilities based on the previous value $x_{t-1}$.

The Brownian motion is captured by a Gaussian density with link:

$$\phi_{bm}(x_{t-1}) = (x_{t-1}, \sigma_{bm}^2)$$

You can also have a trade-off parameter $\lambda$ that balances the prior link with a learnable "pertubation parameter" that accounts for the influence of the data. This allows you to balance the influence of the prior and the mean-field assumption.