Temporal difference learning
From Foster et al., A Model of Hippocampally Dependent Navigation, Using the Temporal Difference Learning Rule
TD-learning is:
- "error-driven"
- "used in feed-forward neural networks"
- "in which input patterns (e.g., patterns of place cell activity) are to be associated with output values (e.g., an expectation of how close the goal is)"
- "where additionally there is information to be had in the sequence in which input patterns and output values present themselves." (2)
TD-learning is not supervised because it does not require a teacher signal available at all times.
TD uses "an error based on the difference between successively occurring output values---a sensible strategy when a consistent relationship between these values is expected." (2)
A problem in a rat maze task (or any RL task with only a single reward) is the sparsity of the reward signal. So when we encounter the goal, how do we update the expectation of reward at other locations?
The idea behind TD-learning is that you use the local consistency assumption for this. Independent of whether we move towards or away from the goal, we know that there must be a "temporal gradient" of the expectation of reward between successive locations. (cf. 2)
TD-learning can be seen as an extension of "the Rescorla-Wagner rule to the temporal domain" (2)
TD-learning "provides an explanation for second-order conditioning, whereby a conditioned stimulus (CS) that has acquired predictive value can itself condition another preceding CS."
In the maze tasks, place cells become predictive CS of reward; also when they are followed by other place cells that are in turn themselves predictive of reward (so not only when followed by the goal).
TD-learning in actor-critic model
Can be combined with an actor-critic model, e.g. in Foster et al, 2000.
Critic learning
Firstly, TD-learning can be used to learn the value function of the critic. The proposal of Foster et al. is that hippocampal place cells fill in the role of learning the value function.
Critic has a single output cell that provides a value for a location p. This is a weighted sum of the firing rates of all place cell inputs $f_i(p)$ with $w_i$ being the weight of the i-th place cell:
$$C(p) = \sum_i w_i f_i(p)$$
This means that these place cells must encode the value function, so that $C(p)=V(p)$. To do this, we can use TD-learning on the place cell weights in $C(p)$.
TD works by enforcing consistency between successive critic outputs.
We have value function:
$$V(p_t) =
Where the brackets indicate the average over trials.
From this it follows that for successive steps:
$$V(p_t) =
Because we desire $C(p)=V(p)$ the same condition should hold for the critic's output (i.e. the output from the place cells):
$$C(p_t) =
For this to hold we need:
$$
This allows us to define a prediction error solely based on the successive critic outputs.
$$ \delta_t = R_t + \gamma C(p_{t+1}) - C(p_t)$$
N.B. But notice that here we do not have access to $
This can further be simplified by making a case distinction.
We only have $R_t = 1$ at the goal. Because this ends the trial we then have $V(p_{t+1})=0$ and should have $C(p_{t+1})=0$
So with $R_t = 1$ we have
$$ \delta_t = 1 - C(p_t)$$
and with $R_t=0$ we have
$$ \delta_t = \gamma C(p_{t+1}) - C(p_t)$$
Once we actually know $\delta_t$, we use it to update the weights of the active place cells:
$$\Delta w_i \propto \delta_t f_i(p_t)$$
"Under various conditions on the learning rate and on the representation provided by the place cells, this rule is bound to make C(p) converge to the value function V(p) as required" (5)
What are those constraints?
"Following standard reinforcement learning practice, we use a fixed learning rate to avoid slow learning. The price to be paid is residual error; however, the results show that this error is insignificant." (5)
I don't really see how a fixed learning rate per se alleviates slow learning.
Is f the activation function here?
Actor learning
Before we saw that the value function at a cell p depended on the activation of the place cells with weights $w_i$. Similarly, the activation of each action cell (e.g. 8 for 8 directions) depends on the place cells with weights $z_{ji}$ from place cell i to action cell j.
$$a_j(p) = \sum_i z_{ji}f_i(p)$$
Each cell will be associated with a probability i.e. a preference for that action. These are all weighted relative to each other (basically a softmax) so that we can pick an action stochastically. This ensures the actor tries enough different values.
Now the challenge is to find out which actions are most desirable, i.e. which action leads to a larger expected value according to V(p)?
Crux: the prediction error of the critic is an appropriate signal to give to the actor for deciding which action is best!
We had the following prediction error:
$$ \delta_t = R_t + \gamma C(p_{t+1}) - C(p_t)$$
If we assume that the critic has learned the correct value function, so $C(p)=V(p)$, what does delta tell us, in the scenarios where we have not yet reached the goal?
With $R_t=0$ we had:
$$ \delta_t = \gamma C(p_{t+1}) - C(p_t)$$
So when $d_t > 0$, we know that $\gamma C(p_{t+1}) - C(p_t) > 0$ so that $\gamma V(p_{t+1}) > V(p_t)$; in other words, the successive step is a better one. Conversely, when $\delta_t < 0$ we know the successive step is worse.
Weight updating:
$$\Delta z_{ji} \propto \delta_i f_i(p_t)g_j(t)$$
with $g_j(t)=1$ if action j was chosen at time t and 0 otherwise.
This amounts to Hebbian learning because the strength between a place cell and action cell is changed only when they fire together and not otherwise. When they fire together, the strength is increased when there is an increase in value (positive delta) and decreased otherwise.
Learning Actor and Critic simultaneously
See Foster p.6.
- We have not yet incorporated that agent actions are not constant; their relative preferences change
- We assumed the critic learned the correct value function
"Given that both mechanisms must work together, it has been suggested that learning in the actor should proceed much more slowly than in the critic" (6)
(... for the assumption that the critic learned the value function already to be valid).
Foster et al. argue however that this is not necessarily true, because the critic's value function is still useful even when not correct yet. For example, when near the platform (thus near the reward signal) it can still lead to improvement in the actor.