Actor-Critic model
In environments with a sparse reward signal, the actor-critic model can be used to estimate rewards.
The actor simulates actions in the environment; the critic scores those actions. The actor adjusts its actions according to the critic; the critic responds to the actor's actions.
The role of the critic is to serve as an intermediary of the sparse reward signal, because the reward signal itself is not informative enough to fulfill this role due to its sparsity.
Example: Foster et. al.
Critic has a single output cell that provides a value for a location p. This is a weighted sum of the firing rates of all place cell inputs $f_i(p)$ with $w_i$ being the weight of the i-th place cell:
$$C(p) = \sum_i w_i f_i(p)$$
Note that place cells overlap and usually have a smooth activation. E.g. they could be modeled as Gaussians around a center where they are most sensitive.
The goal of the critic is to learn the value function V(p) over current location p, that evaluates a proposed coarse of action by the actor.
V(P) is usually defined as "for any location P, the discounted total future reward that is expected, on average, to accrue" after following the proposed course of action from p.
Note: the discounting factor makes sense, because otherwise the critic always suggests that making more steps infinitely increases the expected reward. Also, the further in the future, the more uncertainty, I guess?
$$V(p_t) =
Where the brackets indicate the average over trials.
If we only have a single reward signal (i.e. a single goal) at time T, then most of these terms drop out.
E.g. take T=3 and t=1, then $V(p_1) = 0 + 0 + \gamma^2*1$. Note this equals $\gamma^{3-1}$. We thus see that we end up with the simplified formula:
$$V(p_t) = <\gamma^{T-t}>$$
Because $\gamma$ is a constant between 0 and 1, the value function is monotonically increasing (i.e. each step closer to the goal will have less discounting steps and thus will be strictly larger).
Because steps closer to the goal have a higher value, this value function can guide the actor's actions.
The value function of the critic has to be learned in some way.