Reviewer #3 (Public review):
Summary:
This paper points out an inconsistency of the roles of the striatal spiny neurons projecting to the indirect pathway (iSPN) and the synaptic plasticity rule of those neurons expressing dopamine D2 receptors, and proposes a novel, intriguing mechanisms that iSPNs are activated by the efference copy of the chosen action that they are supposed to inhibit.
The proposed model was supported by simulations and analysis of the neural recording data during spontaneous behaviors.
Strengths:
Previous models suggested that the striatal neurons learn action values functions, but how the information about the chosen action is fed back to the striatum for learning was not clear. The author pointed out that this is a fundamental problem for iSPNs that are supposed to inhibit specific actions and its synaptic inputs are potentiated with dopamine dips.
The authors proposes a novel hypothesis that iSPNs are activated by efference copy of the selected action which they are supposed to inhibit during action selection. Even though intriguing and seemingly unnatural, the authors demonstrated that the model based on the hypothesis can circumvent the problem of iSPNs learning to disinhibit the actions associated with negative reward errors. They further showed by analyzing the cell-type specific neural recording data by Markowitz et al. (2018) that iSPN activities tend to be anti-correlated before and after action selection.
Weaknesses:
(1) It is not correct to call the action value learning using the externally-selected action as "off-policy." Both off-policy algorithm Q-learning and on-policy algorithm SARSA update the action value of the chosen action, which can be different from the greedy action implicated by the present action values. In standard reinforce learning terminology, on-policy or off-policy is regarding the actions in the subsequent state, whether to use the next action value of (to be) chosen action or that of greedy choice as in equation (7).<br /> It is worth noting that this paper suggested that dopamine neurons encode on-policy TD errors: Morris G, Nevet A, Arkadir D, Vaadia E, Bergman H (2006). Midbrain dopamine neurons encode decisions for future action. Nat Neurosci, 9, 1057-63. https://doi.org/10.1038/nn1743
(2) It is also confusing to contract TD learning and Q-learning, as the latter is considered as on type of TD learning. In the TD error signal by state value function (6) is dependent on the chosen action a_{t-1} implicitly in r_t and s_t based on the reward and state transition function.
(3) It is not clear why interferences of the activities for action selection and learning can be avoided, especially when actions are taken with short intervals or even temporal overlaps. How can the efference copy activation for the previous action be dissociated with the sensory cued activation for the next action selection?
(4) Although it may be difficult to single out the neural pathway that carries the efference copy signal to the striatum, it is desired to consider their requirements and difference possibilities. A major issue is that the time delay from actions to reward feedback can be highly variable.
An interesting candidate is the long-latency neurons in the CM thalamus projecting to striatal cholinergic interneurons, which are activated following low-reward actions:<br /> Minamimoto T, Hori Y, Kimura M (2005). Complementary process to response bias in the centromedian nucleus of the thalamus. Science, 308, 1798-801. https://doi.org/10.1126/science.1109154
(5) In the paragraph before Eq. (3), Eq (1) should be Eq. (2) for the iSPN.
Here are comments back to the authors' replies with the revised version:
(1) I do not agree on the use of inaccurate technical terms. On-policy does not require that the policy is greedy with respect to the actions values, as authors seem to assume here.
In fact, the policy (10) is just a standard soft-max action selection based on the action values by the difference of dSPN and iSPN outputs.
Furthermore, in the immediate reward setting tested in this paper, action values are independent of the policy, so there is no distinction between on-policy vs. off-policy. This is also apparent from the "TD" errors in (19) and (21), where there is no TD.
(2) To really compare the different forms of TD, multi-step RL tasks should be used.
(3) This fundamental limitation should be explicitly documented in the manuscript. This is not just the same as any RL algorithms. Having two action representations within each action step make temporal credit assignment more difficult.