episodic memory encod-ing and retrieval

stimulus-locked P2-N2 complex and theresponse-locked ERN/CRN

may fail to communicate common features of these com-ponents

differentiated literature of action-monitoring ERPs

may reflect disparate processes in unique cognitivecircumstances, these aforementioned processes have all been spe-cifically associated with ACC function during attention orientationand/or action selection

the integration of contextual cues with actionselection to optimize goal-driven performance

These differences cannot beattributed to greater engagement of the NE system in the ActiveExperiment (rather than greater DA system engagement) becausegreater NE release would have produced alargernegativity toinfrequent reward feedback

in the Active Experiment, the rawN2 to infrequent reward feedback was significantly smaller thanthe raw N2 to infrequent reward feedback in both the Passive andModerate Experiments, suggesting that greater DA system engage-ment resulted in a larger DA-associated positivity that attenuatedthe raw N2.

interaction of reward condition andfrequency condition such that the effect of reward was largerin the infrequent condition than in the frequent condition

dN2in the Moderate Experiment trended toward being significantlylarger than the dN2 in Active Experiment,

identical task stimuli (colored faces) presented with identicaltask designs

variablescalp distribution dependent on relative engagement of the dif-ferent cortical areas giving rise to the ERP

LC refractory period coincides withP3 generation

increase in amplitude of the N2

By taking into account P3b versusP3a effects and latency information, it may be possible to con-sider surprise in the context of other mental states contributing togoal-oriented behavior

cannotdetermine whether the P300 modulation was purely due to thesurprise conveyed by the visual stimuli, or whether it was relatedto the response selection on each trial

unexpected changes in the worldwithin the context of a task

P300 reflects the arrival of a phasic norepinephrine (NE) signal incortical areas, which serves to increase signal transmission in thecortex

compared our model to an alternative mea-sure of surprise based on the Kullback–Leibler divergence.

This evidence was used to compare competing models de-fined in terms of the explanatory variables inZ1.

identity of trials on which participants responded er-roneously, trials that were rejected during the preprocessing of the EEGdata, and a constant term.

regressor models variance related to stimulus probabilitywithin a block and does not take into account any learning

quantified in terms of how much they change posteriorbeliefs.

Harrison et al. (2006), where thecurrent event depended on the previous

Dj1, can beused to predict the probability of each event occurring

nkjrefers to the number of occurrences of outcomekup untilobservationj

time point at which the averaged P300s were mod-ulated maximally

All stimuli occurred equally oftenover the course of the experiment

not informed

updating of task-relevant information inanticipation of subsequent events

P300 has commonly been linked to therevision of a participant’s expectation about the current task con-text

Thepresent findings suggest that the variance in fERN amplitudeacross conditions results more from the effect of unpredictedpositive feedback than from unpredicted negative feedback

essentialdifference between conditions is associated with neural activityon correct trials rather than neural activity on error trials

error–oddball differ-ence should be smaller than the correct–oddball difference

what causesthe absence of the fERN/N200 on correct trials

in-frequent targets in an oddball task and infrequent error feedbackin a time estimation task both elicit a frontal or frontal-centrallydistributed, negative-going component that reaches maximumamplitude at approximately 280–310 ms.

infrequent oddball ERPs from the infrequent correctERPs.

subtracted the latency-corrected infrequent oddball ERPs fromthe infrequent error ERPs.

maximum negative value of the ERP recorded atchannel FCz within a 150–350-ms window

virtual-ERNs

first by examining the ERPs directly, and second by per-forming a spatial principal components analysis (PCA)

difference between the ERP oncorrect trials and the N200 should be larger than the differencebetween the fERN and the N200

it is equally possible that thedifference between the ERPs on correct and incorrect trials arisesfrom a process associated with correct trials rather than witherror trials

entirelydifferent ERP component as the source of the apparent variancein fERN amplitude.

fERN is elicited by unexpected negativefeedback stimuli, but not by unexpected positive feedback stimuli

N200 increases in pro-portion to the unexpectedness of the event

we refer here to the negativedeflection that is seen in so-called oddball tasks

confound fERN amplitude with the P300

resulted from an increase in the amplitude of thefERN, rather than from overlap with a different ERP compo-nent (such as the P300).

nteraction of valence with expec-tancy

no main effect ofexpectancy

main effect of valence

Participants made more errors in the hardcondition (76%) than in the easy condition (23%)

correct ERP in the easy condition fromthe error ERP in the hard condition

correct ERP in the hard conditionfrom the error ERP in the easy condition

correct feedback in the hard con-dition

sensory events [80]

C wasassociated with X before any association with food

SR represents the association between the stimulusandfood,andisalsoabletoupdatetherewardfunctionofthefood as a result of devaluation

A reduced acquisition of conditionedresponding to C and D, compared to F, which was trainedin compound with a novel stimulus

shifts in value (amount of reward) and identity(reward flavour).

although the RPE correlate has famously been evident insingle units, representation of these more complex or subtleprediction errors may be an ensemble property.

First, it naturally captures SPEs, as we will illus-trate shortly. Second, it also captures RPEs if reward is oneof the features.

expected TD error isthen proportional to the superposition of feature-specific TDerrors,PjdMt(j).

Althoughprediction errors are useful for updating estimates of thereward and transition functions used in model-based algor-ithms, these do not require a TD error.

building on the pioneeringwork of Suri [46], we argue that dopamine transients pre-viously understood to signal RPEs may instead constitutethe SPE signal used to update the SR

dopamine transients are necessary for learn-ing induced by unexpected changes in the sensory features ofexpected rewards [37]

sensitive to movement-related variables such as action initiation and termination

some dopamine neuronsrespond to aversive stimuli

natomically segregated projection frommidbrain to striatum

value is affected by novelty [21] or uncertainty [22]

Finally, to test if the differences in these measures are sufficiently robust to allowcategorisation, we trained a support vector machine (SVM) classifier to accurately predict trajectories as either model-free, model-based or SR agents (Fig 6A-B). When the decoder was given data from the biological behaviour, the SVM classifier most frequently predicted those trajectories to be an SR agent

model-based algorithm was consistently the most successful

attributes a role for dopamine in forming sensory predictionerrors (Gardner et al., 2018) - similar to what has been observed experimentally(Menegas et al., 2017; Sharpe et al., 2017; Takahashi et al., 2017).

This abs(RPE) signal correlated with anumber of brain regions including an IPS locus anterior to wherewe found SPE correlates at p < 0.001 uncorrected (Figure S3).However, a direct comparison between SPE and abs(RPE) re-vealed a region of both posterior IPS that was significantly betterexplained by the SPE than by the abs(RPE) signal at p < 0.05corrected, as well as a region of latPFC that showed a differenceat p < 0.001 uncorrected

the degree to which pIPS encodes an SPE representationNeuronReward and State Prediction Errors in HumansNeuron66, 585–595, May 27, 2010ª2010 Elsevier Inc.589

partici-pants had indeed acquired knowledge about the particularsequence of state transitions during the first session: 99.6% ofpermutation samples provided a poorer explanation of choicesthan the original (p = 0.004).

If participants’ beliefsabout the transition probabilities were updated by error-drivenmodel-based learning (with a fixed learning rate, as assumedin FORWARD), this may have left a bias toward the most recentlyexperienced transitions

exponential decay fromFORWARD to SARSA

volunteers were firstexposed to just the state space in the absence of any rewards,much as in a latent learning design. This provides a pure assess-ment of an SPE

“replay”trajectories that the animal has taken in the past, as well as to follow novel trajectories that the animal has neverbefore taken

online planning process, perhaps similar to tree search or to online dynamic programming

Theta sequences typically occur when an animal is moving

monkeys use a joystick to control a digital predator pursuing digital prey

dlPFC activity correlates with a “state prediction error”

silencing OFC activity on a particular trialselectively impairs the influence of this expected outcome signal on learning but does not impair choosing.

“inference”: the ability to combine separately-learned associationsin order to form new associations between items that may never have been encountered together before​

associate stimuli or actions with specificexpected outcomes

Multi-stepplanning can be thought of as a process that uses this map to guide an extended sequence of behavior towards adistant goal

other lines

Notably, these signals were signed based on the subject’sgoal, consistent with a mechanism for determining how much toupdate beliefs and in which direction: toward confirmation (pos-itive) or reconsideration of one’s rewarding goal (negative)

but may not store the model locally

extracted the feedback-locked BOLD response in left lOFC (at 6 s post-feedback onset)at trialtand regressed this against the (signed) change to hippo-campal CSS (i.e., the change in the difference between LC andHC presentations in [ipsilateral] left hippocampus from the pre-ceding blockt
0.5 to the subsequent blockt+ 0.5)

the unsignedDKLterm, corresponding to the magni-tude of the belief update, independent of its direction, instead re-cruited a dorsal frontoparietal network, consistent with previousfindings related to unsigned state prediction errors during latentlearning

stimulus-outcome update effects in lateral OFC/ventro-lateral prefrontal cortex (VLPFC) and also a distributed networkincluding anterior cingulate cortex, inferior temporal cortex, andposterior cingulate cortex

reduction in the BOLD response for HC items when comparedto LC items

and the other inferred based on thesubject’s knowledge of the inverse relationship between stimuliand outcomes dictated by the task structure

In each CSS block, each stimulus-outcome transition was pre-sented once

firstselect the more desired gift card goal based on the current po-tential payouts and then reverse-infer the stimulus they believedwould most likely lead to that desired outcome

but not about the reward amountobtained on a gift card

advantageous to learn the transition probabilities

reward-size-in-dependent stimulus-outcome associations

little is known about how these different signalsare used in the brain

facilitate theupdating of internal models from which future action is gener-ated (55)

The currentfindings suggest a specific computational functionfor the ACC: It is involved in updating internal models to fa-cilitate future information processing.

“reset”signal for internal models

Participants’internal models of the targets’distribution couldbe expected to differ from the true (generative) distribution

dwell time reflectsupdating

no further behavioral cost onsubsequent trials

the two types could be easily distinguished

mPFC may track changes in activity patterns in regions with community-based representational similarity, providing a signal that could underlie parsing decisions.

three-layer neural network model (Fig. 6a). The network took input representing the current stimulus and was trained to predict which stimulus would occur next.

within the set of Hamiltonian paths, the probability of transitioning from one cluster boundary node (one of the pale nodes in Fig. 1a) to the adjacent one, if not yet visited, is always exactly

passage into a new cluster significantly more often than at other times

set of possible successor items on each step depends only on the current item, this uniformity in transition prob-abilities holds whether one takes into account only the most recent item or the n most recent items

items will fall close together in representational space when they are preceded and followed by similar distributions of items in familiar sequence

non-uniform transition probabilities.

judgments are quite reliable

different account,

transient elevations in predictive uncertainty or surprise as the primary signal driving event segmentation.

educed the total number of stimulus–stimulus transitions and therebyincreased statistical powe

the next day, subjects were presented with object sequences in the scanner

instructedto remember which of two buttons to press for a particular object orientation

prediction errors in the orbitofrontal cortex dur-ing active learning predict later changes in hippocampal representations of the stored model(Boorman et al., 2016)

Indeed, we note that neural signals can be recorded in frontal and parietal cortices, reflectingthe ‘state-prediction errors’ that ensue when predicted state relationships are breached duringbehavioural control (Gla ̈scher et al., 2010)

We hypothesised that implicit knowledge about the graph structurewould influence response times, such that subjects would respond faster if a preceding object in thetest sequence was closer on the graph structure underlying the train sequence. Indeed, we foundthat log-transformed response times were longer the further away the preceding object was on thegraph (Figure 5C, D)

communicabilitysignificantly distorts the graph structure by shortening links that form part of many paths around thegraph structure and lengthening links that would be less frequently visited by a random navigator

it was the symmetrised version alonethat predicted the fMRI suppression effect

was also present behaviourall

fMRI adaptation paradigm

can be read out directly from functional magnetic resonance imaging(fMRI) data in the entorhinal cortex

SR in com-plimentary learning systems, especially in the medial PFC and the hippocampus

rela-tive balance between ‘state prediction errors’ and ‘reward prediction errors’ may be used for arbitration in an MB–MF hybrid learner

we chose to linearly combine the ratings from MB and SR algorithms

response times were slower in the transition revaluation condition compared with both the reward revalu-ation condition (t57= 2.08, P< 0.05) and the control condition (t57= 4.04, P< 0.00

reward revaluation

transition revalua-tion condition

indicate which starting state

indicate their preference

Experiment 1 used a passive learning task, which permitted the simplest possible test of the the-ory, removing the need to model action selection.

Specifically, they learn and store a one-step internal representation or model of the short-term environmental dynamics: specifically, a state transition function T and a reward function R

posterior dorsomedial striatum (Oh et al., 5312014; Hintiryan et al., 2016), a region necessary for learning and expression of goal directed action 532as assessed by outcome devaluation

Likewise, neuroimaging in a saccade task in which subjects constructed and updated a model of the 522location of target appearance observed ACC activation when subjects updated an internal model of 523where saccade targets were likely to appear, (O’Reilly et al., 2013)

, neuroimaging in the Daw two-step task has identified representation of model-based value 520in the BOLD signal in anterior- and mid-cingulate regions

In 465humans, extensive training renders apparently model-based behaviour resistant to a cognitive load 466manipulation (Economides et al., 2015) which normally disrupts model-based control (Otto et al., 4672013), suggesting that it is possible to develop automatized strategies which closely resemble 468planning.

ACC inhibition on model-based control because the effects would not survive multiple comparison 428correction for the large number of model parameters.

xed and are known to be so by 302the human subjects

The absence of transition-outcome interaction has been used in the 298context of the traditional Daw two-step task (Daw 2011) to suggest that behaviour is model-free. 299However, we have previously shown (Akam et al. 2015) that this depends on the subjects not 300learning the transition probabilities from the transitions they experience.

direct biases of 286choice

Subjects learned the task in 3 weeks

Reversals in which first-step action (high or low) had higher reward 216probability, could therefore occur either due to the reward probabilities of the second-step states 217reversing, or due to the transition probabilities linking the first-step actions to the second-step states 218reversing.

except on transitions to neutral 214blocks, 50% of which were accompanied by a change in the transition probabilities

At block transitions, either 213the reward probabilities or the transition probabilities changed

introduction of reversals in the 189transition probabilities mapping the first-step actions to the second-step states. This step was taken 190to preclude subjects developing habitual strategies consisting of mappings from second-step states 191in which rewards had recently been obtained to specific actions at the first step (e.g. rewards in 192state X  chose action x, where action x is that which commonly leads to state X). Such strategies 193can, in principle, generate behaviour that looks very similar to model-based control despite not using 194a forward model which predicts the future state given chosen action (

block-based reward probability distribution

in each second-step state there was a single action rather than a 181choice between two actions available, reducing the number of reward probabilities the subject must 182track from four to two

model-based mechanisms 171which learn action-state transition probabilities and use these to guide choice.

optogenetic silencing of ACC neurons on individual trials reduced the influence of the 110experienced state transition on subsequent choice without affecting the influence of the trial 111outcome

in depth computational analysis 101(Akam et al., 2015

developing a new version in which both the reward probabilities in the leaf states of the decision 103tree and the action-state transition probabilities change over time.

(ACC), a region expected to be centrally involved

representing task contingencies beyond model-57free cached values

Firstly, the ACC provides a massive input to posterior dorsomedial 54striatum (Oh et al., 2014; Hintiryan et al., 2016), a region critical for model-based control as assessed 55through outcome-devaluation

failurestoflexiblyupdatedeci-sionpoliciesthatarecausedbycachingofeitherthesuccessorrepresentation(asinSR-TDorSR-Dynawithinsufficientreplay)ora decisionpolicy(asinSR-MB)shouldbeaccompaniedbyneuralmarkersofnon-updatedfuturestateoccupancypredictions

unlikethehippocampus,partsofthePFCappeartobeinvolvedinactionrepresentationinadditiontostaterepresentation

valueweightswouldbeleanedbyneuronsconnectingthehippocampustoventralstriatum,inthesameTDmannerdiscussedinthispaper

fMRImeasuresoftherepresentationofvisualstimuliintaskswheresuchstimuliarepresentedsequentially

[74]demonstratedthata sophisticatedrepresentationthatincludesrewardhistorycanpro-ducemodel-basedlikebehaviorinthetwo-steprewardrevaluationtask

moresophisticatedstaterepresentations

predictionerrorrelatedBOLDsignalsinhumans

nsteadstoredinprefrontalcortex

successormatrixupdatedbySR-Dynamightitselfexistintherecurrentconnectionsofhippocampalneurons

SR-Dynacansupportrapidactionselectionbyinspectingitslookuptable

ThissimulationdemonstratesthatSR-Dynacanthusproducebehavioridenticalto“full”model-basedvalueiterationinthistask

recurrentneuralnetworksoffera simplewaytocomputeMπ(s,:)basedonspreadingactivationimplementingEq11.

SR-MBcannotsolvethenovel“policy”revaluationtask

SR-TDandSR-MBarethus“on-policy”methods–theirestimatesofVπcanbecomparedtotheestimatesofa traditionalmodel-basedapproach

SR-MBlearnsa one-steptransitionmodel,Tπandusesit,atdecisiontime,toderivea solutiontoEq9

combinetheSRwithfunctionapproximationanddistributedrepresentations

Crucially,despitethefunctionalsimi-laritybetweenthisruleandtheTDupdateprescribedtodopamine,wedonotsuggestthatdopaminecarriesthisseconderrorsignal

standarddopaminergicTDrule

agentis allowedtoexperiencethischangeonlylocally

BecauseMπreflectslong-runcumulativestateoccupancies,ratherthantheindividualone-steptransitiondistribution,P(s’|s,a),SR-TDcannotadjustitsvaluationstolocalchangesinthetransitionswithoutfirstupdat-ingMπatdifferentlocations.

SR-TDcan,withoutfurtherlearning,producea newpolicyreflectingtheshortestpathtotherewardedlocation

firstexploresthegrid-worldran-domly,duringwhichit learnsthesuccessormatrixMπcorrespondingtoa randompolicy

gMpðs0;:Þ

Thethreemodels

Tπis theone-stepstatetransitionmatrixthatis dependentonπ

approximationwillbecorrectwhentheweightw(s0) foreachsuccessorstatecorrespondstoitsone-stepreward,averagedoveractionsins

learnedrewardvaluesinventralstriatum[53]

Doya[51]introduceda circuitbywhichprojectionsviathecerebellumper-formonestepofforwardstateprediction,whichactivatesa dopaminergicpredictionerrorfortheanticipatedstate

andneuro-imagingofpredictionerrorsignalsinhumanstriatum[6]

thusper-hapsinvolvinganalogous(striatal)computationsoperatingoverdistinct(cortical)inputrepre-sentations[47].

Typically,model-basedmethodsareoff-policy(sincehavinglearnedaone-stepmodelit is possibletouseEq2 todirectlycomputetheoptimalpolicy);whereasdif-ferentTDlearningvariantscanbeeitheron-oroff-policy.

butalsointhemoreflexi-blechoiceadjustmentsthatseemtoreflectmodel-basedlearning

n previousversions, subjects at each stage chose between two symbols insteadof two fixed actions and the symbols moved from side to side ateach trial ensuring there was no consistent mapping between thebutton presses and the symbols

decisions that are insensitive to (i) the valuesof the outcomes [8] and (ii) the contingency between specificactions and their outcomes

abitual (when actionsequences are selected) and goal-directed (when single actions areselected) action

a first stage action is the best action

with a small probability (1:7), the rewardingprobability of each key changed randomly to either the high or lowprobability.