Neuroscience and Biobehavioral Reviews 37 (2013) 2071–2080
Contents lists available at ScienceDirect
Neuroscience and Biobehavioral Reviews
jou rn al h om epage: www.elsev ier .com/ locate /neubiorev
eview
he clinical relevance of neuroplasticity in corticostriatal networks duringperant learning
atthew E. Andrzejewskia,∗, Brenda L. McKeeb, Anne E. Baldwinc, Lindsay Burnsd, Pepe Hernandeze
University of Wisconsin-Whitewater, Whitewater, WI, United StatesEdgewood College, Madison, WI, United StatesState University of New York at Geneseo, Geneseo, NY, United StatesPain Therapeutics, Inc., Austin, TX, United StatesUniversity of Pennsylvania, Philadelphia, PA, United States
r t i c l e i n f o
rticle history:eceived 28 November 2012eceived in revised form 14 March 2013ccepted 27 March 2013
eywords:perant learninglutamateopaminelasticityat
a b s t r a c t
Dopamine and glutamate serve crucial functions in neural plasticity, learning and memory, and addic-tion. Contemporary theories contend that these two, widely-distributed neurotransmitter systems playan integrative role in motivational and associative information processing. Combined signaling of thesesystems, particularly through the dopamine (DA) D1 and glutamate (Glu) N-methyl-d-aspartate receptors(NMDAR), triggers critical intracellular signaling cascades that lead to changes in chromatin structure,gene expression, synaptic plasticity, and ultimately behavior. Addictive drugs also induce long-termneuroadaptations at the molecular and genomic levels causing structural changes that alter basic con-nectivity. Indeed, evidence that drugs of abuse engage D1- and NMDA-mediated neuronal cascades sharedwith normal reward learning provides one of the most important insights from contemporary studies onthe neurobiology of addiction. Such drug-induced neuroadaptations likely contribute to abnormal infor-mation processing and behavior, resulting in the poor decision-making, loss of control, and compulsivitythat characterize addiction. Such features are also common to many other neuropsychiatric disorders.Behavior problems, construed as difficulties associated with operant learning and behavior, present com-pelling challenges and unique opportunities for their treatment that require further study. The present
review highlights the integrative work of Ann E. Kelley and colleagues, demonstrating a critical role notonly for NMDAR, D1 receptors (D1R), and their associated signaling cascades, but also for other Glu recep-tors and protein synthesis in operant learning throughout a cortico-striatal-limbic network. Recent workhas extended the impact of appetitive learning to epigenetic processes. A better understanding of theseprocesses will likely assist in discovering therapeutics to engage neural plasticity-related processes andpromote functional behavioral adaptations.
Operant learning is one of the most elementary forms of behav-oral adaptation (Rescorla, 1994). Through interchange with itsnvironment, an animal is able to learn about the consequences ofts actions, and thereby modify the current environment throughew behaviors to produce more favorable conditions (Skinner,953). The resultant change in behavior is dramatic and long-
asting. Some scholars have argued that operant learning is theasis of “knowledge” (Schnaitter, 1987), may underlie “creativity”Pryor et al., 1969), is the basis of decision-making, and con-ributes to the intractable nature of drug addiction. As the behaviorf an organism is altered by response-outcome contingencies,hysiological mechanisms are activated which ensure that theselterations become nearly permanent; they are “stamped in,” ashorndike hypothesized (Thorndike, 1911). Even Skinner intimatedhat response-outcome contingences change us: “Men act upon theorld, and change it, and are changed in turn by the consequences
f their action.” (Skinner, 1957, p. 1).In light of the ubiquity of operant behavioral relations in our psy-
hological lives, the neurobiology of operant learning (i.e., the initialcquisition of an operant response) has received surprisingly littlettention when compared to other basic learning processes suchs spatial learning (e.g., Morris Water Maze) or Pavlovian fear con-itioning. Yet, operant relations are thought to be at work nearlyvery moment of our lives and in many prominent neuropsychi-tric conditions: drug abuse, autism, and other severe problemehaviors. In this review, we highlight the last two decades ofnn Kelley’s research career, when she pursued a greater under-tanding of the neurobiology of operant learning with the hopehat the molecular, cellular, and genomic constituents of operantearning, instantiated in distributed networks, would inform betterreatment alternatives.
. Costly behavioral-health problems and Operant behavior
Drug abuse is one of the most damaging, recalcitrant and costlyehavioral-health problems in the U.S., and indeed, the world.buse of drugs in this country alone costs an estimated $484 bil-
ion annually in health-related problems, accidents, lost work, andnsurance premiums (Policy, 2001). It is also estimated that 540,000eople die each year from drug-related illnesses. These estimateso not include the non-monetary or indirect psychosocial costsaid by parents,1 spouses, siblings, friends, and our community
n general. It is quite likely that every citizen in this nation haseen adversely affected by drug abuse and addiction in some waye.g., as the victim of criminal behavior, an automobile accident, orhrough the actions of a family member). Drug addiction is beingncreasingly viewed in terms of fundamental changes in cognitionsnd behaviors, with emphasis on relating the compulsive nature ofddiction to pathological changes in decision- and emotion-codingetworks (Everitt et al., 2001). Thus, a better understanding of oper-nt learning systems may enhance our understanding of the neuralausation of addiction.
According to the Centers for Disease Control (CDC), 1 in 88hildren have been identified as having autism (Control, 2012).utism spectrum disorders (ASDs) affect individuals from all ethnicackgrounds and socioeconomic levels. ASDs can prove profoundlyebilitating and likely require life-long care at great expense tohe community (>$3,000,000 per individual) (Ganz, 2007). More
ecently, applied behavior analysis (ABA) and certain derivativese.g., Denver Start Model), which emphasize dynamic and flexiblecademic, social, and communicative behavior, have demonstrated
1 Consider the real, but difficult to estimate, cost of “sleepless nights” or increasedtress on the health and well-being of parents of children with drug behavior prob-ems.
ehavioral Reviews 37 (2013) 2071–2080
that incredible gains are possible with early, intensive therapy(Sallows and Graupner, 2005, Dawson et al., 2010, Warren et al.,2011). These models have been so successful that many childrendiagnosed with ASDs are later termed “indistinguishable” fromtheir peers. Some estimate that 40–50% of children diagnosed withautism are fully remediable (McEachin et al., 1993). In addition, theoverwhelming success of ABA therapy in the treatment of autismhas lead to the general idea that it is synonymous with autismtherapy (Dillenburger and Keenan, 2009), much to the displea-sure of practitioners, to name a few, of organizational behaviormanagement (OBM), clinical behavior analysis, and animal train-ing; professions that use behavior analysis applied to situationsnot involving autism. Of interest here is the fact that most ABAprinciples are based on contemporary operant theory and theexperimental analysis of behavior: evaluating possible establishingoperations, identifying the consequential functions of inappro-priate behavior, reinforcing good behavior, punishing unwantedbehavior, and assessing these relations in a greater socio-economiccontext (e.g., behavioral economics). In their seminal piece on ABA,Baer et al. (1968) lay out a clear relationship between operant the-ory and the “conceptual systems” dimension of ABA, although a fullreview of that paper is beyond the purview of this current review.Thus, because the etiology of ASDs are largely viewed as neuro-genetic, and in light of the prominent role operant behavior plays inlearning and therapy vis-à-vis ASDs, a greater understanding of theneurobiology of operant behavior might help our considerations ofASDs.
The term “severe problem behavior” encompasses a wide rangeof issues from school bullying to extreme self-injury. Severe prob-lem behaviors can be displayed by typically-developing children,but are more prevalent in children with developmental and/orintellectual disabilities. Severe problem behaviors create substan-tial social and educational obstacles for individuals due to theirintensity and seeming unpredictability. Treatment may involvesuspensions from school, placement in special environments,engaging the criminal justice system, incarceration or institution-alization. Rather than considering these patterns as “maladaptive”or “inappropriate,” psychologists and educators are now viewingmany of these problem behaviors as functional. In other words,when considered as operant behavior, the reinforcing contingen-cies promoting these severe behavior problems can be determined,assessed, and changed. Due to the dangerous nature of these prob-lems and the intrusion of likely neurophysiological issues, however,many individuals spiral into difficult or untenable living condi-tions or circumstances with a lack of treatment. The possibility thatthese serious problems emerge through a combination of genetic-environment interactions is only now being seriously considered.A better understanding of the neurobiology of operant behaviorwould improve treatment alternatives.
2. Mechanisms of neural plasticity in long-lastingbehavioral change
It is now well accepted that long-lasting behavioral modi-fications via operant contingencies are the result of significantchanges in the brain: the strengthening of synaptic connections,re-configuring of neural ensembles, synthesis of new proteins,upregulation of gene expression, and epigenetic modifications.Long-term potentiation (LTP) has served as one of the most fre-quently interrogated plasticity-related systems and data stronglyimplicate NMDAR activation as a key initiating event. That is,
high frequency patterns of synaptic stimulation activate NMDARresulting in an influx of Ca2+, in turn activating multiple signalingmechanisms, several of which converge on ERK (ExtracellularReceptor signaling Kinase). ERK is thought to regulate a variety of
d Biob
ttscf1iaitol
eapaat(olttluaod5iwaovat
fiwesolticreaiiNbot
oifpcan
M.E. Andrzejewski et al. / Neuroscience an
ranscription factors that coordinate the formation and stabiliza-ion of long-term memories (Levenson et al., 2004). There existsubstantial data confirming the role of the NMDAR-Ca2+-ERK cas-ade in long-lasting behavioral change and memory formation inear conditioning and Morris Water Maze learning (Atkins et al.,998, Blum et al., 1999, Schafe et al., 2000); a more recent report
mplicates this cascade in food-rewarded conditioning, as well,lthough in an invertebrate model (Ribeiro et al., 2005). NMDAR-nduced neural plasticity, through transcriptional regulations viahe ERK pathway, therefore, provides a neural representation ofperant conditioning and an elegant model for studying long-asting behavioral change.
In a direct extension of this model, Kelley and colleagues (Kelleyt al., 1997) first explored the role of NMDAR activation in oper-nt learning within the nucleus accumbens, a site hypothesized tolay a major role in the complex integration of sensory, rewardnd motor information. Following habituation to standard oper-nt conditioning chambers and magazine training, injections ofhe NMDAR antagonist (+/−)-2-amino-5-phosphonopentanoic acidAP-5) were made directly into the nucleus accumbens core (NAc)f food-restricted rats immediately prior to the first four, 15-minuteong, operant conditioning sessions. With a lever now inserted intohe chamber, presses were reinforced with sucrose pellets.2 Acrosshe first 4 training sessions, rats treated with AP-5 made very fewever-presses, in contrast to vehicle-treated rats. All rats were leftntreated for the next 5 sessions and both groups quickly reachedsymptotic levels of lever-pressing. Importantly, a microinjectionf AP-5 into the NAc prior to a 10th session did not have anyiscernible effects. Separate experiments found no effect of AP-
on spontaneous, unconditional eating and motor behavior indentically-treated (e.g., surgery, deprivation, etc.) rats. Therefore,
hen compared to saline-infusions, AP-5 infusions/NMDAR block-de in the NAC impaired initial operant learning, but had no effectn subsequent performance, nor did NMDAR blockade affect moti-ation for sucrose or spontaneous motor behavior. Thus, these datappear consistent with the general consensus that NMDAR activa-ion is crucial for learning via its role in neural plasticity.
These studies, conducted in Ann Kelley’s laboratory, are therst demonstrating a role for NMDA receptors in operant learningithin a key node of a cortico-limbic-striatal network. Hernandez
t al. (2005) directly replicated this effect, and, notably, demon-trated a time-limited contextual role for NMDAR activation inperant learning for post-session AP-5 infusions had no effect onearning. In other words, NMDAR activation during exposure tohe chamber and operant contingencies was required for learn-ng to occur but not necessary after the session. This findingontrasts with post-session drug effects on other behavioral prepa-ations, such as fear conditioning (Castellano et al., 1993). Kelleyt al. (1997) also showed that infusions of AP-5 into the nucleusccumbens shell (NAS) had very little effect on operant learn-ng, suggesting that operant conditioning entails plastic changesn a discrete network rather than ubiquitous neural action ofMDARs. A more precise characterization of this network couldenefit countless neuropsychiatric conditions that involve learning
r plasticity-related deficiencies by helping neurobiologists iden-ify discrete nuclei that are critical for carrying out behavior while
2 This first procedure employed two levers, with a VR-2 schedule programmedn one of them, counterbalanced across rats. The second, “incorrect” lever was orig-nally present to measure possible displacement or undiscriminated behavior. Weound it to be superfluous and complicated, rather than clarifying, subsequent inter-retation. Thus, we eliminated this second lever in later studies. In addition, wehanged the starting reinforcement schedule to an FR-1, while slowly migrating to
VR-2 during 5, instead of 4, initial sessions. These minor procedural changes doot appear to impact any of our findings given a number of replications.
ehavioral Reviews 37 (2013) 2071–2080 2073
simultaneously identify specific receptor mediation of said behav-ior.
To expand on these results, Baldwin et al. (2000) found that AP-5infusions in the basolateral amygdala (BLA) and the medial pre-frontal cortex (mPFC) also impaired operant learning, but AP-5 hadno effect on operant learning when infused in the dorsal (dSUB) orventral (vSUB) subiculum. Further, these effects were again limitedto the initial conditioning phase as NMDAR blockade had no effecton subsequent operant performance, spontaneous motor behavioror spontaneous feeding. McKee et al. (2010) extended the role ofNMDAR activation in operant learning to the dorsal medial striatum(DMS) and anterior cingulate cortex (ACC), but found no role forthe orbito-frontal cortex (OFC) in operant learning. Control studiesfound no evidence for motivational or motor deficits. Andrzejewskiet al. (2004) also explored the role of NMDARs in the central nucleusof the amygdala (CeA) and 2 other striatal subnuclei. While learningdeficits were observed after AP-5 infusions into the CeA and poste-rior lateral striatum (PLS), but not the dorso lateral striatum (DLS),there were also profound effects on spontaneous motor and feed-ing behavior with AP-5 infusions in the CeA and PLS. These resultssuggest that operant learning depends upon NMDAR activationwithin a distributed network, each possibly contributing distinctsensory, motivational, motor, and learning processing. Certainly,future studies are needed to evaluate the limits of the “operant”network.
Together, these initial studies indicate that the NAC, BLA, mPFC,DMS and ACC are critical areas in a cortico-limbic striatal net-work controlling operant learning that is not needed for laterperformance. Although further work may clarify this network andperhaps more specific roles of each region, such a network appearsto underlie the learning of addictive or maladaptive behaviors thatmay be more striatally regulated once established.
3. Dopamine involvement in reward processing andplasticity
Reinforcement-based processing also depends heavily on meso-corticolimbic DA systems, comprising DA neurons in the ventraltegmental area (VTA) and their projections to nucleus accum-bens (NAc), amygdala, prefrontal cortex (PFC), and other forebrainregions, but the exact nature of the role of DA in reward processingis still a source of contention. One early theory suggested thatDA-mediated the pleasures of reward because many natural anddrug rewards activate mesocorticolimbic systems and their block-ade impairs the behavioral effectiveness of most reinforcers (Wiseand Bozarth, 1985). A second hypothesis contends that mesocorti-colimbic DA neurons learn and predict reward deliveries, becausethey fire to appetitively-conditioned stimuli, but not to the uncon-ditional stimuli (or to the rewards themselves) (Schultz, 1998,2002). A third, very influential hypothesis, asserts that mesocor-ticolimbic DA systems encode incentive properties attributed tothe neural representations of stimuli and rewards. Indeed, DAdoes not mediate the hedonic influence of sweet rewards, but isrequired for behavior directed toward the same rewards (Berridgeand Robinson, 1998). Fourth, some have argued that mesocorti-colimbic DA systems subserve effort-related functions that impactreinforced-behavior due to the fact that DA depletions have lit-tle impact on operant responding when reinforced on an “easy”schedule (an FR-5, for example), but have dramatic effects on moreeffortful schedules (Salamone et al., 1994, Salamone et al., 2001).Nevertheless, while DA’s role in operant behavior is unequivocal,the exact nature and details of its role likely remain a function of
the preparation used and the theoretical orientation of the experi-menter.
We tested the role of DA on operant learning via D1R activity inmany of the same structures noted above. Baldwin et al. (2002b)
2 d Biobehavioral Reviews 37 (2013) 2071–2080
sbCisdabcsbatcei
4a
jouoisIpast2tSirawfip(tpdtle(dEu2sautib
d2(oPs
Fig. 1. Cumulative lever presses across sessions. The behavior of two representativerats, one vehicle-treated and one AP-5-treated, following infusions into the nucleus
074 M.E. Andrzejewski et al. / Neuroscience an
howed that D1R blockade in the PFC impaired operant learningut had no effect on performance. D1R blockade in the BLA andeA also impaired operant learning (Andrzejewski et al., 2005),
n a dose dependent fashion. However, the role of D1R in othertructures has been difficult to dissociate from other D1R-mediatedrug effects. For example, Hernandez et al. (2005) demonstrated
profound effect on operant behavior following pre-session D1Rlockade in the NAc; however, nose-poking into the food tray (oftenonsidered a Pavlovian appetitively conditioned response) was alsoubstantially reduced. Andrzejewski et al. (2006) found that D1Rlockade in vSUB, but not the dSUB, impaired operant learning, butgain, motivational deficits were discovered. While it appears likelyhat DA D1R activation is a crucial for directing the plasticity asso-iated with operant learning, the precise role remains somewhatlusive. Emerging evidence, however, led us to postulate a criticalnteractive role of NMDAR and D1R in operant learning.
. Intracellular convergence of NMDAR and DA D1Rctivation: coincidence detectors
From this evidence, we began to theorize that NMDARs in con-unction with DA D1Rs, and in particular the coincident detectionf incoming signals, play a critical role in shaping synaptic config-rations, and likely predominant neural ensembles, that underlieperant learning (Jay et al., 2004). NDMARs and DA D1Rs interactn dynamic ways. For example, NMDA-dependent LTP in striatallices is blocked by D1 but not D2 antagonists (Weiss et al., 2000).n vivo evidence for NMDA-D1 interaction in plasticity-relatedhenomena suggests that LTP takes place in multiple circuitriesnd structures. For example, LTP in hippocampal-prefrontal cortexynapses is dependent on co-activation of NMDA and D1 recep-ors, as well as intracellular cascades involving PKA (Jay et al.,004). In both striatum and prefrontal cortex, D1 activation poten-iates NMDA-receptor-mediated responses (Cepeda et al., 1993;eamans et al., 2001; Wang and O’Donnell, 2001). The potentiat-on of hippocampal-evoked spiking activity of accumbens neuronsequires cooperative action of both D1 and NMDA receptors, while
similar synergism is observed for the amygdalo-accumbens path-ay (Floresco et al., 2001b,a). Molecular studies complement thesendings, showing NMDA-receptor dependence of D1-mediatedhosphorylation of CREB (cAMP response element binding protein)Das et al., 1997; Carlezon and Konradi, 2004), a transcription fac-or thought to be an evolutionarily conserved modulator of memoryrocesses and key protein in cellular pathways affected by addictiverugs (Silva et al., 1998; Nestler, 2001). Strong support for the con-ention of coincident activation comes from the demonstration ofong-term enhancement of synaptic strength when corticostriatalxcitation and dopaminergic activation are temporally coordinatedWickens et al., 1996). Other data suggest that glutamate andopamine signals, via NMDA and D1 activation, converge to induceRK activation in the hippocampus and striatum, thereby reconfig-ring networks involved in learning and drug use (Valjent et al.,005, Kaphzan et al., 2006). Thus, given the requirements neces-ary for learning, it is intriguing to speculate that the coordinatedrrival of dopaminergic and glutamatergic signals, and its neuromolec-lar consequences, serve as the coincidence detector that initiatesranscriptional changes leading to enduring synaptic alterations. It ismportant to note that these very cascades are the ones proposed toe modified in the addictive process (Hyman and Malenka, 2001).
In a direct test of this hypothesis, Baldwin et al. (2002b) foundoses of AP-5 and R(+)-7-chloro-8-hydroxy-3-methyl-1-phenyl-,3,4,5-tetrahydro-1H-3-benzazepine hydrochloride (SCH-23390)
a D1R antagonist) in the PFC that had no discernible effect onperant learning. However, when combined and infused into theFC of naïve rats, operant learning was significantly impaired,uggesting strong synergy between the two receptors. That is,
accumbens core (NAc) prior to the first 5, 15-minute long sessions. Infusions ceasedafter 75 min (5 × 15 min). Best-fit functions with variance accounted for measuresare also plotted.
plasticity associated with operant behavior is possible with a smallamount of NMDAR or D1R blockade, but not both. Although wehave seen some dose-dependent effects, we wondered if operantlearning was an “all or nothing” phenomena, like concept learning(Osler and Trautman, 1961). In our experience, it appeared, thatour rats first spent their time in the chamber exploring, nose-poking, sniffing, grooming, rearing, etc., while only occasionallylever-pressing. After a couple of sessions, control rats “got it”and proceeded to lever press much more frequently, and reared,explored, sniffed, groomed, etc., less (e.g., responses for whichthere were no programmed consequences), just as Staddon andSimmelhag (1971) demonstrated in their seminal experiment onsuperstitious behavior. Therefore, initial operant learning mayengage a “tipping point” or threshold-like process, in contrastto a more gradual and smoothly changing one. Fig. 1 shows thecumulative responses of two rats with cannulae targeting the NAc.One was infused with vehicle prior to the first five sessions whilethe second was infused with AP-5. The similarity in functionsis striking and seems to conform to our notion: there is a verygradual and slow increase in responding, transitioning, relativelyquickly, to a high, and steady, rate of responding. Note that theAP-5-treated rat is delayed in this transition, suggesting that this“tipping point” is delayed by NMDAR blockade.
While these behavioral data and other observations may presenta convincing argument regarding this “tipping point” hypothesis, itwould be of great import if the neurobiology followed suit, for thiswould imply a “critical period” for operant learning and suggesttargets for intervention in a time-dependent fashion. At the veryleast, it appears that operant learning is highly contextualized vis-a-vis temporal, environmental and neurophysiological relations.
5. An intracellular signaling model of operant learning
The intracellular molecular constituents of learning (in general,not necessarily operant learning), as noted earlier, have receiveda great amount of interest. Our own findings regarding the roleof NMDAR activation were thoroughly informed by these find-
ings regarding LTP. However, the intracellular signaling cascadesresponsible for LTP are now well-elucidated. Are they the same cas-cades responsible for reconfiguring the synaptic pathways duringoperant learning? Baldwin et al. (2002a) inhibited protein kinase
M.E. Andrzejewski et al. / Neuroscience and Biobehavioral Reviews 37 (2013) 2071–2080 2075
Fig. 2. Role of ERK in operant learning. Panel A shows that U0126 infused into the NAc prior to learning sessions has no effect when compared to vehicle-infused controls.P g an or e extl
aLcdt3dwaTa
l2colfiapetEe6yhfidoaPAawsbbbw
anel B shows neither ERK-1 nor ERK-2 phosphorylation is increased in rats learnineinforcers. Panel C shows sparse pERK labeling in the NAc (left panel) relative to thearning sessions.
ctivity, crucial constituents of intracellular signaling necessary forTP, in the NAc of rats prior to operant learning sessions with theompound 1-(5-isoquinolinesulfonyl)-2-methylpiperazine dihy-rochloride (H-7). In a separate group of rats, cAMP-dependent pro-ein kinase (PKA) activity was inhibited by the drug Rp-adenosine′,5′-cyclic monophosphothioate triethlyamine (Rp-cAMPS) imme-iately prior to operant learning sessions. In both cases, learningas impaired suggesting that protein kinase signaling generally,
nd PKA activity specifically, were necessary for operant learning.hus, several key intracellular components of the neural plasticityssociated with operant learning have been identified.
PKA, PKC and other protein kinase activities converge intracellu-arly, according to several prominent models, at ERK (Valjent et al.,005, Kaphzan et al., 2006). Phosphorylated ERK (pERK) translo-ates to the nucleus of neurons, where it is modulates the activityf CREB, widely held as an evolutionarily conserved mediator ofong-term neural plasticity. Surprisingly, we have found little roleor ERK in operant learning. First, U0126 (a pERK inhibitor) infusednto the NAc prior to operant learning sessions produced no observ-ble effect (Fig. 2, panel A). We used the identical paradigms andreparations as with previous reports, however, given our lack ofxperience with this drug, it is possible that this negative effect washe result of an unknown technical problem. Second, we exploredRK phosphorylation after operant learning using standard West-rn blots and commercially available antibodies. Two groups of
rats were run: 1) standard operant training (FR-1/VR-2) and 2)oked control (received the same number of reinforcers but did notave to lever-press to produce them). Brains were collected withinve minutes of the 5th session and processed by Western blot. Noifferences in ERK, pERK or the pERK/ERK ratio were noted in anyf 12 areas studied, including the NAc (Fig. 2, panel B). There was
slight, but statistically significant, effect in pERK in the vSUB andFC, constituting roughly a 20% increase relative to yoked controls.lthough the effect was statistically significant, it was very modestnd quite possibly a Type 1 error given the number of comparisonse conducted. Third, we attempted to visualize, and hopefully,
emi-quantify pERK throughout the brain after operant learning
y using standard immunohistochemical methods on free-floatingrain sections. These rats were treated identically to the Westernlot experiments, however following brain collection, whole brainsere sliced and pERK antibodies were used to localize pERK (Fig. 3).
perant response when compared to yoked controls, receiving the same numbers ofensive cytoplasmic and dendritic labeling in the vSUB (right panel) during operant
Once again, while there was significant pERK staining in thePFC and vSUB, there was very little in the NAc (Fig. 2, panel C).These data conform closely with the results of Westerns and sug-gest a limited role for ERK in operant learning, in contrast to themyriad of studies demonstrating a crucial role for this kinase inother forms of learning (Levenson et al., 2004; Chwang et al., 2006;Kaphzan et al., 2006). However, coincident NMDAR/D1R activationcan recruit ERK-independent signaling routes to the nucleus.
6. CREB’s role in neural plasticity
pERK’s modulation of pCREB is critical during learning becauseCREB is a transcription factor increasing or silencing the expressionof certain genes. These genes are thought to be the regulators ofthe synthesis of particular proteins that form the building blocks ofreceptors, membranes, and other structures crucial to neural plas-ticity. Indeed, we have demonstrated that protein synthesis in theNAc is critical during operant learning (Hernandez et al., 2002).Using the protein synthesis inhibitor, anisomycin, we showed thatimmediate post-session infusions into the NAc blocked subsequentoperant learning, implicating transcription factors and de novo pro-tein synthesis. Interestingly, infusions 2 or 4 h after the session hadno effect; anisomycin also had no effect during a performance testor a feeding test. Once again, it appears that we have uncoveredkey features of a tightly controlled, temporally and contextually,learning system involving multiple structures, receptors, signalingmechanisms, and now, protein synthesis.
The finding of protein synthesis dependency of operant learningwas arguably one of the more important in our laboratory, yet itposed a large open-ended question regarding the specificity of thisprotein synthesis. We therefore conducted several experimentsto identify which genes may be synthesized/upregulated duringoperant learning. Using standard in situ hybridization methodswith rats treated much like the ones used for the pERK Westernstudies, we found that the immediate early genes (IEGs) Homer1aand egr1 (zif-268) were upregulated, compared to control rats,immediately after the 3rd operant training session within dis-
crete cortico-limbic-striatal nodes. Gene expression was elevatedwidely throughout the cortex and striatum, and in some cases, thehippocampus, but surprisingly, not in the ventral striatum (i.e.,NAc). In contrast to the “early learning group”, a second group of
2 d Biob
reginwgetaoeV
7a
mmppdta(ae
eaamiasro
8
DlgiD
Ftp
076 M.E. Andrzejewski et al. / Neuroscience an
ats experienced 23 operant learning sessions. Yet Homer1a andgr1 expression was now reduced compared to the early learningroup, in nearly all nuclei studied, suggesting that these genes arenvolved in plasticity-related functions during early exposure, butot later exposure, to operant contingencies. The single exceptionas the ventrolateral striatum (VLS), which appears to remain,
enetically speaking, “on line” even during extended operantxposure. Even though many scholars have termed long operantraining as “habit formation” these responses remain adaptablend flexible (consider the “temporary” effect of reinforcementr the reduction one would see when operant contingencies areliminated or extinguished): it is interesting to speculate that theLS may subserve this monitoring function.
. Other glutamate receptors also assist in plasticityssociated with operant learning
Homer1a is thought to regulate and traffic group 1etabotrophic glutamate receptors (mGluR1 and mGluR5).GluR5s potentiate the activity of NMDARs by altering their
ermeability to Ca2+ (Pisani et al., 2001), raising the interestingossibility that one mechanism of NMDAR-induced plasticity mayepend heavily on mGluR5 activity. Recently, we directly testedhe role of mGluR5 activity on operant learning by blocking theirctivity with the drug 3-((2-methyl-4-thiazolyl)ethynyl)pyridineMTEP). Our preliminary results suggest that blockade of mGluR5ctivity in the DMS impairs operant learning, although follow upxperiments on this finding are ongoing.
AMPA receptor activation and operant learning has also beenxplored in our laboratory. Hernandez et al. (2002) demonstrated
time-limited role for AMPAR activation in the NAc during oper-nt learning. The effect, however, endured for many sessions anday have been the result of some down-regulation or long-term
nternalization of glutamate receptors. While this contention needsdditional empirical support, we found it very surprising that pre-ession blockade of AMPAR would produce such a long-term effectelative to post-session blockade, which produced no change inperant learning.
. Epigenetic changes during operant learning
In addition to activating transcription factors, NMDAR and1R activity also induces modifications, such as histone acety-
ation, to chromatin, the protein that organizes and condensesenomic DNA. These modifications provide recruitment signalsnvolved in gene transcription/silencing and influence access toNA by the transcriptional machinery. NMDAR activation and
ig. 3. Acetylated histone H3 density during operant performance is elevated inhe DMS relative to yoked controls, but not in the NAc, PFC, or ACC. Representativeictomicrographs of stained DMS sections in shown on the right.
ehavioral Reviews 37 (2013) 2071–2080
associated intracellular signaling cascades, including histone 3(H3) acetylation, govern long-lasting behavioral change, Pavlovianfear conditioning and instrumental Morris Water Maze learning(Atkins et al., 1998; Blum et al., 1999; Schafe et al., 2000). Werecently began to explore whether operant learning modifieschromatin. Indeed, Histone H3 acetylation expression increasedin certain structures during performance of an operant behavior,versus sucrose feed controls. In this experiment, rats lever pressingon an RI-30′′ schedule were sacrificed 30 min after a session. Brainswere collected, processed and incubated with anti-acetyl-HistoneH3 (Lysine 14) using standard protocols.
Interestingly, relative to yoked controls, we saw elevated his-tone H3 acetylation in the DMS, a structure widely considered asa key contributor to operant learning. These are some of the firstdata that we know of showing histone modifications during oper-ant learning. However, increases in the global level of histone H3acetylation could be a result of modifications at promoters of genesother than IEGs and, further, the rats used in this experiment hadextensive training. Thus, additional information on the locus ofthat acetylation during operant learning is necessary. Nevertheless,these data, in conjunction with many other reports, strongly sug-gest that epigenetic processes are engaged during operant learning.Long-lasting modifications, like histone acetylation, may help usunderstand the enduring nature of operant behavior, its resistanceto change, and the recalcitrance of certain disorders to treatment.
Epigenetic processes also appear to be modified during drugadministration and learning. During cocaine self-administration, aD1R-dependent instrumental paradigm, chromatin modificationsare induced in certain regions of the striatum at the promotersof many plasticity-related genes, such as Cbp, NR2B, Psd95, andGluR2. Cbp is critical for stimulation-induced activation of CREBand has intrinsic histone acetyltranferase (HAT) activity (Shaywitzand Greenberg, 1999). Transgenic mice expressing a truncated formof Cbp have several learning deficits (Wood et al., 2005). NR2B, asubunit of the NMDAR complex, contains the glutamate bindingsite and is essential for LTP, while the subunit NR2A is not (Fosteret al., 2010a,b). The NR2B subunit is phosphorylated by CaMKII,dephosphorylated by PP1, and mediates NMDAR internalization(Roche et al., 2001). Psd-95 inhibits NR2B-mediated internalizationof NMDAR (Roche et al., 2001) and governs synaptic localization andstabilization of NMDARs (Li et al., 2003). GluR2 is a subunit of theAMPAR and contains a crucial phosphorylation site also modulatedby intracellular protein kinase and protein phosphatase activity.Phosphorylation of GluR2 partially governs AMPARs permeabilityto calcium and other cations. Interestingly, mGluR5 stimulation inthe rat dorsal striatum induces GluR2 phosphorylation, an effectblocked by NMDAR antagonism (Ahn and Choe, 2010).
9. An Intra-cellular convergence model of operant learning
Against this backdrop of dynamic and interesting work, wecreated a model of NMDAR-DA D1R convergence that may pro-mote of greater understanding of the neural plasticity involved inoperant learning. Fig. 4 illustrates the prevailing hypothesis thatglutamate-coded sensory/information processing signals activateNMDAR, and AMPAR, leading to Ca2+ influx into the cell. DA activa-tion of D1Rs activates adenyl cyclase (AC, designated with a blackarrow), and in turn, cAMP. The two signaling pathways interact inseveral places, for example, as CaM, induced by NMDAR activation,influences AC (although this is a somewhat oversimplified repre-sentation). PKA activates MEK, but also inhibits Ras/Raf (designatedwith a bar-headed line), suggesting that not only do the pathways
converge, but also may compete for signal dominance.
Several points of possible convergence are demonstrated, mostnotably the activation of CREB, MEK and ERK. Critical plasticity-related effects are also demonstrated, like the CREB-dependent
M.E. Andrzejewski et al. / Neuroscience and Biob
Fig. 4. An intracellular signaling model of operant leatning. The functional andstructural changes involved in neural plasticity implicates coordinated NMDARand DA D1R activation throughout cortical-striatal-limbic networks. This figuresummarizes the prevailing models of convergence and divergence of intracellu-lar signals, following NMDAR and DA D1R activation, leading to activation and/orphorphorylation of key enzymes, inhibition of particular signals, transcription ofcritical immediate early genes, and possible chromatin modifications. L-type VGCC,L-type voltage gated calcium channel; AMPA, �-amino-3-hydroxy-5-methyl-4-isoxazolepropionic acid; PLC, phospholipase C; AC, adenylate cyclase; G�, Guaninenucleotide binding protein (G-protein) alpha subunit; G�q, G-protein alpha q sub-unit; CaM, calmodulin; CaMKII, calmodulin kinase II; CaMKIV, calmodulin kinaseIV; MEK, Mitogen-activated protein kinase kinase; DARPP-32, dopamine- and cAMP-regulated neuronal phosphoprotein; Ser133, Serine 133; Elk, e twenty-six (ETS)-liketH
tfispytseAclwsaair
and findings on other clinical problems. Those implications fall into
ranscription of IEGs Arc, Homer1a, and egr1. Homer1a traf-cks mGluR5 receptors (represented by a gray arrow), whichubsequently potentiate Ca2+ influx via G�q-protein coupled phos-holipase C (PLC) activity (this potentiation is represented with aellow arrow and lightening bolts); mGluR5 activity also poten-iates DA D1R activation. Arc is transported to recently-activatedynapses, likely performing a sort of “tagging” role. Recently,merging data suggest an important role for Arc and ERK inMPAR-subunit insertion and regulation of L-type voltage gatedalcium channels. DARPP-32, activated by PKA activity, accumu-ates in the nucleus, inhibiting protein phosphatase 1 (PP1) activity,
hich is directly involved in chromatin modifications via intrin-ic dephosphorylation activity (symbolized by a half-circle-headedrrow “grasping” a phosphate group). Histone deactylease (HDACs)
ctions are represented with an inverted-arrow headed line “grasp-ng” acetyl groups from Histone 3 (H3). These histone modificationselax or compact chromatin thereby enabling or suppressing gene
ehavioral Reviews 37 (2013) 2071–2080 2077
transcription (the particular modifications denoted in the Figure donot necessarily represent the actual modifications required at thepromoters of the IEGs for transcription) (Fig. 4 is based on (Sweatt,2001, Kelley and Berridge, 2002, Haberny and Carr, 2005, Ostlundand Balleine, 2005, Valjent et al., 2005). Therefore, the neuro-molecular convergence of information from cortico-striatal-limbicNMDAR and DA D1R provides a possible substrate for plasticity inreward-based learning. The specific brain nuclei and neurons rep-resented in this model are only now coming into focus, but likelyinvolve key striatal, limbic, and cortical sites. Our strong suspi-cion is that medium spiny neurons, in the striatum especially, maybe well-suited for plasticity-related functions due to their unusu-ally high density of voltage-dependent ion channels that produceexceptional state-transitions (Houk and Wise, 1995) in combina-tion with the convergence of widespread, glutamate-coded cortical,limbic, and thalamic afferents, as well as monoaminergic inputsfrom midbrain.
Kelley et al. (1997) initially pronounced a crucial role for the NAcin neural plasticity and operant learning. Indeed, our laboratory hasexplored the role of nucleus accumbens in a variety of behavioralparadigms using an expertly-arranged multi-disciplinary approach(e.g., the experimental analysis of behavior, behavioral neuro-science, molecular and cellular neuroscience, etc.). Dr. Kelley wasone of the experts on the structure, physiology, connectivity andfunction of nucleus accumbens. However, several of our own exper-iments appear to contradict Dr. Kelley’s initial pronouncement. Theconvincing lack of MEK/ERK involvement in the NAc during oper-ant learning and the lack of gene expression serve as two boldexceptions to the contention that plasticity in the NAc is crucialfor operant learning. First, it may be that the MEK/ERK is not beinvolved in operant learning anywhere in the brain. Our studies of12 other sites yielded very little difference between operant learn-ing and yoked controls. Perhaps, the MEK/ERK pathway is involvedduring the “critical period” or “tipping point” when rats seem to“get it” and our studies did not have the temporal resolution todetect this effect, particularly as ERK activation is a dynamic andrelatively rapid event. Perhaps our doses of U0126 were too low toinhibit ERK activation. However, an equally likely hypothesis is thatCREB-mediated transcription of genes involved in neural plasticityare activated directly by other signaling pathways, such as PKAc orCAM (see Fig. 4), bypassing the MEK/ERK pathway. And perhaps,we have not identified the critical plasticity-related genes or themyriad of possible epigenetic modifications to NAc neurons thatenable and instantiate operant behavior. We hope to engage thesequestions with the same rigor and enthusiasm that Ann did.
10. Clinical implications
The prevailing hypothesis of this review is that the model pre-sented in Fig. 4 can inform treatment of many clinical problems.Of obvious relevance is drug addiction, for drug abuse profoundlyaffects many of the same molecular processes engaged by operantlearning. In recent years, some of the most remarkable findings inresearch on addiction are those demonstrating significant overlapof the mechanisms mediating drug addiction and normal reward-related learning (Hyman and Malenka, 2001; Nestler, 2001; Wanget al., 2010). We are certain that many of the reviews in this spe-cial edition have elegantly highlighted the relationship betweendrug addiction and normal reward-related learning. Admittedly,this relationship has proved to be crucial in our understanding ofaddiction, however, we would like to cite some important new linksbetween Dr. Kelley’s work on operant learning with emerging data
two general themes: (1) clinical problems with associated learn-ing impairments that could be served by a better understanding ofhow operant learning proceeds via neuromolecular mechanisms of
2 d Biob
plrob
nsacbsetovcsaLti“nosatus
Esigawgw2gNiuiw
btbonsdfTyaaa
dgmeo
078 M.E. Andrzejewski et al. / Neuroscience an
lasticity and (2) clinical problems associated with ongoing, alreadyearned, and possibly very resistant, operant behavior and its neu-omolecular constituents. This latter case subsumes the problemf addiction, we think, as it is properly viewed as ongoing operantehavior with very damaging and long-lasting side effects.
As noted in the introduction, autism spectrum disorders areow thought to affect 1 in 88 children. Communication deficits,ocial interaction problems and stereotypic behavior patterns char-cterize autism, although communication skills can be typical inhildren with Asperger’s. Early intensive behavior therapy (EIBT),ased on operant principles, forms the backbone of comprehen-ive treatment regimens that are yielding incredible results. Thisarly therapy, which is highly individualized and contextualized,ypically involves at least 40 h of one-on-one therapy per week,ften for many years. Data indicate that the earlier the inter-ention begins, the better the success rate. In many of theseases (some estimates are between 40 and 50%), complete main-treaming into regular classrooms is possible with minimal or nodditional supports (Lovaas, 1987; Sallows and Graupner, 2005;eBlanc and Fagiolini, 2011). These findings intimate neural plas-icity as a driving component in the success of EIBT. Researchersn the autism treatment community are widely speculating aboutcritical periods” of development which coincides with heightenedeural plasticity (LeBlanc and Fagiolini, 2011). Thus, our research onperant learning may have two possible implications: 1) it is pos-ible that the autistic “brain” may have reduced plastic potential,nd only through intensive practice and therapy are these reduc-ions overcome and 2) it may be possible, with a more completenderstanding of operant learning, to induce periods of plasticityo older children could benefit from therapy.
While it is a highly speculative contention that operant learning,IBT, and neural plasticity share underlie ASDs, there are severalources of converging supportive evidence. To begin, the lead-ng heritable cause of ASDs is Fragile X syndrome (FXS), a singleene trinucleotide repeat problem with the FMR1 gene. FXS isssociated with learning impairments, social behavioral deficits asell as some physical (primarily facial) abnormalities. The FMR1
ene encodes the Fragile X mental retardation protein (FMRP),hich is required for normal neural development (Crawford et al.,
001; Antar et al., 2004). In addition, FMRP strongly modulatesroup 1 mGluR activity, and lack of FMRP activity dysregulatesMDAR LTP (Antar et al., 2004). Our recent work with the mGluR5
nhibitor MTEP suggests a role in operant learning for this receptornder “normal” conditions. Pharmacotherapies based on modulat-
ng mGluR5 activity are now being investigated for use in humansith FXS (Hagerman et al., 2012).
Another form of autism, referred to as “regressive autism”ecause children with this form develop typically for a period andhen lose “normal” communication and social skills, has recentlyeen linked to decreased activity of PKA and the catalytic subunitf PKA, namely the c-isoform. When compared post-mortem toon-regressive autistic controls, regressive autism frontal corticeshowed decreased activity and expression of PKA (Ji et al., 2011). Noifferences were noted in other cortical regions, nor was there a dif-erence between non-regressive autism and non-autistic controls.hus, regressive autism may be linked to PKA-mediated phosphor-lation of proteins and anomalous intracellular signaling. Oncegain, our work has demonstrated a crucial role for PKA in oper-nt learning, converging nicely with this recent work on regressiveutism.
Rubenstein-Taybi syndrome (RTS) is an autosomal dominantisorder caused by mutations of the CREB binding protein (CREBBP)
ene. Short stature, broad thumbs, distinctive facial features, andoderate to severe learning difficulties characterize RTS (Bartsch
t al., 2010). Of critical import here is the obvious link betweenperant learning, CREB function, and RTS. Perhaps children with
ehavioral Reviews 37 (2013) 2071–2080
RTS could benefit from EIBT or some pharmacological therapy thatenables, supplements, or supplants CREB modulation of gene tran-scription. CREB phosphorylation appears to control IEG functionand the synthesis of new proteins, and likely regulates neural plas-ticity associated with operant learning.
Lastly, our data and intracellular model implicate epigeneticprocesses as responsible for the enduring nature of operant behav-ior. Our common consideration of operant behavior as “habitformation”, repeated demonstrations of spontaneous recovery,and the seemingly unlimited recall period associated with oper-ant repertoires contribute strongly to this idea. Indeed, manysevere problem behaviors have proven exceedingly recalcitrant totreatment, thus leading to restricted social opportunities, chem-ical restraint, hospitalization and institutionalization. However, abroad class of diagnostic tools, often referred to as the “functionalanalysis of problem behavior” or “functional behavior assessment(FBA)”, have been developed to identify the controlling relationsfor these severe behaviors. Generally, these behavior classes areviewed as operant, reinforced by attention, access to preferreditems/activities, or escape/avoidance of unwanted circumstances(Lerman and Iwata, 1993). With this information in hand, ther-apy can be directed in such a way as to provide alternative sourcesof reinforcement or alternative appropriate operants that producethose wanted circumstances, potentially even long after the origi-nal operant learning of the inappropriate behavior. Is it possible thata greater understanding of operant learning could provide phar-macotherapeutic targets, like histone acetylation, that enhanceoperant extinction and/or promote new operant learning?
While many of these notions are highly speculative, the workof Dr. Ann Kelley and colleagues in the area of operant learning islikely to inform, at the very least, the nature and course of drugaddiction. We would also like to extend our theory and findingsto help understand the learning deficits associated with ASDs, FXSand RTS, as well as the difficultly associated with the strength ofcertain severe problematic operant repertoires.
References
Ahn, S.M., Choe, E.S., 2010. Alterations in GluR2 AMPA receptor phosphor-ylation at serine 880 following group I metabotropic glutamate receptorstimulation in the rat dorsal striatum. J. Neurosci. Res. 88 (5), 992–999,http://dx.doi.org/10.1002/jnr.22275.
Andrzejewski, M.E., Sadeghian, K., Kelley, A., 2004. Central amygdalar and dorsalstriatal NMDA-receptor involvement in instrumental learning and spontaneousbehavior. Behav. Neurosci., 118.
Andrzejewski, M.E., Spencer, R.C., Kelley, A.E., 2005. Instrumental learning, butnot performance, requires dopamine D1-receptor activation in the amygdala.Neuroscience 135, 335–345.
Andrzejewski, M.E., Spencer, R.C., Kelley, A.E., 2006. Dissociating ventral and dor-sal subicular dopamine D-sub-1 receptor involvement in instrumental learning,spontaneous motor behavior, and motivation. Behav. Neurosci. 120, 542–553.
Antar, L.N., Afroz, R., Dictenberg, J.B., Carroll, R.C., Bassell, G.J., 2004. Metabotropicglutamate receptor activation regulates fragile x mental retardation protein andFMR1 mRNA localization differentially in dendrites and at synapses. J. Neurosci.24, 2648–2655.
Baldwin, A.E., Holahan, M.R., Sadeghian, K., Kelley, A.E., 2000. N-methyl-D-aspartatereceptor-dependent plasticity within a distributed corticostriatal networkmediates appetitive instrumental learning. Behav Neurosci 114 (1), 84–98.
Baldwin, A.E., Sadeghian, K., Holahan, M.R., Kelley, A.E., 2002a. Appetitive instru-mental learning is impaired by inhibition of cAMP-dependent protein kinasewithin the nucleus accumbens. Neurobiol. Learn. Mem. 77, 44–62.
Baldwin, A.E., Sadeghian, K., Kelley, A.E., 2002b. Appetitive instrumental learningrequires coincident activation of NMDA and dopamine D1 receptors within themedial prefrontal cortex. J. Neurosci. 22, 1063–1071.
Bartsch, O., Kress, W., Kempf, O., Lechno, S., Haaf, T., Zechner, U., 2010. Inheritanceand variable expression in Rubinstein-Taybi syndrome. Am. J. Med. Genet. PartA 152A, 2254–2261.
Baer, D.M., Wolf, M.M., Risley, T.R., 1968. Some current dimensions of appliedbehavior analysis. J Appl Behav Anal 1 (1), 91–97.
erridge, K.C., Robinson, T.E., 1998. What is the role of dopamine in reward: hedonicimpact, reward learning, or incentive salience? Brain Res. Brain Res. Rev. 28,309–369.
lum, S., Moore, A.N., Adams, F., Dash, P.K., 1999. A mitogen-activated protein kinasecascade in the CA1/CA2 subfield of the dorsal hippocampus is essential for long-term spatial memory. J. Neurosci. 19, 3535–3544.
arlezon Jr., W.A., Konradi, C., 2004. Understanding the neurobiological con-sequences of early exposure to psychotropic drugs: linking behavior withmolecules. Neuropharmacology 47 (Suppl. 1), 47–60.
astellano, C., Introini-Collison, I.B., McGaugh, J.L., 1993. Interaction of beta-endorphin and GABAergic drugs in the regulation of memory storage. Behav.Neural Biol. 60, 123–128.
epeda, C., Buchwald, N.A., Levine, M.S., 1993. Neuromodulatory actions ofdopamine in the neostriatum are dependent upon the excitatory amino acidreceptor subtypes activated. Proc. Natl. Acad. Sci. U.S.A. 90, 9576–9580.
hwang, W.B., O‘Riordan, K.J., Levenson, J.M., Sweatt, J.D., 2006. ERK/MAPK regulateshippocampal histone phosphorylation following contextual fear conditioning.Learn. Mem. 13, 322–328.
ontrol, C.f.D., 2012. Autism Spectrum Disorders. Centers for Disease Control.rawford, D.C., Acuna, J.M., Sherman, S.L., 2001. FMR1 and the fragile X syndrome:
human genome epidemiology review. Genet. Med. 3, 359–371.as, S., Grunert, M., Williams, L., Vincent, S.R., 1997. NMDA and D1 receptors regulate
the phosphorylation of CREB and the induction of c-fos in striatal neurons inprimary culture. Synapse 25, 227–233.
awson, G., Rogers, S., Munson, J., Smith, M., Winter, J., Greenson, J., Donaldson, A.,Varley, J., 2010. Randomized, controlled trial of an intervention for toddlers withautism: the Early Start Denver Model. Pediatrics 125, e17–e23.
illenburger, K., Keenan, M., 2009. None of the As in ABA stand for autism: dispellingthe myths. J. Intellect. Dev. Disabil. 34, 193–195.
veritt, B.J., Dickinson, A., Robbins, T.W., 2001. The neuropsychological basis ofaddictive behaviour. Brain Res. Brain Res. Rev. 36, 129–138.
loresco, S.B., Blaha, C.D., Yang, C.R., Phillips, A.G., 2001a. Dopamine D1 and NMDAreceptors mediate potentiation of basolateral amygdala-evoked firing of nucleusaccumbens neurons. J. Neurosci. 21, 6370–6376.
loresco, S.B., Blaha, C.D., Yang, C.R., Phillips AG, 2001b. Modulation of hippocampaland amygdalar-evoked activity of nucleus accumbens neurons by dopamine:cellular mechanisms of input selection. J. Neurosci. 21, 2851–2860.
oster, K.A., McLaughlin, N., Edbauer, D., Phillips, M., Bolton, A., Constantine-Paton,M., Sheng, M., 2010a. Distinct roles of NR2A and NR2B cytoplasmic tails inlong-term potentiation. J. Neurosci. 30, 2676–2685.
oster, K.A., McLaughlin, N., Edbauer, D., Phillips, M., Bolton, A., Constantine-Paton,M., Sheng, M., 2010b. Distinct roles of NR2A and NR2B cytoplasmic tails inlong-term potentiation. J. Neurosci. 30, 2676–2685.
anz, M.L., 2007. The lifetime distribution of the incremental societal costs of autism.Arch. Pediatr. Adolesc. Med. 161, 343–349.
aberny, S.L., Carr, K.D., 2005. Food restriction increases NMDA receptor-mediatedcalcium-calmodulin kinase II and NMDA receptor/extracellular signal-regulatedkinase 1/2-mediated cyclic amp response element-binding protein phosphory-lation in nucleus accumbens upon D-1 dopamine receptor stimulation in rats.Neuroscience 132, 1035–1043.
agerman, R., Lauterborn, J., Au, J., Berry-Kravis, E., 2012. Fragile X syndrome andtargeted treatment trials. Results Probl. Cell Differ. 54, 297–335.
ernandez, P.J., Andrzejewski, M.E., Sadeghian, K., Panksepp, J.B., Kelley, A.E., 2005.AMPA/kainate, NMDA, and dopamine D1 receptor function in the nucleus
accumbens core: a context-limited role in the encoding and consolidation ofinstrumental memory. Learn. Mem. 12, 285–295.
ernandez, P.J., Sadeghian, K., Kelley, A.E., 2002. Early consolidation of instrumentallearning requires protein synthesis in the nucleus accumbens. Nat. Neurosci. 5,1327–1331.
ouk, J.C., Wise, S.P., 1995. Distributed modular architectures linking basal ganglia,cerebellum, and cerebral cortex: their role in planning and controlling action.Cereb. Cortex 5, 95–110.
yman, S.E., Malenka, R.C., 2001. Addiction and the brain: the neurobiology ofcompulsion and its persistence. Nat. Rev. Neurosci. 2, 695–703.
ay, T.M., Rocher, C., Hotte, M., Naudon, L., Gurden, H., Spedding, M., 2004. Plasticityat hippocampal to prefrontal cortex synapses is impaired by loss of dopamineand stress: importance for psychiatric diseases. Neurotox. Res. 6, 233–244.
i, L., Chauhan, V., Flory, M.J., Chauhan, A., 2011. Brain region-specific decrease inthe activity and expression of protein kinase A in the frontal cortex of regressiveautism. PLoS ONE 6, e23751.
aphzan, H., O‘Riordan, K.J., Mangan, K.P., Levenson, J.M., Rosenblum, K., 2006.NMDA and dopamine converge on the NMDA-receptor to induce ERK activationand synaptic depression in mature hippocampus. PLoS ONE 1, e138.
elley, A.E., Berridge, K.C., 2002. The neuroscience of natural rewards: relevance toaddictive drugs. J. Neurosci. 22, 3306–3311.
elley, A.E., Smith-Roe, S.L., Holahan, M.R., 1997. Response-reinforcement learn-ing is dependent on N-methyl-d-aspartate receptor activation in the nucleusaccumbens core. Proc. Natl. Acad. Sci. U.S.A. 94, 12174–12179.
eBlanc, J.J., Fagiolini, M., 2011. Autism: a “critical period” disorder? Neural Plas.,921680.
erman, D.C., Iwata, B.A., 1993. Descriptive and experimental analyses of variables
maintaining self-injurious behavior. J. Appl. Behav. Anal. 26, 293–319.
evenson, J.M., O‘Riordan, K.J., Brown, K.D., Trinh, M.A., Molfese, D.L., Sweatt, J.D.,2004. Regulation of histone acetylation during memory formation in the hip-pocampus. J. Biol. Chem. 279, 40545–40559.
ehavioral Reviews 37 (2013) 2071–2080 2079
Li, B., Otsu, Y., Murphy, T.H., Raymond, L.A., 2003. Developmental decrease in NMDAreceptor desensitization associated with shift to synapse and interaction withpostsynaptic density-95. J. Neurosci. 23, 11244–11254.
Lovaas, O.I., 1987. Behavioral treatment and normal educational and intellectualfunctioning in young autistic children. J. Consult. Clin. Psychol. 55, 3–9.
McEachin, J.J., Smith, T., Lovaas, O.I., 1993. Long-term outcome for children withautism who received early intensive behavioral treatment. Am. J. Ment. Retard.97, 359–372, discussion 373-391.
McKee, B.L., Kelley, A.E., Moser, H.R., Andrzejewski, M.E., 2010. Operant learningrequires NMDA-receptor activation in the anterior cingulate cortex and dor-somedial striatum, but not in the orbitofrontal cortex. Behav. Neurosci. 124,500–509.
Osler, S.F., Trautman, G.E., 1961. Concept attainment: II. Effect of stimulus com-plexity upon concept attainment at two levels of intelligence. J. Exp. Psychol.62, 9–13.
Ostlund, S.B., Balleine, B.W., 2005. Lesions of medial prefrontal cortex disrupt theacquisition but not the expression of goal-directed learning. J. Neurosci. 25,7763–7770.
Pisani, A., Gubellini, P., Bonsi, P., Conquet, F., Picconi, B., Centonze, D., Bernardi, G.,Calabresi, P., 2001. Metabotropic glutamate receptor 5 mediates the poten-tiation of N-methyl-D-aspartate responses in medium spiny striatal neurons.Neuroscience 106, 579–587.
Policy OoNDC (2001). The Economic Costs of Drug Abuse in the United States 1992-1998.
Pryor, K.W., Haag, R., O‘Reilly, J., 1969. The creative porpoise: training for novelbehavior. J. Exp. Anal. Behav. 12, 653–661.
Rescorla, R.A., 1994. A note on depression of instrumental responding after one trialof outcome devaluation. Q. J. Exp. Psychol. B 47, 27–37.
Ribeiro, M.J., Schofield, M.G., Kemenes, I., O‘Shea, M., Kemenes, G., Benjamin, P.R.,2005. Activation of MAPK is necessary for long-term memory consolidationfollowing food-reward conditioning. Learn. Mem. 12, 538–545.
Nucleus accumbens dopamine release increases during instrumental leverpressing for food but not free food consumption. Pharmacol. Biochem. Behav.49, 25–31.
Salamone, J.D., Wisniecki, A., Carlson, B.B., Correa, M., 2001. Nucleus accum-bens dopamine depletions make animals highly sensitive to high fixed ratiorequirements but do not impair primary food reinforcement. Neuroscience 105,863–870.
Sallows, G.O., Graupner, T.D., 2005. Intensive behavioral treatment for childrenwith autism: four-year outcome and predictors. Am. J. Ment. Retard. 110, 417–438.
Schafe, G.E., Atkins, C.M., Swank, M.W., Bauer, E.P., Sweatt, J.D., LeDoux, J.E., 2000.Activation of ERK/MAP kinase in the amygdala is required for memory consoli-dation of pavlovian fear conditioning. J. Neurosci. 20, 8177–8187.
Schnaitter, R., 1987. Knowledge as action: the epistemology of radical behavior-ism. In: Modgil, S., Modgil, C. (Eds.), BF Skinner: Consensus and Controversy.Routledge, New York, pp. 57–68.
Schultz, W., 1998. Predictive reward signal of dopamine neurons. J. Neurophysiol.80, 1–27.
Schultz, W., 2002. Getting formal with dopamine and reward. Neuron 36, 241–263.
Seamans, J.K., Durstewitz, D., Christie, B.R., Stevens, C.F., Sejnowski, T.J., 2001.Dopamine D1/D5 receptor modulation of excitatory synaptic inputs to layer Vprefrontal cortex neurons. Proc. Natl. Acad. Sci. U.S.A. 98, 301–306.
Shaywitz, A.J., Greenberg, M.E., 1999. CREB: a stimulus-induced transcription factoractivated by a diverse array of extracellular signals. Annu. Rev. Biochem. 68,821–861.
Skinner, B.F., 1953. Science and Human Behavior. The MacMillan Company, NewYork.
Skinner, B.F., 1957. Verbal Behavior. Appleton-Century-Crofts, New York.Staddon, J.E.R., Simmelhag, V.L., 1971. The “superstition” experiment: a reexannna-
tion of its implications for the principles of adaptive behavior. Psychol. Rev. 78,3–43.
Sweatt, J.D., 2001. The neuronal MAP kinase cascade: a biochemical signal inte-gration system subserving synaptic plasticity and memory. J. Neurochem. 76,1–10.
Thorndike, E., 1911. Animal Intelligence. Macmillan, New York.Valjent, E., Pascoli, V., Svenningsson, P., Paul, S., Enslen, H., Corvol, J.C., Stipanovich,
A., Caboche, J., Lombroso, P.J., Nairn, A.C., Greengard, P., Herve, D., Girault, J.A.,2005. Regulation of a protein phosphatase cascade allows convergent dopamineand glutamate signals to activate ERK in the striatum. Proc. Natl. Acad. Sci. U.S.A.102, 491–496.
accumbens is critical for motivation for drug reinforcement. Neuropsychophar-macology 35 (4), 913–928, http://dx.doi.org/10.1038/npp.2009.193.
arren, Z., McPheeters, M.L., Sathe, N., Foss-Feig, J.H., Glasser, A., Veenstra-Vanderweele, J., 2011. A systematic review of early intensive intervention forautism spectrum disorders. Pediatrics 127, e1303–e1311.
eiss, F., Maldonado-Vlaar, C.S., Parsons, L.H., Kerr, T.M., Smith, D.L., Ben-Shahar,O., 2000. Control of cocaine-seeking behavior by drug-associated stimuli inrats: effects on recovery of extinguished operant-responding and extracellulardopamine levels in amygdala and nucleus accumbens. Proc. Natl. Acad. Sci. U.S.A.97, 4321–4326.
ehavioral Reviews 37 (2013) 2071–2080
Wickens, J.R., Begg, A.J., Arbuthnott, G.W., 1996. Dopamine reverses the depressionof rat corticostriatal synapses which normally follows high-frequency stimula-tion of cortex in vitro. Neuroscience 70, 1–5.
Wise, R.A., Bozarth, M.A., 1985. Brain mechanisms of drug reward and euphoria.
Wood, M.A., Kaplan, M.P., Park, A., Blanchard, E.J., Oliveira, A.M., Lombardi, T.L., Abel,T., 2005. Transgenic mice expressing a truncated form of CREB-binding protein(CBP) exhibit deficits in hippocampal synaptic plasticity and memory storage.Learn. Mem. 12, 111–119.
