2010-2016精選論文2013-1312.5602v1

1、PlayingAtariwithDeepReinfcementLearningVolodymyrMnihKayKavukcuogluDavidSilverAlexGravesIoannisAntonoglouDaanWierstraMartinRiedmillerDeepMindTechnologiesvladkaydavidalex.gravesioannisdaanmartin.riedmiller@AbstractWepresen

2、tthefirstdeeplearningmodeltosuccessfullylearncontrolpoliciesdirectlyfromhighdimensionalsensyinputusingreinfcementlearning.ThemodelisaconvolutionalneuralwktrainedwithavariantofQlearningwhoseinputisrawpixelswhoseoutputisav

3、aluefunctionestimatingfuturerewards.WeapplyourmethodtosevenAtari2600gamesfromtheArcadeLearningEnvironmentwithnoadjustmentofthearchitecturelearningalgithm.Wefindthatitoutperfmsallpreviousapproachesonsixofthegamessurpasses

4、ahumanexpertonthreeofthem.1IntroductionLearningtocontrolagentsdirectlyfromhighdimensionalsensyinputslikevisionspeechisoneofthelongstingchallengesofreinfcementlearning(RL).MostsuccessfulRLapplicationsthatoperateonthesedom

5、ainshavereliedonhcraftedfeaturescombinedwithlinearvaluefunctionspolicyrepresentations.Clearlytheperfmanceofsuchsystemsheavilyreliesonthequalityofthefeaturerepresentation.Recentadvancesindeeplearninghavemadeitpossibletoex

6、tracthighlevelfeaturesfromrawsensydataleadingtobreakthroughsincomputervision[112216]speechrecognition[67].ThesemethodsutilisearangeofneuralwkarchitecturesincludingconvolutionalwksmultilayerperceptronsrestrictedBoltzmannm

7、achinesrecurrentneuralwkshaveexploitedbothsupervisedunsupervisedlearning.ItseemsnaturaltoaskwhethersimilartechniquescouldalsobebeneficialfRLwithsensydata.Howeverreinfcementlearningpresentsseveralchallengesfromadeeplearni

8、ngperspective.Firstlymostsuccessfuldeeplearningapplicationstodatehaverequiredlargeamountsofhlabelledtrainingdata.RLalgithmsontheotherhmustbeabletolearnfromascalarrewardsignalthatisfrequentlysparsenoisydelayed.Thedelaybet

9、weenactionsresultingrewardswhichcanbethoussoftimestepslongseemsparticularlydauntingwhencomparedtothedirectassociationbetweeninputstargetsfoundinsupervisedlearning.Anotherissueisthatmostdeeplearningalgithmsassumethedatasa

10、mplestobeindependentwhileinreinfcementlearningonetypicallyencounterssequencesofhighlycrelatedstates.FurthermeinRLthedatadistributionchangesasthealgithmlearnsnewbehaviourswhichcanbeproblematicfdeeplearningmethodsthatassum

11、eafixedunderlyingdistribution.ThispaperdemonstratesthataconvolutionalneuralwkcanovercomethesechallengestolearnsuccessfulcontrolpoliciesfromrawvideodataincomplexRLenvironments.ThewkistrainedwithavariantoftheQlearning[26]a

12、lgithmwithstochasticgradientdescenttoupdatetheweights.Toalleviatetheproblemsofcrelateddatanonstationarydistributionsweuse1arXiv:1312.5602v1[cs.LG]19Dec2013maximisingtheexpectedvalueofrγQ?(s?a?)Q?(sa)=Es?～E?rγmaxa?Q?(s?a?

13、)???sa?(1)ThebasicideabehindmanyreinfcementlearningalgithmsistoestimatetheactionvaluefunctionbyusingtheBellmanequationasaniterativeupdateQi1(sa)=E[rγmaxa?Qi(s?a?)|sa].Suchvalueiterationalgithmsconvergetotheoptimalactionv

14、aluefunctionQi→Q?asi→∞[23].Inpracticethisbasicapproachistotallyimpracticalbecausetheactionvaluefunctionisestimatedseparatelyfeachsequencewithoutanygeneralisation.Insteaditiscommontouseafunctionapproximattoestimatetheacti

15、onvaluefunctionQ(saθ)≈Q?(sa).Inthereinfcementlearningcommunitythisistypicallyalinearfunctionapproximatbutsometimesanonlinearfunctionapproximatisusedinsteadsuchasaneuralwk.Werefertoaneuralwkfunctionapproximatwithweightsθa

16、saQwk.AQwkcanbetrainedbyminimisingasequenceoflossfunctionsLi(θi)thatchangesateachiterationiLi(θi)=Esa～ρ()?(yi?Q(saθi))2?(2)whereyi=Es?～E[rγmaxa?Q(s?a?θi?1)|sa]isthetargetfiterationiρ(sa)isaprobabilitydistributionoversequ

17、encessactionsathatwerefertoasthebehaviourdistribution.Theparametersfromthepreviousiterationθi?1areheldfixedwhenoptimisingthelossfunctionLi(θi).Notethatthetargetsdependonthewkweightsthisisincontrastwiththetargetsusedfsupe

18、rvisedlearningwhicharefixedbefelearningbegins.Differentiatingthelossfunctionwithrespecttotheweightswearriveatthefollowinggradient?θiLi(θi)=Esa～ρ()s?～E??rγmaxa?Q(s?a?θi?1)?Q(saθi)??θiQ(saθi)?.(3)Ratherthancomputingthefull

19、expectationsintheabovegradientitisoftencomputationallyexpedienttooptimisethelossfunctionbystochasticgradientdescent.Iftheweightsareupdatedaftereverytimesteptheexpectationsarereplacedbysinglesamplesfromthebehaviourdistrib

20、utionρtheemulatErespectivelythenwearriveatthefamiliarQlearningalgithm[26].Notethatthisalgithmismodelfree:itsolvesthereinfcementlearningtaskdirectlyusingsamplesfromtheemulatEwithoutexplicitlyconstructinganestimateofE.Itis

21、alsooffpolicy:itlearnsaboutthegreedystrategya=maxaQ(saθ)whilefollowingabehaviourdistributionthatensuresadequateexplationofthestatespace.Inpracticethebehaviourdistributionisoftenselectedbyan?greedystrategythatfollowsthegr

22、eedystrategywithprobability1??saromactionwithprobability?.3RelatedWkPerhapsthebestknownsuccessstyofreinfcementlearningisTDgammonabackgammonplayingprogramwhichlearntentirelybyreinfcementlearningselfplayachievedasuperhuman

23、levelofplay[24].TDgammonusedamodelfreereinfcementlearningalgithmsimilartoQlearningapproximatedthevaluefunctionusingamultilayerperceptronwithonehiddenlayer1.HoweverearlyattemptstofollowuponTDgammonincludingapplicationsoft

24、hesamemethodtochessGocheckerswerelesssuccessful.ThisledtoawidespreadbeliefthattheTDgammonapproachwasaspecialcasethatonlywkedinbackgammonperhapsbecausethestochasticityinthedicerollshelpsexplethestatespacealsomakesthevalue

25、functionparticularlysmooth[19].FurthermeitwasshownthatcombiningmodelfreereinfcementlearningalgithmssuchasQlearningwithnonlinearfunctionapproximats[25]indeedwithoffpolicylearning[1]couldcausetheQwktodiverge.Subsequentlyth

26、emajityofwkinreinfcementlearningfocusedonlinearfunctionapproximatswithbetterconvergenceguarantees[25].1InfactTDGammonapproximatedthestatevaluefunctionV(s)ratherthantheactionvaluefunctionQ(sa)learntonpolicydirectlyfromthe